This repository was archived by the owner on Jul 7, 2023. It is now read-only.
Source/target pair text files; Icelandic parsing support; fixes #154
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lukaszkaiser
suggested changes
Jul 14, 2017
| vocab = text_encoder.SubwordTextEncoder(vocab_filepath) | ||
| return vocab | ||
|
|
||
| tokenizer = Tokenizer() |
There was a problem hiding this comment.
This line should be removed.
|
Yes, sorry, I missed this change in the final merge with upstream. Should be OK now. |
|
My pleasure! :-) |
|
I just found one small problem (source_vocab is used instead of target_vocab in wmt.tabbed_generator). I'll correct it in my next PR and release. It'd be great though to add an option to download the Icelandic parsing set instead of assuming it's there in a file. Is the set public? |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds text data generators that read source and target sentences in pairs, separated by tabs (\t), from a single source file.
It also adds scaffolding to support parsing of Icelandic text, either from characters or from subword tokens. (More coming later on that front.)
A few smaller fixes are included, such as replacing magic constants to do with reserved token ids with more meaningful identifiers, and getting rid of annoying strings in decoder output trace messages.