Add NemoAsrReader handling of UTF8 text #2358
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Joaquin Anton <janton@nvidia.com>
|
!build |
|
CI MESSAGE: [1700932]: BUILD STARTED |
|
CI MESSAGE: [1700932]: BUILD FAILED |
jantonguirao
force-pushed
the
nemo_asr_reader_text_utf8
branch
from
8b9bce1
to
c8b7f53
Compare
|
CI MESSAGE: [1703805]: BUILD STARTED |
Signed-off-by: Joaquin Anton <janton@nvidia.com>
jantonguirao
force-pushed
the
nemo_asr_reader_text_utf8
branch
from
d9d5db1
to
863eef4
Compare
mzient
reviewed
Oct 15, 2020
|
|
||
| { | ||
| std::stringstream ss; | ||
| ss << R"code({"audio_filepath": "path/to/audio1.wav", "duration": 1.45, "text": "这是一个测试"})code" << std::endl; |
There was a problem hiding this comment.
Perhaps we should get the utf-8 bytes and store it as a char array (providing the characters' numerical values) - otherwise it's prone to breaking when the file is transcoded.
klecki
reviewed
Oct 16, 2020
| @@ -85,14 +85,6 @@ in seconds, of the audio samples. | |||
|
|
|||
| Samples with a duration longer than this value will be ignored.)code", | |||
| 0.0f) | |||
| .AddOptionalArg("normalize_text", | |||
| nemo_asr_manifest = os.path.join(tmp_dir.name, "nemo_asr_manifest.json") | ||
| create_manifest_file(nemo_asr_manifest, names, lengths, rates, ref_text_literal) | ||
| ref_text = [np.frombuffer(bytes(s, "utf8"), dtype=np.uint8).reshape(-1) for s in ref_text_literal] |
There was a problem hiding this comment.
Checked, not needed
jantonguirao
force-pushed
the
nemo_asr_reader_text_utf8
branch
from
98bf028
to
81937a1
Compare
Signed-off-by: Joaquin Anton <janton@nvidia.com>
jantonguirao
force-pushed
the
nemo_asr_reader_text_utf8
branch
from
81937a1
to
6f073f8
Compare
mzient
approved these changes
Oct 16, 2020
klecki
approved these changes
Oct 16, 2020
|
!build |
|
CI MESSAGE: [1707805]: BUILD STARTED |
|
CI MESSAGE: [1707805]: BUILD PASSED |
JanuszL
approved these changes
Oct 16, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Joaquin Anton janton@nvidia.com
Why we need this PR?
What happened in this PR?
Fill relevant points, put NA otherwise. Replace anything inside []
NeMo manifests are encoded in utf8. Made sure that the output of the reader is the expected utf8 encoded string
Tests added
Removed trailing 0 in text outputs
Removed text normalization, which will be left to the python code
NemoAsrReader
N/A
New tests added
Docstr updated
JIRA TASK: [DALI-1635]