Add aishell recipe #30
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I have an error with the newest master branch of k2. |
|
Hm. Try doing:
export CUDA_LAUNCH_BLOCKING=1
and see if you can get a Python stack trace.
…On Sun, Sep 19, 2021 at 10:40 AM fanlu ***@***.***> wrote:
I have an error with the newest master branch of k2.
2021-09-19 09:40:39,362 INFO [decode.py:419] Decoding started
2021-09-19 09:40:39,362 INFO [decode.py:420] {'exp_dir': PosixPath('conformer_ctc/exp'), 'lang_dir': PosixPath('data/lang_char'), 'lm_dir': PosixPath('data/lm'), 'subsampling_factor': 4, 'feature_dim': 80, 'nhead': 4, 'attention_dim': 512, 'num_en$
oder_layers': 12, 'num_decoder_layers': 6, 'vgg_frontend': False, 'use_feat_batchnorm': True, 'search_beam': 20, 'output_beam': 7, 'min_active_states': 30, 'max_active_states': 10000, 'use_double_scores': True, 'epoch': 40, 'avg': 18, 'method': 'a$
tention-decoder', 'num_paths': 100, 'lattice_score_scale': 0.5, 'export': False, 'feature_dir': PosixPath('data/fbank'), 'max_duration': 20, 'bucketing_sampler': True, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.$
, 'on_the_fly_feats': False, 'shuffle': True, 'return_cuts': True, 'num_workers': 2}
2021-09-19 09:40:39,633 INFO [lexicon.py:113] Loading pre-compiled data/lang_char/Linv.pt
2021-09-19 09:40:39,834 INFO [decode.py:430] device: cuda:0
2021-09-19 09:40:50,195 INFO [decode.py:470] averaging ['conformer_ctc/exp/epoch-23.pt', 'conformer_ctc/exp/epoch-24.pt', 'conformer_ctc/exp/epoch-25.pt', 'conformer_ctc/exp/epoch-26.pt', 'conformer_ctc/exp/epoch-27.pt', 'conformer_ctc/exp/epoch-2$
.pt', 'conformer_ctc/exp/epoch-29.pt', 'conformer_ctc/exp/epoch-30.pt', 'conformer_ctc/exp/epoch-31.pt', 'conformer_ctc/exp/epoch-32.pt', 'conformer_ctc/exp/epoch-33.pt', 'conformer_ctc/exp/epoch-34.pt', 'conformer_ctc/exp/epoch-35.pt', 'conformer$
ctc/exp/epoch-36.pt', 'conformer_ctc/exp/epoch-37.pt', 'conformer_ctc/exp/epoch-38.pt', 'conformer_ctc/exp/epoch-39.pt', 'conformer_ctc/exp/epoch-40.pt']
2021-09-19 09:41:17,977 INFO [decode.py:483] Number of model parameters: 115125888
[F] /asr_storage/fanlu/k2/k2/csrc/array.h:501:void k2::Array1<T>::Init(k2::ContextPtr, int32_t, k2::Dtype) [with T = int; k2::ContextPtr = std::shared_ptr<k2::Context>; int32_t = int] Check failed: size >= 0 (-119123572 vs. 0) Array size MUST be g$
eater than or equal to 0, given :-119123572
[ Stack-Trace: ]
/asr_storage/fanlu/k2/build_release/lib/libk2_log.so(k2::internal::GetStackTrace()+0x3c) [0x2af6a59029fc]
/asr_storage/fanlu/k2/build_release/lib/libk2context.so(k2::internal::Logger::~Logger()+0x2a) [0x2af6a40b037a]
/asr_storage/fanlu/k2/build_release/lib/libk2context.so(k2::Array1<int>::Init(std::shared_ptr<k2::Context>, int, k2::Dtype)+0x26d) [0x2af6a40c56dd]
/asr_storage/fanlu/k2/build_release/lib/libk2context.so(+0x25a19d) [0x2af6a425619d]
/asr_storage/fanlu/k2/build_release/lib/libk2context.so(+0x272d92) [0x2af6a426ed92]
/asr_storage/fanlu/k2/build_release/lib/libk2context.so(k2::Index(k2::RaggedShape&, int, k2::Array1<int> const&, k2::Array1<int>*)+0x1b2) [0x2af6a4270eb2]
/asr_storage/fanlu/k2/build_release/lib/_k2.cpython-38-x86_64-linux-gnu.so(+0x147977) [0x2af6a329e977]
/asr_storage/fanlu/k2/build_release/lib/_k2.cpython-38-x86_64-linux-gnu.so(+0x1197b0) [0x2af6a32707b0]
/asr_storage/fanlu/k2/build_release/lib/_k2.cpython-38-x86_64-linux-gnu.so(+0x10be3d) [0x2af6a3262e3d]
/asr_storage/fanlu/k2/build_release/lib/_k2.cpython-38-x86_64-linux-gnu.so(+0x10c08e) [0x2af6a326308e]
/asr_storage/fanlu/k2/build_release/lib/_k2.cpython-38-x86_64-linux-gnu.so(+0x24663) [0x2af6a317b663]
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#30 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAZFLOYNHUQGWJI5WSJL7TDUCVERLANCNFSM5DBGHUOQ>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
|
|
.. and if you can't get a Python stack trace, then:
(a) I'd like to figure out what is wrong with our assert-failure mechanism
that we can't get this,
(b) to debug the immediate issue, you may have to insert debug print
statements to figure out where we are-- some kind of binary search into the
code.
…On Sun, Sep 19, 2021 at 10:47 AM Daniel Povey ***@***.***> wrote:
Hm. Try doing:
export CUDA_LAUNCH_BLOCKING=1
and see if you can get a Python stack trace.
On Sun, Sep 19, 2021 at 10:40 AM fanlu ***@***.***> wrote:
> I have an error with the newest master branch of k2.
>
> 2021-09-19 09:40:39,362 INFO [decode.py:419] Decoding started
> 2021-09-19 09:40:39,362 INFO [decode.py:420] {'exp_dir': PosixPath('conformer_ctc/exp'), 'lang_dir': PosixPath('data/lang_char'), 'lm_dir': PosixPath('data/lm'), 'subsampling_factor': 4, 'feature_dim': 80, 'nhead': 4, 'attention_dim': 512, 'num_en$
> oder_layers': 12, 'num_decoder_layers': 6, 'vgg_frontend': False, 'use_feat_batchnorm': True, 'search_beam': 20, 'output_beam': 7, 'min_active_states': 30, 'max_active_states': 10000, 'use_double_scores': True, 'epoch': 40, 'avg': 18, 'method': 'a$
> tention-decoder', 'num_paths': 100, 'lattice_score_scale': 0.5, 'export': False, 'feature_dir': PosixPath('data/fbank'), 'max_duration': 20, 'bucketing_sampler': True, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.$
> , 'on_the_fly_feats': False, 'shuffle': True, 'return_cuts': True, 'num_workers': 2}
> 2021-09-19 09:40:39,633 INFO [lexicon.py:113] Loading pre-compiled data/lang_char/Linv.pt
> 2021-09-19 09:40:39,834 INFO [decode.py:430] device: cuda:0
> 2021-09-19 09:40:50,195 INFO [decode.py:470] averaging ['conformer_ctc/exp/epoch-23.pt', 'conformer_ctc/exp/epoch-24.pt', 'conformer_ctc/exp/epoch-25.pt', 'conformer_ctc/exp/epoch-26.pt', 'conformer_ctc/exp/epoch-27.pt', 'conformer_ctc/exp/epoch-2$
> .pt', 'conformer_ctc/exp/epoch-29.pt', 'conformer_ctc/exp/epoch-30.pt', 'conformer_ctc/exp/epoch-31.pt', 'conformer_ctc/exp/epoch-32.pt', 'conformer_ctc/exp/epoch-33.pt', 'conformer_ctc/exp/epoch-34.pt', 'conformer_ctc/exp/epoch-35.pt', 'conformer$
> ctc/exp/epoch-36.pt', 'conformer_ctc/exp/epoch-37.pt', 'conformer_ctc/exp/epoch-38.pt', 'conformer_ctc/exp/epoch-39.pt', 'conformer_ctc/exp/epoch-40.pt']
> 2021-09-19 09:41:17,977 INFO [decode.py:483] Number of model parameters: 115125888
> [F] /asr_storage/fanlu/k2/k2/csrc/array.h:501:void k2::Array1<T>::Init(k2::ContextPtr, int32_t, k2::Dtype) [with T = int; k2::ContextPtr = std::shared_ptr<k2::Context>; int32_t = int] Check failed: size >= 0 (-119123572 vs. 0) Array size MUST be g$
> eater than or equal to 0, given :-119123572
>
>
> [ Stack-Trace: ]
> /asr_storage/fanlu/k2/build_release/lib/libk2_log.so(k2::internal::GetStackTrace()+0x3c) [0x2af6a59029fc]
> /asr_storage/fanlu/k2/build_release/lib/libk2context.so(k2::internal::Logger::~Logger()+0x2a) [0x2af6a40b037a]
> /asr_storage/fanlu/k2/build_release/lib/libk2context.so(k2::Array1<int>::Init(std::shared_ptr<k2::Context>, int, k2::Dtype)+0x26d) [0x2af6a40c56dd]
> /asr_storage/fanlu/k2/build_release/lib/libk2context.so(+0x25a19d) [0x2af6a425619d]
> /asr_storage/fanlu/k2/build_release/lib/libk2context.so(+0x272d92) [0x2af6a426ed92]
> /asr_storage/fanlu/k2/build_release/lib/libk2context.so(k2::Index(k2::RaggedShape&, int, k2::Array1<int> const&, k2::Array1<int>*)+0x1b2) [0x2af6a4270eb2]
> /asr_storage/fanlu/k2/build_release/lib/_k2.cpython-38-x86_64-linux-gnu.so(+0x147977) [0x2af6a329e977]
> /asr_storage/fanlu/k2/build_release/lib/_k2.cpython-38-x86_64-linux-gnu.so(+0x1197b0) [0x2af6a32707b0]
> /asr_storage/fanlu/k2/build_release/lib/_k2.cpython-38-x86_64-linux-gnu.so(+0x10be3d) [0x2af6a3262e3d]
> /asr_storage/fanlu/k2/build_release/lib/_k2.cpython-38-x86_64-linux-gnu.so(+0x10c08e) [0x2af6a326308e]
> /asr_storage/fanlu/k2/build_release/lib/_k2.cpython-38-x86_64-linux-gnu.so(+0x24663) [0x2af6a317b663]
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <#30 (comment)>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAZFLOYNHUQGWJI5WSJL7TDUCVERLANCNFSM5DBGHUOQ>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
>
|
|
... Another way to debug this might be to run it in gdb, as in
gdb --args python3 decode.py ...
(gdb) r
and see if you can get any info from the C++ stack.
…On Sun, Sep 19, 2021 at 10:48 AM Daniel Povey ***@***.***> wrote:
.. and if you can't get a Python stack trace, then:
(a) I'd like to figure out what is wrong with our assert-failure
mechanism that we can't get this,
(b) to debug the immediate issue, you may have to insert debug print
statements to figure out where we are-- some kind of binary search into the
code.
On Sun, Sep 19, 2021 at 10:47 AM Daniel Povey ***@***.***> wrote:
> Hm. Try doing:
> export CUDA_LAUNCH_BLOCKING=1
> and see if you can get a Python stack trace.
>
> On Sun, Sep 19, 2021 at 10:40 AM fanlu ***@***.***> wrote:
>
>> I have an error with the newest master branch of k2.
>>
>> 2021-09-19 09:40:39,362 INFO [decode.py:419] Decoding started
>> 2021-09-19 09:40:39,362 INFO [decode.py:420] {'exp_dir': PosixPath('conformer_ctc/exp'), 'lang_dir': PosixPath('data/lang_char'), 'lm_dir': PosixPath('data/lm'), 'subsampling_factor': 4, 'feature_dim': 80, 'nhead': 4, 'attention_dim': 512, 'num_en$
>> oder_layers': 12, 'num_decoder_layers': 6, 'vgg_frontend': False, 'use_feat_batchnorm': True, 'search_beam': 20, 'output_beam': 7, 'min_active_states': 30, 'max_active_states': 10000, 'use_double_scores': True, 'epoch': 40, 'avg': 18, 'method': 'a$
>> tention-decoder', 'num_paths': 100, 'lattice_score_scale': 0.5, 'export': False, 'feature_dir': PosixPath('data/fbank'), 'max_duration': 20, 'bucketing_sampler': True, 'num_buckets': 30, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 1.$
>> , 'on_the_fly_feats': False, 'shuffle': True, 'return_cuts': True, 'num_workers': 2}
>> 2021-09-19 09:40:39,633 INFO [lexicon.py:113] Loading pre-compiled data/lang_char/Linv.pt
>> 2021-09-19 09:40:39,834 INFO [decode.py:430] device: cuda:0
>> 2021-09-19 09:40:50,195 INFO [decode.py:470] averaging ['conformer_ctc/exp/epoch-23.pt', 'conformer_ctc/exp/epoch-24.pt', 'conformer_ctc/exp/epoch-25.pt', 'conformer_ctc/exp/epoch-26.pt', 'conformer_ctc/exp/epoch-27.pt', 'conformer_ctc/exp/epoch-2$
>> .pt', 'conformer_ctc/exp/epoch-29.pt', 'conformer_ctc/exp/epoch-30.pt', 'conformer_ctc/exp/epoch-31.pt', 'conformer_ctc/exp/epoch-32.pt', 'conformer_ctc/exp/epoch-33.pt', 'conformer_ctc/exp/epoch-34.pt', 'conformer_ctc/exp/epoch-35.pt', 'conformer$
>> ctc/exp/epoch-36.pt', 'conformer_ctc/exp/epoch-37.pt', 'conformer_ctc/exp/epoch-38.pt', 'conformer_ctc/exp/epoch-39.pt', 'conformer_ctc/exp/epoch-40.pt']
>> 2021-09-19 09:41:17,977 INFO [decode.py:483] Number of model parameters: 115125888
>> [F] /asr_storage/fanlu/k2/k2/csrc/array.h:501:void k2::Array1<T>::Init(k2::ContextPtr, int32_t, k2::Dtype) [with T = int; k2::ContextPtr = std::shared_ptr<k2::Context>; int32_t = int] Check failed: size >= 0 (-119123572 vs. 0) Array size MUST be g$
>> eater than or equal to 0, given :-119123572
>>
>>
>> [ Stack-Trace: ]
>> /asr_storage/fanlu/k2/build_release/lib/libk2_log.so(k2::internal::GetStackTrace()+0x3c) [0x2af6a59029fc]
>> /asr_storage/fanlu/k2/build_release/lib/libk2context.so(k2::internal::Logger::~Logger()+0x2a) [0x2af6a40b037a]
>> /asr_storage/fanlu/k2/build_release/lib/libk2context.so(k2::Array1<int>::Init(std::shared_ptr<k2::Context>, int, k2::Dtype)+0x26d) [0x2af6a40c56dd]
>> /asr_storage/fanlu/k2/build_release/lib/libk2context.so(+0x25a19d) [0x2af6a425619d]
>> /asr_storage/fanlu/k2/build_release/lib/libk2context.so(+0x272d92) [0x2af6a426ed92]
>> /asr_storage/fanlu/k2/build_release/lib/libk2context.so(k2::Index(k2::RaggedShape&, int, k2::Array1<int> const&, k2::Array1<int>*)+0x1b2) [0x2af6a4270eb2]
>> /asr_storage/fanlu/k2/build_release/lib/_k2.cpython-38-x86_64-linux-gnu.so(+0x147977) [0x2af6a329e977]
>> /asr_storage/fanlu/k2/build_release/lib/_k2.cpython-38-x86_64-linux-gnu.so(+0x1197b0) [0x2af6a32707b0]
>> /asr_storage/fanlu/k2/build_release/lib/_k2.cpython-38-x86_64-linux-gnu.so(+0x10be3d) [0x2af6a3262e3d]
>> /asr_storage/fanlu/k2/build_release/lib/_k2.cpython-38-x86_64-linux-gnu.so(+0x10c08e) [0x2af6a326308e]
>> /asr_storage/fanlu/k2/build_release/lib/_k2.cpython-38-x86_64-linux-gnu.so(+0x24663) [0x2af6a317b663]
>>
>> —
>> You are receiving this because you are subscribed to this thread.
>> Reply to this email directly, view it on GitHub
>> <#30 (comment)>, or
>> unsubscribe
>> <https://github.com/notifications/unsubscribe-auth/AAZFLOYNHUQGWJI5WSJL7TDUCVERLANCNFSM5DBGHUOQ>
>> .
>> Triage notifications on the go with GitHub Mobile for iOS
>> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
>> or Android
>> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>>
>>
>
|
There are instructions at the end of the log, telling you how to use gdb to debug your Python code. |
|
Are there Python stack traces at the end of the log that are not posted? |
|
@csukuangfj the end of the log is |
Could you set a pdb breakpoint at line 826 and print the shape information of Some useful instructions are: import pdb
pdb.set_trace()
print(token_seq.shape.tot_sizes())
print(new2old.shape)
# token_seq, _ = token_seq.index(... |
|
|
Does this |
|
|
We resolved it offline. It is a bug that has been fixed in #50 |
pkufool
commented
Sep 26, 2021
egs/aishell/ASR/prepare.sh
Outdated
| # |-- lexicon.txt | ||
| # `-- speaker.info | ||
|
|
||
| if [ ! -d $dl_dir/aishell/wav ]; then |
There was a problem hiding this comment.
$dl_dir/aishell/data_aishell/wav
|
Latest result: Tensorboard: https://tensorboard.dev/experiment/WE1DocDqRRCOSAgmGyClhg/#scalars
|
|
Could you also update https://github.com/k2-fsa/icefall/blob/master/README.md ? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add aishell recipe to icefall. It contains two models now, one is phone based Tdnn-Lstm model, the other is char based (with chinese chars as modelling units) comformer-ctc model.
The decoding results seems not as good as expected now, I am still tuning the models.TODOs: