cuda decoder:Input features did not have expected dimension: expected 43, got 40

[lkf123010]

all the model file and configure file is run ok in cpu mode(online2-wav-nnet3-latgen-faster)。but it run error in cuda code mode(batched-wav-nnet3-cuda)，information below:

batched-wav-nnet3-cuda --acoustic-scale=1.0 --add-pitch=true --beam=15.0 --max-active=7000 --beam=15.0 --lattice-beam=6.0 --ivector-extraction-config=/data/ai/kaldi_gpu/asrp/model/matchmaker_inviter/conf/ivector_extractor.conf --ivector-silence-weighting.max-state-duration=40 --ivector-silence-weighting.silence-phones=1 --ivector-silence-weighting.silence-weight=0.0 --mfcc-config=/data/ai/kaldi_gpu/asrp/model/matchmaker_inviter/conf/mfcc.conf --online-pitch-config=/data/ai/kaldi_gpu/asrp/model/matchmaker_inviter/conf/online_pitch.conf --word-symbol-table=/data/ai/kaldi_gpu/asrp/model/matchmaker_inviter/words.txt /data/ai/kaldi_gpu/asrp/model/matchmaker_inviter//final.mdl /data/ai/kaldi_gpu/asrp/model/matchmaker_inviter//HCLG.fst 'scp:echo IC100001W10279 IC100001W10279.wav|' 'ark:|lattice-scale --acoustic-scale=10.0 ark:- ark:- | gzip -c >local/tmp/lat.IC100001W10279.gz'
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:SelectGpuId():cu-device.cc:223) CUDA setup operating under Compute Exclusive Mode.
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:FinalizeActiveGpu():cu-device.cc:308) The active GPU is [0]: Tesla M40 24GB free:22689M, used:256M, total:22945M, free/total:0.988832 version 5.2
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:RemoveOrphanNodes():nnet-nnet.cc:948) Removed 1 orphan nodes.
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:RemoveOrphanComponents():nnet-nnet.cc:847) Removing 2 orphan components.
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:Collapse():nnet-utils.cc:1472) Added 1 components, removed 2
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:Initialize():batched-threaded-nnet3-cuda-pipeline.cc:32) BatchedThreadedNnet3CudaPipeline Initialize with 2 control threads, 20 worker threads and batch size 200
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ComputeDerivedVars():ivector-extractor.cc:204) Done.
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ExecuteWorker():batched-threaded-nnet3-cuda-pipeline.cc:801) CudaDecoder batch_size=200 num_channels=260
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ExecuteWorker():batched-threaded-nnet3-cuda-pipeline.cc:801) CudaDecoder batch_size=200 num_channels=260
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:NnetBatchComputer():nnet-batch-compute.cc:51) NnetBatchComputer input_dim_ 43 ivector_dim_100 output_dim_ 4256 seconds.
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:NnetBatchComputer():nnet-batch-compute.cc:51) NnetBatchComputer input_dim_ 43 ivector_dim_100 output_dim_ 4256 seconds.
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ComputeDerivedVars():ivector-extractor.cc:204) Done.
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ComputeDerivedVars():ivector-extractor.cc:204) Done.
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ComputeDerivedVars():ivector-extractor.cc:204) Done.
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ComputeDerivedVars():ivector-extractor.cc:204) Done.
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ComputeDerivedVars():ivector-extractor.cc:204) Done.
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ComputeDerivedVars():ivector-extractor.cc:204) Done.
lattice-scale --acoustic-scale=10.0 ark:- ark:-
ERROR (batched-wav-nnet3-cuda[5.5.669~3-b1d80]:SplitUtteranceIntoTasks():nnet-batch-compute.cc:888) Input features did not have expected dimension: expected 43, got 40

[ Stack-Trace: ]
batched-wav-nnet3-cuda(kaldi::MessageLogger::LogMessage() const+0x98c) [0x9a5c06]
batched-wav-nnet3-cuda(kaldi::MessageLogger::LogAndThrow::operator=(kaldi::MessageLogger const&)+0x11) [0x5d16fb]
batched-wav-nnet3-cuda(kaldi::nnet3::NnetBatchComputer::SplitUtteranceIntoTasks(bool, kaldi::CuMatrix const&, kaldi::CuVector const*, kaldi::CuMatrix const*, int, std::vector<kaldi::nnet3::NnetInferenceTask, std::allocatorkaldi::nnet3::NnetInferenceTask >)+0x1d2) [0x682782]
batched-wav-nnet3-cuda(kaldi::cuda_decoder::BatchedThreadedNnet3CudaPipeline::ComputeBatchNnet(kaldi::nnet3::NnetBatchComputer&, int, std::vector<kaldi::cuda_decoder::BatchedThreadedNnet3CudaPipeline::TaskState, std::allocatorkaldi::cuda_decoder::BatchedThreadedNnet3CudaPipeline::TaskState* >&)+0x1f4) [0x5ef4b0]
batched-wav-nnet3-cuda(kaldi::cuda_decoder::BatchedThreadedNnet3CudaPipeline::ExecuteWorker(int)+0x4ed) [0x5efc23]
batched-wav-nnet3-cuda(std::thread::_State_impl<std::_Bind_simple<std::_Mem_fn<void (kaldi::cuda_decoder::BatchedThreadedNnet3CudaPipeline::)(int)> (kaldi::cuda_decoder::BatchedThreadedNnet3CudaPipeline, int)> >::_M_run()+0x2a) [0x5f1328]
batched-wav-nnet3-cuda() [0x2b83f7f]
/lib64/libpthread.so.0(+0x7e65) [0x7fa2b0969e65]
/lib64/libc.so.6(clone+0x6d) [0x7fa28d82688d]

[whaozl]

Do you solve this problem? Can you tell me? I also have this probelm.

ERROR (batched-wav-nnet3-cuda[5.5]:SplitUtteranceIntoTasks():nnet-batch-compute.cc:885) Input features did not have expected dimension: expected 43, got 40

[JiayiFu]

It seems you should input the mfcc+pitch feature to the model but you only input mfcc feature?

[whaozl]

@JiayiFu but, I use src/online2bin/online2-wav-nnet3-latgen-faster can be used and is normal. but src/cudadecoderbin/batched-wav-nnet3-cuda is above this error.
and the same nnet_online folder. and I also run:

steps/online/nnet3/prepare_online_decoding.sh --add_pitch true data/lang_chain exp/nnet3/extractor exp/chain/tdnn_1a_sp exp/chain/nnet_online

and, refer this https://www.cnblogs.com/sunhongwen/p/9417954.html.
but the latest version kaldi, this batched-wav-nnet3-cuda have this problem. I am solving this problem.

[lkf123010]

it work after re-training the model without pitch

[whaozl]

@lkf23010, but with pitch, it don't work, and have the same problem.

[whaozl]

@lkf123010 , you can delete this bug from the branch. This is not bug. I had solved this problem.

because batched-wav-nnet3-cuda had supported gpu feature extract, but not to pitch feature extract.

so, you can change this defalut config int batched-wav-nnet3-cuda:

batched-threaded-nnet3-cuda-pipeline.h#L59

change this gpu_feature_extract is false. and then make this folder.

[lkf123010]

OK，Thanks @whaozl

[dgxlsir]

all the model file and configure file is run ok in cpu mode(online2-wav-nnet3-latgen-faster)。but it run error in cuda code mode(batched-wav-nnet3-cuda)，information below:

batched-wav-nnet3-cuda --acoustic-scale=1.0 --add-pitch=true --beam=15.0 --max-active=7000 --beam=15.0 --lattice-beam=6.0 --ivector-extraction-config=/data/ai/kaldi_gpu/asrp/model/matchmaker_inviter/conf/ivector_extractor.conf --ivector-silence-weighting.max-state-duration=40 --ivector-silence-weighting.silence-phones=1 --ivector-silence-weighting.silence-weight=0.0 --mfcc-config=/data/ai/kaldi_gpu/asrp/model/matchmaker_inviter/conf/mfcc.conf --online-pitch-config=/data/ai/kaldi_gpu/asrp/model/matchmaker_inviter/conf/online_pitch.conf --word-symbol-table=/data/ai/kaldi_gpu/asrp/model/matchmaker_inviter/words.txt /data/ai/kaldi_gpu/asrp/model/matchmaker_inviter//final.mdl /data/ai/kaldi_gpu/asrp/model/matchmaker_inviter//HCLG.fst 'scp:echo IC100001W10279 IC100001W10279.wav|' 'ark:|lattice-scale --acoustic-scale=10.0 ark:- ark:- | gzip -c >local/tmp/lat.IC100001W10279.gz'
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:SelectGpuId():cu-device.cc:223) CUDA setup operating under Compute Exclusive Mode. LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:FinalizeActiveGpu():cu-device.cc:308) The active GPU is [0]: Tesla M40 24GB free:22689M, used:256M, total:22945M, free/total:0.988832 version 5.2
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:RemoveOrphanNodes():nnet-nnet.cc:948) Removed 1 orphan nodes. LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:RemoveOrphanComponents():nnet-nnet.cc:847) Removing 2 orphan components.
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:Collapse():nnet-utils.cc:1472) Added 1 components, removed 2 LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:Initialize():batched-threaded-nnet3-cuda-pipeline.cc:32) BatchedThreadedNnet3CudaPipeline Initialize with 2 control threads, 20 worker threads and batch size 200
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ComputeDerivedVars():ivector-extractor.cc:204) Done.
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ExecuteWorker():batched-threaded-nnet3-cuda-pipeline.cc:801) CudaDecoder batch_size=200 num_channels=260 LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ExecuteWorker():batched-threaded-nnet3-cuda-pipeline.cc:801) CudaDecoder batch_size=200 num_channels=260
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:NnetBatchComputer():nnet-batch-compute.cc:51) NnetBatchComputer input_dim_ 43 ivector_dim_100 output_dim_ 4256 seconds. LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:NnetBatchComputer():nnet-batch-compute.cc:51) NnetBatchComputer input_dim_ 43 ivector_dim_100 output_dim_ 4256 seconds.
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ComputeDerivedVars():ivector-extractor.cc:204) Done. LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ComputeDerivedVars():ivector-extractor.cc:204) Done.
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ComputeDerivedVars():ivector-extractor.cc:204) Done. LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ComputeDerivedVars():ivector-extractor.cc:204) Done.
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor
LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ComputeDerivedVars():ivector-extractor.cc:204) Done. LOG (batched-wav-nnet3-cuda[5.5.6693-b1d80]:ComputeDerivedVars():ivector-extractor.cc:204) Done.
lattice-scale --acoustic-scale=10.0 ark:- ark:-
ERROR (batched-wav-nnet3-cuda[5.5.669~3-b1d80]:SplitUtteranceIntoTasks():nnet-batch-compute.cc:888) Input features did not have expected dimension: expected 43, got 40

[ Stack-Trace: ]
batched-wav-nnet3-cuda(kaldi::MessageLogger::LogMessage() const+0x98c) [0x9a5c06]
batched-wav-nnet3-cuda(kaldi::MessageLogger::LogAndThrow::operator=(kaldi::MessageLogger const&)+0x11) [0x5d16fb]
batched-wav-nnet3-cuda(kaldi::nnet3::NnetBatchComputer::SplitUtteranceIntoTasks(bool, kaldi::CuMatrix const&, kaldi::CuVector const*, kaldi::CuMatrix const*, int, std::vector<kaldi::nnet3::NnetInferenceTask, std::allocatorkaldi::nnet3::NnetInferenceTask >)+0x1d2) [0x682782] batched-wav-nnet3-cuda(kaldi::cuda_decoder::BatchedThreadedNnet3CudaPipeline::ComputeBatchNnet(kaldi::nnet3::NnetBatchComputer&, int, std::vector<kaldi::cuda_decoder::BatchedThreadedNnet3CudaPipeline::TaskState, std::allocatorkaldi::cuda_decoder::BatchedThreadedNnet3CudaPipeline::TaskState* >&)+0x1f4) [0x5ef4b0]
batched-wav-nnet3-cuda(kaldi::cuda_decoder::BatchedThreadedNnet3CudaPipeline::ExecuteWorker(int)+0x4ed) [0x5efc23]
batched-wav-nnet3-cuda(std:🧵:_State_impl<std::Bind_simple<std::Mem_fn<void (kaldi::cuda_decoder::BatchedThreadedNnet3CudaPipeline::)(int)> (kaldi::cuda_decoder::BatchedThreadedNnet3CudaPipeline, int)> >::_M_run()+0x2a) [0x5f1328]
batched-wav-nnet3-cuda() [0x2b83f7f]
/lib64/libpthread.so.0(+0x7e65) [0x7fa2b0969e65]
/lib64/libc.so.6(clone+0x6d) [0x7fa28d82688d]

hello, I am doing the same thing as you, and the same error, could you please share your scrips to me, I will check my scrips, thank you very much!

[whaozl]

@dgxlsir you can try to use batched-wav-nnet3-cuda2, and script refer link1 and link2.
oh, Do you use pitch feature?

[dgxlsir]

@dgxlsir you can try to use batched-wav-nnet3-cuda2, and script refer link1 and link2.
oh, Do you use pitch feature?
yes, and thanks for your help, i will try your advice!

[c976237222]

@lkf123010 , you can delete this bug from the branch. This is not bug. I had solved this problem.

because batched-wav-nnet3-cuda had supported gpu feature extract, but not to pitch feature extract.

so, you can change this defalut config int batched-wav-nnet3-cuda:

batched-threaded-nnet3-cuda-pipeline.h#L59

change this gpu_feature_extract is false. and then make this folder.

Hello, please. I have changed this option to false, but it still shows the same problem. Do you know what else could be the cause