Issue at step 8 test.py

[hypernovas]

Hi there,

Unfortunately, I got another error here. The error message a bit long but I feel it's because I cannot open FAKEBOB/test-iv-CSI/ivector/ivector.scp as highlighted below. Any suggestions?

The file is not empty, and with a size of 416 bytes.

---

./data/test-set/2830/2830-3980-0065.wav

./data/test-set/2830/2830-3980-0071.wav

./data/test-set/2830/2830-3980-0059.wav

./data/test-set/2830/2830-3980-0058.wav

./data/test-set/2830/2830-3980-0070.wav

./data/test-set/2830/2830-3980-0064.wav

fix_data_dir.sh: kept all 495 utterances.

fix_data_dir.sh: old files are kept in /Users/user1/FAKEBOB/test-iv-CSI/audio/.backup

/Users/user1/FAKEBOB/pre-models/steps/make_mfcc.sh --write-utt2num-frames true --mfcc-config /Users/user1/FAKEBOB/pre-models/conf/mfcc.conf --nj 1 --cmd $train_cmd /Users/user1/FAKEBOB/test-iv-CSI/audio /Users/user1/FAKEBOB/test-iv-CSI/log /Users/user1/FAKEBOB/test-iv-CSI/mfcc

/Users/user1/FAKEBOB/pre-models/steps/make_mfcc.sh: empty argument to --cmd option

/Users/user1/FAKEBOB/pre-models/steps/compute_vad_decision.sh --nj 1 --cmd $train_cmd --vad-config /Users/user1/FAKEBOB/pre-models/conf/vad.conf /Users/user1/FAKEBOB/test-iv-CSI/audio /Users/user1/FAKEBOB/test-iv-CSI/log /Users/user1/FAKEBOB/test-iv-CSI/mfcc

/Users/user1/FAKEBOB/pre-models/steps/compute_vad_decision.sh: empty argument to --cmd option

/Users/user1/FAKEBOB/pre-models/sid/extract_ivectors.sh --cmd $train_cmd --nj 1 --num-threads 1 /Users/user1/FAKEBOB/pre-models /Users/user1/FAKEBOB/test-iv-CSI/audio /Users/user1/FAKEBOB/test-iv-CSI/ivector

/Users/user1/FAKEBOB/pre-models/sid/extract_ivectors.sh: empty argument to --cmd option

ivector-plda-scoring --normalize-length=true 'ivector-copy-plda --smoothing=0.0 /Users/user1/FAKEBOB/pre-models/plda - |' 'ark:ivector-subtract-global-mean /Users/user1/FAKEBOB/pre-models/mean.vec scp:/Users/user1/FAKEBOB/test-iv-CSI/ivector.scp ark:- | transform-vec /Users/user1/FAKEBOB/pre-models/transform.mat ark:- ark:- | ivector-normalize-length ark:- ark:- |' 'ark:ivector-subtract-global-mean /Users/user1/FAKEBOB/pre-models/mean.vec scp:/Users/user1/FAKEBOB/test-iv-CSI/ivector/ivector.scp ark:- | transform-vec /Users/user1/FAKEBOB/pre-models/transform.mat ark:- ark:- | ivector-normalize-length ark:- ark:- |' "cat /Users/user1/FAKEBOB/test-iv-CSI/ivector/trials | cut -d' ' -f 1,2 |" /Users/user1/FAKEBOB/test-iv-CSI/ivector/scores

ivector-copy-plda --smoothing=0.0 /Users/user1/FAKEBOB/pre-models/plda -

ivector-subtract-global-mean /Users/user1/FAKEBOB/pre-models/mean.vec scp:/Users/user1/FAKEBOB/test-iv-CSI/ivector.scp ark:-

ivector-normalize-length ark:- ark:-

LOG (ivector-subtract-global-mean[5.5.713~3-31c2]:main():ivector-subtract-global-mean.cc:108) Wrote 5 mean-subtracted iVectors

transform-vec /Users/user1/FAKEBOB/pre-models/transform.mat ark:- ark:-

LOG (transform-vec[5.5.713~3-31c2]:main():transform-vec.cc:85) Applied transform to 5 vectors.

LOG (ivector-normalize-length[5.5.713~3-31c2]:main():ivector-normalize-length.cc:90) Processed 5 iVectors.

LOG (ivector-normalize-length[5.5.713~3-31c2]:main():ivector-normalize-length.cc:94) Average ratio of iVector to expected length was 1.33827, standard deviation was 0.0804826

ivector-subtract-global-mean /Users/user1/FAKEBOB/pre-models/mean.vec scp:/Users/user1/FAKEBOB/test-iv-CSI/ivector/ivector.scp ark:-

ivector-normalize-length ark:- ark:-

WARNING (ivector-subtract-global-mean[5.5.713~3-31c2]:Open():util/kaldi-table-inl.h:106) Failed to open script file /Users/user1/FAKEBOB/test-iv-CSI/ivector/ivector.scp

ERROR (ivector-subtract-global-mean[5.5.713~3-31c2]:SequentialTableReader():util/kaldi-table-inl.h:860) Error constructing TableReader: rspecifier is scp:/Users/user1/FAKEBOB/test-iv-CSI/ivector/ivector.scp

[ Stack-Trace: ]

0 libkaldi-base.dylib 0x000000010878692f kaldi::KaldiGetStackTrace() + 63

1 libkaldi-base.dylib 0x00000001087866a2 kaldi::MessageLogger::LogMessage() const + 354

2 ivector-subtract-global-mean 0x0000000107bdd178 kaldi::MessageLogger::LogAndThrow::operator=(kaldi::MessageLogger const&) + 24

3 ivector-subtract-global-mean 0x0000000107bdcf07 kaldi::SequentialTableReader<kaldi::KaldiObjectHolder<kaldi::Vector > >::SequentialTableReader(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&) + 151

4 ivector-subtract-global-mean 0x0000000107bd9adc main + 684

5 libdyld.dylib 0x00007fff689df7fd start + 1

kaldi::KaldiFatalErrortransform-vec /Users/user1/FAKEBOB/pre-models/transform.mat ark:- ark:-

LOG (transform-vec[5.5.713~3-31c2]:main():transform-vec.cc:85) Applied transform to 0 vectors.

LOG (ivector-normalize-length[5.5.713~3-31c2]:main():ivector-normalize-length.cc:90) Processed 0 iVectors.

LOG (ivector-plda-scoring[5.5.713~3-31c2]:main():ivector-plda-scoring.cc:96) Reading train iVectors

LOG (ivector-plda-scoring[5.5.713~3-31c2]:main():ivector-plda-scoring.cc:122) Read 5 training iVectors, errors on 0

LOG (ivector-plda-scoring[5.5.713~3-31c2]:main():ivector-plda-scoring.cc:126) Average renormalization scale on training iVectors was 0.987333

LOG (ivector-plda-scoring[5.5.713~3-31c2]:main():ivector-plda-scoring.cc:129) Reading test iVectors

LOG (ivector-plda-scoring[5.5.713~3-31c2]:main():ivector-plda-scoring.cc:147) Read 0 test iVectors.

ERROR (ivector-plda-scoring[5.5.713~3-31c2]:main():ivector-plda-scoring.cc:149) No test iVectors present.

[ Stack-Trace: ]

0 libkaldi-base.dylib 0x000000010ac6e92f kaldi::KaldiGetStackTrace() + 63

1 libkaldi-base.dylib 0x000000010ac6e6a2 kaldi::MessageLogger::LogMessage() const + 354

2 ivector-plda-scoring 0x000000010a08d3a8 kaldi::MessageLogger::LogAndThrow::operator=(kaldi::MessageLogger const&) + 24

3 ivector-plda-scoring 0x000000010a08cd2c main + 4012

4 libdyld.dylib 0x00007fff689df7fd start + 1

WARNING (ivector-plda-scoring[5.5.713~3-31c2]:Close():kaldi-io.cc:515) Pipe ivector-subtract-global-mean /Users/user1/FAKEBOB/pre-models/mean.vec scp:/Users/user1/FAKEBOB/test-iv-CSI/ivector/ivector.scp ark:- | transform-vec /Users/user1/FAKEBOB/pre-models/transform.mat ark:- ark:- | ivector-normalize-length ark:- ark:- | had nonzero return status 256

ERROR (ivector-plda-scoring[5.5.713~3-31c2]:~SequentialTableReaderArchiveImpl():util/kaldi-table-inl.h:678) TableReader: error detected closing archive 'ivector-subtract-global-mean /Users/user1/FAKEBOB/pre-models/mean.vec scp:/Users/user1/FAKEBOB/test-iv-CSI/ivector/ivector.scp ark:- | transform-vec /Users/user1/FAKEBOB/pre-models/transform.mat ark:- ark:- | ivector-normalize-length ark:- ark:- |'

[ Stack-Trace: ]

0 libkaldi-base.dylib 0x000000010ac6e92f kaldi::KaldiGetStackTrace() + 63

1 libkaldi-base.dylib 0x000000010ac6e6a2 kaldi::MessageLogger::LogMessage() const + 354

2 ivector-plda-scoring 0x000000010a08d3a8 kaldi::MessageLogger::LogAndThrow::operator=(kaldi::MessageLogger const&) + 24

3 ivector-plda-scoring 0x000000010a092375 kaldi::SequentialTableReaderArchiveImpl<kaldi::KaldiObjectHolder<kaldi::Vector > >::~SequentialTableReaderArchiveImpl() + 213

4 ivector-plda-scoring 0x000000010a091c0e kaldi::SequentialTableReaderArchiveImpl<kaldi::KaldiObjectHolder<kaldi::Vector > >::~SequentialTableReaderArchiveImpl() + 14

5 ivector-plda-scoring 0x000000010a08d03d main + 4797

6 libdyld.dylib 0x00007fff689df7fd start + 1

libc++abi.dylib: terminating with uncaught exception of type kaldi::KaldiFatalError: kaldi::KaldiFatalError

Traceback (most recent call last):

File "test.py", line 97, in

decisions, _ = iv_csi_model.make_decisions(audio_list, debug=debug, n_jobs=n_jobs)

File "/Users/user1/FAKEBOB/ivector_PLDA_CSI.py", line 124, in make_decisions

score_array = self.score(audios, fs=fs, bits_per_sample=bits_per_sample, n_jobs=n_jobs, debug=debug)

File "/Users/user1/FAKEBOB/ivector_PLDA_CSI.py", line 116, in score

score_array = self.kaldi_helper.score(audio_list, self.utt_ids, n_jobs=n_jobs, flag=1, train_ivector_scp=self.train_ivector_scp, debug=debug)

File "/Users/user1/FAKEBOB/ivector_PLDA_kaldiHelper.py", line 363, in score

score_array = self.resolve_score()

File "/Users/user1/FAKEBOB/ivector_PLDA_kaldiHelper.py", line 286, in resolve_score

scores_mat = np.loadtxt(scores_file, dtype=str)

File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/lib/npyio.py", line 981, in loadtxt

fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)

File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/lib/_datasource.py", line 269, in open

return ds.open(path, mode, encoding=encoding, newline=newline)

File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/lib/_datasource.py", line 623, in open

raise IOError("%s not found." % path)

OSError: /Users/user1/FAKEBOB/test-iv-CSI/ivector/scores not found. `

[GolferChen]

Since the file is not empty, I infer maybe it is the file permission problem. Please let me know whether or not, so I can update the code to resolve the permission problem automatically.

[GolferChen]

For quick debug, please add the following code to Line 253 in ivector_PLDA_kaldiHelper.py and re-run the code.

permission_command = "chmod -R 777 " + self.ivector_dir
import os
os.system(permission_command)

The programe will run well after adding the above code if the issue is indeed raised by file permission problem.

[hypernovas]

Hmm, I didn't notice this before but it seems that the problem is ivector.scp is put under "/Users/user1/FAKEBOB/test-iv-CSI/"
instead of "/Users/user1/FAKEBOB/test-iv-CSI/ivector/"

[GolferChen]

Actually, there should be two ivector.scp files, the one placed in /Users/user1/FAKEBOB/test-iv-CSI/ is for training voices (i.e., the enrollment voices) , and the other one placed in /Users/user1/FAKEBOB/test-iv-CSI/ivector/ is for testing voices (i.e., voices to be scored).

So please check whether both files exist.

[GolferChen]

Are you sure every step of kaldi runs successfully, including mfcc extraction, vad decision and ivector extractor ?

The warning message "empty argument to --cmd option" appears many times in your provided log.

I doubt that the ivectors for testing voices did not get generated, so /Users/user1/FAKEBOB/test-iv-CSI/ivector/ivector.scp does not exist. You can verify this by checking if there are many .ark files in /Users/user1/FAKEBOB/test-iv-CSI/ivector/.

[hypernovas]

Ok. The only file in /Users/user1/FAKEBOB/test-iv-CSI/ivector/ is trials

[GolferChen]

I got it. The ivectors for testing voices did not get generated. Can you check /Users/user1/FAKEBOB/test-iv-CSI/mfcc/ ? Are there many ".ark" files there ?

[GolferChen]

I doubt that your step 7 actually did not run normally.

[hypernovas]

Got a chance to set up an ubuntu environment, it just works more smooth. Everything is working fine now. Some notes that might help.

1. data.tgz file contains some links (instead of audio files) which create some errors
2. Some weird bugs can be fixed by setting up nj to be 1, there could be some sync problem in local machine

Thanks a lot for all helps.

[GolferChen]

Ok. Thanks for your feedback.