Fix both utt2sid and utt2lid when removing long/short data

[jonghwanhyeon]

Stage 3 of tts.sh removes long/short data. At this time, authors use utils/fix_data_dir.sh to remove the entry of the removed utterance as follows:

espnet/egs2/TEMPLATE/tts1/tts.sh

Lines 500 to 509 in e5d133c

	# fix_data_dir.sh leaves only utts which exist in all files
	_fix_opts=""
	if [ -e "${data_feats}/org/${dset}/utt2sid" ]; then
	_fix_opts="--utt_extra_files utt2sid "
	fi
	if [ -e "${data_feats}/org/${dset}/utt2lid" ]; then
	_fix_opts="--utt_extra_files utt2lid "
	fi
	# shellcheck disable=SC2086
	utils/fix_data_dir.sh ${_fix_opts} "${data_feats}/${dset}"

However, if there exist both utt2sid and utt2lid, _fix_opts are overwritten as "--utt_extra_files utt2lid " because authors used the assignment operator. So I tried to fix the above code as follows:

# fix_data_dir.sh leaves only utts which exist in all files
_fix_opts=""
if [ -e "${data_feats}/org/${dset}/utt2sid" ]; then
    _fix_opts+="--utt_extra_files utt2sid "
fi
if [ -e "${data_feats}/org/${dset}/utt2lid" ]; then
    _fix_opts+="--utt_extra_files utt2lid "
fi
# shellcheck disable=SC2086
utils/fix_data_dir.sh ${_fix_opts} "${data_feats}/${dset}"

However, after I tested it, it seems that utils/fix_data_dir.sh does not support multiple options. So, I suggest the following workaround in this PR becuase utils/fix_data_dir.sh is a part of kaldi:

# fix_data_dir.sh leaves only utts which exist in all files
if [ -e "${data_feats}/org/${dset}/utt2sid" ]; then
    utils/fix_data_dir.sh --utt_extra_files utt2sid "${data_feats}/${dset}"
fi
if [ -e "${data_feats}/org/${dset}/utt2lid" ]; then
    utils/fix_data_dir.sh --utt_extra_files utt2lid "${data_feats}/${dset}"
fi

[codecov]

Codecov Report

Merging #4609 (4cb824c) into master (f0edec2) will increase coverage by 0.00%.
The diff coverage is n/a.

❗ Current head 4cb824c differs from pull request most recent head dfed926. Consider uploading reports for the commit dfed926 to get more accurate results

@@           Coverage Diff            @@
##           master    #4609    +/-   ##
========================================
  Coverage   83.06%   83.07%            
========================================
  Files         508      508            
  Lines       43646    43775   +129     
========================================
+ Hits        36253    36364   +111     
- Misses       7393     7411    +18     

Flag	Coverage Δ
test_integration_espnet1	66.36% <ø> (ø)
test_integration_espnet2	49.53% <ø> (+<0.01%)	⬆️
test_python	70.61% <ø> (+<0.01%)	⬆️
test_utils	23.28% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
espnet2/enh/espnet_enh_s2t_model.py	81.72% <0.00%> (-0.64%)	⬇️
espnet2/tasks/abs_task.py	75.52% <0.00%> (+0.03%)	⬆️
espnet2/tasks/enh_s2t.py	96.66% <0.00%> (+0.05%)	⬆️
espnet2/bin/asr_inference.py	84.98% <0.00%> (+0.08%)	⬆️
espnet2/asr/espnet_model.py	81.38% <0.00%> (+0.27%)	⬆️
espnet2/enh/loss/wrappers/fixed_order.py	91.30% <0.00%> (+0.39%)	⬆️
espnet2/enh/espnet_model.py	86.56% <0.00%> (+1.08%)	⬆️
espnet2/asr/ctc.py	96.72% <0.00%> (+1.63%)	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[kan-bayashi]

Thank you for your report and fixing.
In your case, the fixing is not performed when there is no utt2lid or utt2sid.
So I changed the strategy. Could you check my suggestion?

[jonghwanhyeon]

No problem! Thanks 😄

[kan-bayashi]

Thank you for your kind contribution!