Fix joint tokenization in st.sh

[pyf98]

Hi, in st.sh, the source and target texts should be merged before tokenization, if token_joint is True.

This modification is based on the same stage in mt.sh. Please check if it's correct.

[ftshijt]

Looks cool to me, I will merge it after CI passed. Many thanks!

[codecov]

Codecov Report

Merging #4143 (e2489b1) into master (6f42960) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #4143   +/-   ##
=======================================
  Coverage   80.43%   80.43%           
=======================================
  Files         442      442           
  Lines       38557    38557           
=======================================
  Hits        31015    31015           
  Misses       7542     7542           

Flag	Coverage Δ
test_integration_espnet1	67.13% <ø> (ø)
test_integration_espnet2	51.14% <ø> (ø)
test_python	66.51% <ø> (ø)
test_utils	24.45% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6f42960...e2489b1. Read the comment docs.

[siddalmia]

espnet/egs2/TEMPLATE/st1/st.sh

Lines 299 to 307 in 3ae0dcc

	if "${token_joint}"; then
	# if token_joint, the bpe training will use both src_lang and tgt_lang to train a single bpe model
	[ -z "${src_bpe_train_text}" ] && src_bpe_train_text="${data_feats}/${train_set}/text.${src_case}.${src_lang}"
	[ -z "${tgt_bpe_train_text}" ] && tgt_bpe_train_text="${data_feats}/${train_set}/text.${tgt_case}.${tgt_lang}"
	# Prepare data as text.${src_lang}_${tgt_lang})
	cat $src_bpe_train_text $tgt_bpe_train_text > ${data_feats}/${train_set}/text.${src_lang}_${tgt_lang}
	tgt_bpe_train_text="${data_feats}/${train_set}/text.${src_lang}_${tgt_lang}"
	else

This will no longer be needed.

[pyf98]

espnet/egs2/TEMPLATE/st1/st.sh

Lines 299 to 307 in 3ae0dcc

	if "${token_joint}"; then
	# if token_joint, the bpe training will use both src_lang and tgt_lang to train a single bpe model
	[ -z "${src_bpe_train_text}" ] && src_bpe_train_text="${data_feats}/${train_set}/text.${src_case}.${src_lang}"
	[ -z "${tgt_bpe_train_text}" ] && tgt_bpe_train_text="${data_feats}/${train_set}/text.${tgt_case}.${tgt_lang}"
	# Prepare data as text.${src_lang}_${tgt_lang})
	cat $src_bpe_train_text $tgt_bpe_train_text > ${data_feats}/${train_set}/text.${src_lang}_${tgt_lang}
	tgt_bpe_train_text="${data_feats}/${train_set}/text.${src_lang}_${tgt_lang}"
	else

This will no longer be needed.

Thanks for pointing it out. I removed these lines. Could you review it again?

[siddalmia]

Looks good!