Duplicated token seqs are used for rescoring

[csukuangfj]

While implementing rescoring with conformer lm, I find that there are duplicated token seqs.

The reason is that the following code

icefall/icefall/decode.py

Lines 222 to 237 in 810b193

	if isinstance(lattice.aux_labels, torch.Tensor):
	word_seq = k2.ragged.index(lattice.aux_labels, path)
	else:
	word_seq = lattice.aux_labels.index(path)
	word_seq = word_seq.remove_axis(word_seq.num_axes - 2)
	# Each utterance has `num_paths` paths but some of them transduces
	# to the same word sequence, so we need to remove repeated word
	# sequences within an utterance. After removing repeats, each utterance
	# contains different number of paths
	#
	# `new2old` is a 1-D torch.Tensor mapping from the output path index
	# to the input path index.
	_, _, new2old = word_seq.unique(
	need_num_repeats=False, need_new2old_indexes=True
	)

does not remove 0s from word_seq.

Previous versions remove 0s from word_seq, see

icefall/icefall/decode.py

Lines 218 to 227 in abadc71

	# word_seq is a k2.RaggedTensor sharing the same shape as `path`
	# but it contains word IDs. Note that it also contains 0s and -1s.
	# The last entry in each sublist is -1.
	if isinstance(lattice.aux_labels, torch.Tensor):
	word_seq = k2.ragged.index(lattice.aux_labels, path)
	else:
	word_seq = lattice.aux_labels.index(path, remove_axis=True)
	# Remove 0 (epsilon) and -1 from word_seq
	word_seq = word_seq.remove_values_leq(0)

It does not affect the final WER, but it incurs extra unncessary computations.

[csukuangfj]

One consequence after the fix is that we can use a larger value for num_paths, which could potentially affect the WER.