Implement Rnnt decoding

[pkufool]

I have figured out all the implementation details of #905.

The decoding method is supposed to be used as follows:

std::vector<std::shared_ptr<RnntDecodingStream>> streams_ptr;
// encoder_out shape [batch][t][dim]
Tensor encoder_out = model.enoder(input);
for (int32_t i = 0; i < batch; ++i) {
  // decoding_streams contains all the RnntDecodingStream of alive waves.
  streams_ptr.push_back(decoding_streams[wav_id[i]]);
}
auto  streams = RnntDecodingStreams(streams_ptr, config);
for (int32_t i = 0; i < t; ++i) {
  // shape has a shape of [stream][context]
  RaggedShape shape;
  Array2<int32_t> contexts;
  streams.GetContext(&shape, &contexts);
  expand_encoder_out = encoder_out[:, i, :].index_select(0,  shape.RowIds(1));
  decoder_out = model.decoder(contexts);
  auto log_probs = model.joiner(expand_encoder_out + decoder_out);
  streams.Advance(log_probs);
}
streams.Detach();
FsaVec ofsa;
Array1<int32_t> out_map;
// num_frames contains the number of frames we already decoded for each stream.
streams.FormatOutput(num_frames, &ofsa, &out_map);

[danpovey]

wow, great work!!!

[pkufool]

We can merge v2.0-pre branch into master at some time, so we can integrate the torch-scirpt thing easily. @danpovey @csukuangfj What do you think?

[csukuangfj]

I think we can merge master to v2.0-pre.

[danpovey]

Either way is OK with me, I guess @csukuangfj can decide.

[danpovey]

It looks good! Sorry, I don't have time to do a super-detailed review.
My only thought is, is there any way to break up Advance() into smaller pieces
that could be documented independently, to make it clearer?
I know this is the way I drafted it, but I think it's going to be hard for others to understand in future.

[pkufool]

OK, I will try to break it into some stages.

[pkufool]

Add python wrapper, will try it on icefall today...

[danpovey]

A few more cosmetic comments.
Perhaps we could merge after resolving these?
I have a couple suggestions of renaming things, which may require small changes in the Icefall PR k2-fsa/icefall#250.

[danpovey]

misspelling: scores.
also appended->append.

[danpovey]

Detach the RnntDecodingStreams -> Detach the RnntDecodingStreams object.

(This may be my fault, but..) I am concerned that the name Detach() may be too confusable with .detach() in PyTorch. Perhaps could call it TerminateAndFlushToStreams() ?

[danpovey]

Instead of (we don't actually do that, but think about it) , could say: Before unstack, we will (conceptually) pad...

[danpovey]

... also, it's not very standard in C++ to use strings as flags. I think having a boolean "pad_right" might be more conventional.

[danpovey]

and->or

[csukuangfj]

Suggested change

	logprobs.Dim0() == states.TotSize(1).
	logprobs.Dim0() == states_.TotSize(1).

[csukuangfj]

Suggested change

	Advance decoding streams by one frame. Args:
	Advance decoding streams by one frame.

[csukuangfj]

Is this line necesseary?
I think context_state is always less than pow(vocab_size, exp).

[csukuangfj]

It is not needed when col is 0.

[csukuangfj]

Will this kind of encoding cause overflow when vocab_size is large?
For instance, when vocab_size is 5000 and context_size is 4, it will lead to overflow
since 5000^3 cannot be represented by 2^31.

[pkufool]

We always use int64_t for context_state and state_value, I think.

[csukuangfj]

In that case, it overflows when context size is 7.

[danpovey]

Yes, we can just make that a limitation/requirement. Right now we plan on using context_size=1 or 2 anyway.

[csukuangfj]

Doesn't 0 mean blank here?

[csukuangfj]

Should its type be double? I find that scores of ragged arrays in this file have double elements.

By the way, it will not increase memory usage if you order them in the following way:

double score;
int32_t graph_arc_idx01;
int32_t dest_state;

[csukuangfj]

This is an important assumption. We should document it explicitly, I think.

[csukuangfj]

Looks like num_context_states is used nowhere.
Shall we delete it?

[danpovey]

.. could comment it out perhaps. The concept may be useful, and it may be referred to in other docs.