Implement ArrayOfRagged

[LvHang]

Implement the ArrayOfRagged class.
The RowSplits()/RowIds() function returns the array of individual shapes.
The MetaRowSplits()/MetaRowIds() function reflects the relationship between the elements and the 'src-level' (each ragged) info.
I'm going to test it.

[csukuangfj]

There is an ongoing PR #926

[pkufool]

I only implemented RowSplits()/RowIds() that needed by rnnt decoding method.

[LvHang]

Oh, I wasn't aware of this before. I skimmed Kang's PR just now, and as Kang said, this one tried to implement all the functions. I will try to compare the codes and make them unified.

One quick question: the class should be named as "Array1OfRaggedShape(RaggedShape *src, int32_t num_srcs)" or "ArrayOfRaggedShape(RaggedShape *srcs, int32_t num_srcs)"?

[pkufool]

I think it is Array1OfRaggedShape.

[LvHang]

OK, I‘m confused as it seems not unified in Dan's comments and definitions.

[danpovey]

OK, I‘m confused as it seems not unified in Dan's comments and definitions.

likely just a typo, but I'm OK with keeping it as ArrayOfRaggedShape, since at the current time we only have one (and unlikely to create another with 2 axes).

[LvHang]

The 'bin/cu_array_of_ragged_test' has passed.
Thank Fangjun and Wei for answering tons of my questions!

In addition, I have a small question when I read the source code about "context", and how to use it in "ParallelRunner".
Assume, we have a GPU context with "StreamA" (i.e. a PytorchCudaContext c is initialized).
In the code, most of the time (except in "intersect_dense_pruned.cu"), it uses as ParallelRunner pr(c); With w(pr.NewStream());, which doesn't pass the actual parameter to function--NewStream().
But, without actual parameter or "num_work_items < 10k", it will return the default stream (i.e. 0x0)，the "g_stream_override_" will not change. So, in "K2_EVAL" [inside K2_EVAL->EvalDevice(c->GetCudaStream())], we still use "StreamA".
So, why do we try to use "ParallelRunner" with the default "NewStream()"?

[LvHang]

Similar to the change of "MetaRowSplits()", I also change the following statements:
Array2<int32_t> Offsets() --> const Array2<int32_t> &Offsets() const
Array1<int32_t*> MetaRowIds() --> Array1<const int32_t *> MetaRowIds()
Array1<int32_t> MetaRowIds(int32_t axis) --> const Array1<int32_t> &MetaRowIds(int32_t axis) const

Tests have passed.

[danpovey]

The 'bin/cu_array_of_ragged_test' has passed. Thank Fangjun and Wei for answering tons of my questions!

In addition, I have a small question when I read the source code about "context", and how to use it in "ParallelRunner". Assume, we have a GPU context with "StreamA" (i.e. a PytorchCudaContext c is initialized). In the code, most of the time (except in "intersect_dense_pruned.cu"), it uses as ParallelRunner pr(c); With w(pr.NewStream());, which doesn't pass the actual parameter to function--NewStream(). But, without actual parameter or "num_work_items < 10k", it will return the default stream (i.e. 0x0)，the "g_stream_override_" will not change. So, in "K2_EVAL" [inside K2_EVAL->EvalDevice(c->GetCudaStream())], we still use "StreamA". So, why do we try to use "ParallelRunner" with the default "NewStream()"?

The concept of ParallelRunner is that sometimes instead of dependencies a->b->c->d, we have things like a->b->d and a->c->d, so we can do b and c in parallel. CUDA does have the concept of a "stream" that conceptually allows you to take advantage of this situation; and of course in principle we can use CPU streams as well to deal with this more efficiently (in addition to any other parallelization that we might do across streams within K2_EVAL.. which is possible in principle, even though we are not doing it now).

In practice, it turned out that the overhead of creating and deleting streams in CUDA is way too high for it to make sense to use this mechanism. So in practice, the ParallelRunner thing is currently giving no benefit. It's possible, though, that we might in future run on different hardware where it would make a difference; or that we might devise a way to have the mechanism be useful.

[LvHang]

The concept of ParallelRunner is that sometimes instead of dependencies a->b->c->d, we have things like a->b->d and a->c->d, so we can do b and c in parallel. CUDA does have the concept of a "stream" that conceptually allows you to take advantage of this situation; and of course in principle we can use CPU streams as well to deal with this more efficiently (in addition to any other parallelization that we might do across streams within K2_EVAL.. which is possible in principle, even though we are not doing it now).

In practice, it turned out that the overhead of creating and deleting streams in CUDA is way too high for it to make sense to use this mechanism. So in practice, the ParallelRunner thing is currently giving no benefit. It's possible, though, that we might in future run on different hardware where it would make a difference; or that we might devise a way to have the mechanism be useful.

Thanks, Dan, Got it.
When I tried to know more about the concept of 'Context' of k2, I went through the source codes, so I encountered the above question. I think the ParallelRunner might be useful in the future.
The following words are left for other newbies like me who want to understand and know more about k2's 'Context'. I think the posters (https://medium.com/gpgpu/multi-gpu-programming-6768eeb42e2c; https://on-demand.gputechconf.com/gtc/2014/presentations/S4158-cuda-streams-best-practices-common-pitfalls.pdf, etc) will be helpful to know more about the parallelization of CUDA streams initially. When you read the source code, "default_context.cu" will show the basic conceptual implementation of k2's 'Context'. It's good to read. But, from the CMakeFile, you will know the 'pytorch_context' is actually used now.

[pkufool]

Would you please add some tests for Offsets, I think this is good to merge after fixing this. Thanks!

[LvHang]

The Offsets test code has been added and the bin/cu_array_of_ragged_test passed on my machine.

[LvHang]

The "linear_fsa_with_self_loops_test.py" seems to cause the style check fails.

[csukuangfj]

The "linear_fsa_with_self_loops_test.py" seems to cause the style check fails.

It should be fixed with the latest master.

[LvHang]

The "linear_fsa_with_self_loops_test.py" seems to cause the style check fails.

It should be fixed with the latest master.

It says the following line is too long (I checked the latest code, It's true):

k2/k2/python/tests/linear_fsa_with_self_loops_test.py

Line 58 in d977865

expected_labels = expected_labels0 + expected_labels1 + expected_labels2 # noqa

[csukuangfj]

The "linear_fsa_with_self_loops_test.py" seems to cause the style check fails.

It should be fixed with the latest master.

It says the following line is too long (I checked the latest code, It's true):

k2/k2/python/tests/linear_fsa_with_self_loops_test.py

Line 58 in d977865

expected_labels = expected_labels0 + expected_labels1 + expected_labels2 # noqa

Rebase on the latest master should solve your problem.

[LvHang]

The "linear_fsa_with_self_loops_test.py" seems to cause the style check fails.

It should be fixed with the latest master.

It says the following line is too long (I checked the latest code, It's true):

k2/k2/python/tests/linear_fsa_with_self_loops_test.py

Line 58 in d977865

expected_labels = expected_labels0 + expected_labels1 + expected_labels2 # noqa

Rebase on the latest master should solve your problem.

Thanks, I missed the "#noqa" in the latest version. Done.

[pkufool]

merging, thanks!