[Metal] Add pass for splitting kernel with huge number of args

[echuraev]

The Metal has some limitations on the number of input parameters. More
information can be found here:
https://developer.apple.com/documentation/metal/buffers/about_argument_buffers?language=objc

In this commit a new pass for splitting functions with big number of
arguments to smaller parts was added. In parameter max_function_args
we can specify the maximum number of kernel arguments for specific
target and then split kernel when the number of arguments exceeds the
value of max_function_args. Currently this pass works only for concat
layer.

Thanks for contributing to TVM! Please refer to guideline https://tvm.apache.org/docs/contribute/ for useful information and tips. After the pull request is submitted, please request code reviews from Reviewers by @ them in the pull request thread.

[jwfromm]

@mbrookhart can you take a look at this one?

[mbrookhart]

cc @masahi

I remember Masa doing something like this for Vulkan at one point, but I'm not sure if that was a branch or if it ever got merged. If it's merged somewhere, maybe we should combine make this a generally available tool?

[masahi]

Not exactly, but I've dealt with a similar issue. My mitigation was to limit the maximum fusion depth, which breaks large parameter kernels into smaller ones. But that is not guaranteed to work and not predictable. I can imagine that having a pass like this that allows more fine-grained controls might be necessary in some cases.

@echuraev FYI you can cap the fuse depth by

tvm/tests/python/relay/test_pass_fuse_ops.py

Line 755 in 720e7b1

with tvm.transform.PassContext(config={"relay.FuseOps.max_depth": max_fused_ops}):

[mbrookhart]

Minor nit.

In general, I see the utility for single Relay kernels that have too many inputs, but I wonder if you'll hit this more for kernels post-fusion. This doesn't seem to tackle that for anything other than giant concatenation?

[echuraev]

Not exactly, but I've dealt with a similar issue. My mitigation was to limit the maximum fusion depth, which breaks large parameter kernels into smaller ones. But that is not guaranteed to work and not predictable. I can imagine that having a pass like this that allows more fine-grained controls might be necessary in some cases.

@echuraev FYI you can cap the fuse depth by

tvm/tests/python/relay/test_pass_fuse_ops.py

Line 755 in 720e7b1

with tvm.transform.PassContext(config={"relay.FuseOps.max_depth": max_fused_ops}):

Thank you! I thought about the reducing fuse depth, but as you mentioned, it is not predictable and not guaranteed to work. This is why I think that this approach with splitting kernels is more robust.

[echuraev]

Minor nit.

In general, I see the utility for single Relay kernels that have too many inputs, but I wonder if you'll hit this more for kernels post-fusion. This doesn't seem to tackle that for anything other than giant concatenation?

Yes, now it is solving only problem with concat layer. I thought that maybe this pass can be useful in case of split layer for example. But I wasn't able to reproduce the same problem for a split layer on several simple tests.

[mbrookhart]

I have no problems merging this as it stands, but I do think I have a bigger question:

Should we put some sort of logic into fusion to automatically stop fusion if the argument list grows too large per this setting? That should be more robust than arbitrarily limiting the fusion depth. It could of course be a second PR.

[echuraev]

I have no problems merging this as it stands, but I do think I have a bigger question:

Should we put some sort of logic into fusion to automatically stop fusion if the argument list grows too large per this setting? That should be more robust than arbitrarily limiting the fusion depth. It could of course be a second PR.

It's a good point and looks reasonable to add such logic into the fusion algorithm. It could help us to avoid some possible problems with number of arguments in the future. I think it would be better to do such logic in separate PR, due to problem with concat layer can appear and without fusing. For example, in the original problem we had many inputs because for each input to concat we had some preprocessing. And the fusing algorithm wasn't able to fuse these inputs into tuple due to this preprocessing. I don't think that the limit on the fusion depth can solve this problem.