What is the best way to support 5d tensor (or even higher dimension)?

[randxie]

I am using ggml to implement a model that requires 5d tensor. When I read the ggml.h and ggml.c, it seems that there's no generic permute / transpose that can support arbitrary dimension tensor. What is the suggested way to deal with 5d tensor in ggml?

[slaren]

I am not sure that there is a good way to add a 5th dimension to the tensors. The reality is that lots of the ops assume 4 dimensions, and changing GGML_MAX_DIMS is not really possible without a significant refactoring. You can try to figure a way to merge two of the dimensions into the same one.

[randxie]

Hi @slaren, thanks for the insight! I had the same conclusion by reading the code. Is supporting arbitrary dimension on the roadmap? I understand that have a fixed shape is easy to reason about, but not as flexible if we want to support more model architecture.

FYI, the model I am experimenting with is nougat.

[slaren]

I don't think there are any specific plans to add support for more dimensions. Most operations only work with one to three dimensions, and the rest is treated as a batch, so you should be able to get the same results if you merge the last two dimensions into one.

[balisujohn]

@randxie what are the operations you need to perform on a 5d tensor? Maybe they could be factored into up to 4d operations on slices of a 5d tensor? Otherwise, I think ggml is probably not a viable choice because it seems really locked into its upper tensor dimension limit, probably would need to be majorly rewritten to accommodate arbitrary tensor dimensionality.

[balisujohn]

@slaren what are your thoughts on changing tensor max dims to a absurdly huge fixed number like 256, then leaving the ops current dimensionality requirements alone. I could be mistaken but I think this would give people the ability to extend specific ops functionality to meet their dimensionality needs? What would the refactoring cost be for a one time refactor to another fixed tensor dimension limit?

[slaren]

Ultimately adding more dimensions is not necessary unless there is an operator that requires them. Any additional dimensions are just treated as a batch size, and can always be consolidated into a single dimension. If there is an operation that needs more dimensions, I think it would be better to study that in a case by case basis. Adding a bunch of dimensions that will not work with most ops and will massively increase the size of the ggml_tensor struct is not really an option.