Zipping many parameters

[drewrotunno]

Hello, first, I am very happy to be using Rydiqule! I was trying to do something a little extra, and ran into perhaps a limitation of the system worth discussing, or perhaps someone can show me the work-around, or what I'm doing wrong. I wanted to look at all the Zeeman sub-levels in EIT, and effectively drive many coupling transitions over the same detuning scan, but with varying Rabi frequencies and decay branches on each edge depending on the states it drives. So I'm looking at something like 30 or 42 zipped detuning scans, and I can zip the parameters just fine, but in the raw Hamiltonian representation, is still a [dd.....*d, n * n] matrix.
Using test detuning scans of size 3, I run into issues just looking at sub-sets:
{ Unable to allocate 127. GiB for an array with shape (3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 22) and data type complex128 }

and if I make the whole thing, I see errors in get_hamiltonian()'s mesh_grid and other counting / einsum arrangement functions such as :
{ c:_______\Anaconda3\lib\site-packages\numpy\lib\function_base.py in (.0)
4976 s0 = (1,) * ndim
-> 4977 output = [np.asanyarray(x).reshape(s0[:i] + (-1,) + s0[i + 1:])
4978 for i, x in enumerate(xi)]
ValueError: maximum supported dimension for an ndarray is 32, found 42
}

I don't see this being an easy fix,, and I don't think I'm even asking for anyone to work on it, I'm just curious how one would go about it, give how the rest of the graph-to-hamiltonian pipeline works. A big fix may be to internally write or generate the Hamiltonians where they're already zipped and many parameters scan in the same axis, losing dimensions in the big matrix representation. Perhaps if it were possible to perform a 'hard zip' that re-writes the Hamiltonian dimension, before generating the hamiltonian of that size, that's what I might be hoping for? It seems the issue is when generating and passing this big dimension matrix.
I can also imagine doing a loop over this detuning value as an external work around. It's funny, the issue is that Rydiqule is too generous at incorporating arbitrary parameters, I quickly blew up the dimensionality with p->30!

Any advice on this would be appreciated. Thanks!

[dihm]

@drewrotunno We are of this issue, and have actually addressed it in version 1.1, in anticipation of more native support for the exact types of calculations you are attempting. Starting with v1.1, rydiqule no longer builds out the unzipped hamiltonian dimensions, instead is respects the zipped parameters from the start and only creates a single dimension for each zipped parameter set.

[drewrotunno]

Glad to hear! If I were more familiar with github, I would have noticed the big update a week before I posted this. I'll give 1.1 a try and see if that fixes my issue. Thank you!

[dihm]

No worries. This is a natural consequence of not having a proper mailing list
¯\_(ツ)_/¯