Large Multi-Dimensional Arrays how to create entirely on disk?

[handloomweaver]

I successfully created a trillion element bcolz array on disk. Fantastic. The compression and the disk-based ness makes a 2 terabyte array 13.7GB and on disk. But problem is, I want a multi-directional array and I can't reshape it to (1000000, 1000000) using .reshape because it does it all in RAM. non-starter. Then I tried to create the shape from the start using .zeros((1000000, 1000000)) or .ones and again it just does it in memory. Is there a way to create? If not can you change this behavior

[FrancescAlted]

Yes, a reshape is not efficiently implemented, and due to how the carray
object is implemented, this requires a copy anyway. Of course, the copy
could be done out-of-core for disk-based carrays, but not there yet. As
always, Pull Requests are welcome indeed.

2014-09-18 16:31 GMT+02:00 handloomweaver notifications@github.com:

I successfully created a trillion element bcolz array on disk. Fantastic.
The compression and the disk-based ness makes a 2 terabyte array 13.7GB and
on disk. But problem is, I want a multi-directional array and I can't
reshape it to (1000000, 1000000) using .reshape because it does it all in
RAM. non-starter. Then I tried to create the shape from the start using
.zeros((1000000, 1000000)) or .ones and again it just does it in memory. Is
there a way to create? If not can you change this behavior

—
Reply to this email directly or view it on GitHub
#55.

Francesc Alted