Cannot save very large sparse matrix

[felixgao]

I have a very large matrix the full matrix size 35390104 x 28005, currently I am saving them as two small part files, if I tried to combine them and save them as one large sparse matrix I got the following error. The total file size is around 15GB.

Traceback (most recent call last):
File "/home/felix/test.py", line 58, in
main(sys.argv[1:])
File "/home/felix/test.py", line 49, in main
save_npz(matrix, "/tmp/combined_matrix.npz")
File "/home/felix/test.py", line 22, in save_npz
np.savez(output, row=row, col=col, data=data, shape=shape)
File "/usr/local/lib/python2.7/dist-packages/numpy/lib/npyio.py", line 530, in savez
_savez(file, args, kwds, False)
File "/usr/local/lib/python2.7/dist-packages/numpy/lib/npyio.py", line 592, in _savez
zip.write(tmpfile, arcname=fname)
File "/usr/lib/python2.7/zipfile.py", line 1102, in write
zinfo.file_size))
struct.error: 'L' format requires 0 <= number <= 4294967295

I am using 1.8.0 version of numpy, is there a work around of the zipfile limitation?

[abalkin]

What version of python are you using? Try 2.7.6 if yours is older.

[pv]

Seems to be this Python bug: http://bugs.python.org/issue9720

[felixgao]

Using python 2.7.3, don't have access to upgrade it. I will see if I can talk to the IT to get it upgraded. Thanks.