ENH: fast reads on large gzip files

[GaelVaroquaux]

Fixes #209

[coveralls]

Coverage increased (+0.01%) to 90.16% when pulling 46122c4 on GaelVaroquaux:fast_gzip_read into ff2b545 on nipy:master.

[coveralls]

Coverage remained the same when pulling dee72f8 on GaelVaroquaux:fast_gzip_read into ff2b545 on nipy:master.

[matthew-brett]

Gael - any progress?

[GaelVaroquaux]

Gael - any progress?

Man, oh man... It will be at least another 2 weeks with no activity.
Maybe a month :(

[matthew-brett]

Seriously? Do you want me to finish this up? I don't get to internet much
so that will also be a little slow.

On Mon, Dec 2, 2013 at 4:51 PM, Gael Varoquaux notifications@github.comwrote:

Gael - any progress?

Man, oh man... It will be at least another 2 weeks with no activity.
Maybe a month :(

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/210#issuecomment-29661440
.

[GaelVaroquaux]

On Mon, Dec 02, 2013 at 02:16:26PM -0800, Matthew Brett wrote:

Seriously? Do you want me to finish this up? I don't get to internet much
so that will also be a little slow.

As you wish. I don't mind working on it, but right now I have a lot of
people running after my and trying to get me to do things, so it will
take a little while.

G

[matthew-brett]

OK - I've done a PR to your branch. Can you review and merge? Then I'll merge with trunk.

[matthew-brett]

Gael - have you got a few minutes for this?

[larsoner]

@matthew-brett this one seems pretty straightforward... do you want to open a new PR with your branch that has your commit on top of Gael's? I'm happy to review before you merge if you want.

[satra]

@matthew-brett and @GaelVaroquaux - what's the status with this PR?

i'm running into this with some diffusion data. it takes about 30 mins to read the compressed file. however, in addition to the gzip issue, even uncompressed it takes 6 min to read 4G. this is reading from local disk /tmp on a node with 256G of ram.

In [1]: import nibabel as nb

In [2]: img = nb.load('/tmp/diff_preproc.nii')

In [3]: %time data = img.get_data()
CPU times: user 24.5 s, sys: 5min 58s, total: 6min 23s
Wall time: 6min 23s

$ mri_info /tmp/diff_preproc.nii 
Volume information for /tmp/diff_preproc.nii
          type: nii
    dimensions: 140 x 140 x 96 x 552
   voxel sizes: 1.5000, 1.5000, 1.5000
          type: FLOAT (3)

[satra]

scratch the above load times - something is wrong with that node.

[coveralls]

Coverage increased (+0.85%) when pulling 4c3572b on GaelVaroquaux:fast_gzip_read into ff2b545 on nipy:master.

[GaelVaroquaux]

@matthew-brett : I am sorry for going missing in action for so long. I am getting more and more committed with day-to-day management, and thus I have a hard time trying to contribute to projects that I don't know very well.

Anyhow, thanks for waiting for me. I feel honored!

[matthew-brett]

:)