Fix closing upon download completion in fetcher

[samuelstjean]

The fileobject was not close originally and the md5 was checking a stil opened object, still always raising an error message on the first download. Now the object is closed before checking for the md5 and should work as intended.

[arokem]

Since that same sequence repeats so many times, wouldn't you rather make a helper function out of that?

def _get_file_data(fname, opener):
    data = open(fname, 'wb')
    data.write(opener.read())
    data.close()

Or something along these lines

[MrBago]

How about something like:

from shutil import copyfileobj

def _get_file_data(fname, url):
    with openurl(url) as opener:
        with open(fname, 'wb') as data:
            copyfileobj(opener, data)

You got to watch out for a few things with something like this. You want to make sure that you close both opener and data, even when there is an error. The with statement will take care of that. Also write(file.read()) reads the whole thing into memory and then writes the whole thing in one go, that can be a problem with large files it's better to chunk it out. copyfileobj takes care of that.

[samuelstjean]

Seems like with openurl(url) as opener: is not valid. I get something like

//home/stjs2902/git/dipy/dipy/data/fetcher.py in _get_file_data(fname, url)
21
22 def _get_file_data(fname, url):
---> 23 with urlopen(url) as opener:
24 with open(fname, 'wb') as data:
25 copyfileobj(opener.read(), data)

AttributeError: addinfourl instance has no attribute 'exit'

Is the with statement essential? We could close it by hand, since it seems that it's missing something it doesn't like.

[MrBago]

Good catch, apparently urllib doesn't support the with statement directly until python 3, so we'll need to use contextlib.

http://stackoverflow.com/questions/1522636/should-i-call-close-after-urllib-urlopen