Fetcher downloads data every time it is called

[BramshQamar]

Every fetcher in fetcher.py downloads data every time it's called. Even when data is already in place.

for example:

from dipy.data.fetcher import fetch_scil_b0, read_siemens_scil_b0
fetch_scil_b0()

output:

Data size is approximately 9.2MB Downloading "datasets_multi-site_all_companies.zip" to /Users/bramshqamar/.dipy Download Progress: [##################################] 100.00% of 9.19 MBFiles successfully downloaded to /Users/bramshqamar/.dipy Out[1]: ({'datasets_multi-site_all_companies.zip': ('https://digital.lib.washington.edu/researchworks/bitstream/handle/1773/38479/datasets_multi-site_all_companies.zip', None)}, '/Users/bramshqamar/.dipy')

lets fetch again:
fetch_scil_b0()

output:

Data size is approximately 9.2MB Downloading "datasets_multi-site_all_companies.zip" to /Users/bramshqamar/.dipy Download Progress: [##################################] 100.00% of 9.19 MBFiles successfully downloaded to /Users/bramshqamar/.dipy Out[2]: ({'datasets_multi-site_all_companies.zip': ('https://digital.lib.washington.edu/researchworks/bitstream/handle/1773/38479/datasets_multi-site_all_companies.zip', None)}, '/Users/bramshqamar/.dipy')

This is the case with every fetcher.

[BramshQamar]

I am on Mac. I have Python 3.6.1, Numpy 1.12.1, and Nibabel 2.3.0.

[arokem]

Is the data on your hard-drive in between calls? Look in ~/.dipy

[skoudoro]

this data does not have md5, can it be the problem @arokem?

[arokem]

Might be. I can replicate this bug with the SCIL b0 dataset, but not with other datasets. @BramshQamar: Did you experience this also with other datasets?

I bet we need to change something around this line: https://github.com/nipy/dipy/blob/master/dipy/data/fetcher.py#L175

[skoudoro]

I bet we need to change something around this line:

or update the dataset. having a digital signature like md5 should be good practice, right? this permit to make sure that the dataset does not change.

What do you think?

[arokem]

Yes. Even better.

[skoudoro]

So, can you update it? I do not know how to access to https://digital.lib.washington.edu/

[arokem]

I don't think you need to access that website. Just add this line: #1643

[BramshQamar]

I was trying with some fetchers I wrote and fetch_scil_b0. I will add md5 in my fetchers and will test them.
Thank You @arokem @skoudoro

[skoudoro]

fix by #1643