new data pull in & validate

[briankleinqiu]

Edited the data folder so that make data will pull in the raw ds005 the new func ds005, and the mni templates folder.

make validate should validate all of that with the new mni_hashlist i put in.

Also renamed mni_icbm152... folder itself to 'templates' and the mni_icbm152_t1... file we'll be using into 'mni_standard.nii' for convenience and clarity. mni_standard.nii is the one matthew mentioned as most useful for displaying the patterns on

This will prbly screw up travis ci somewhere but we'll see

[BenjaminHsieh]

yeah...the coverage dropped like crazy, not sure how to fix... can you play around with the coverage in makefile of project directory and/or add coverage of makefile of data directory and see how travis responds? If nothing works raise an issue and ask jarrod/ross

[changsiyao]

Are you generating a hash list every time you run data.py?
You should generate the hash list yourself and everybody should just compare their hash to the txt file you uploaded.
I don't see why you need the make_hash_list function.

[briankleinqiu]

The make_hash_list function I just left in, in case I wanted to make another hash list, so it's not actually being called. I git added a prior hash list that I had generated before to compare to, yeah. I'll mess with it around some more and see with the travis ci.

[changsiyao]

But it seems like that's messing up travis. I had to put the generate hash script in the tools folder to avoid travis problems, if you recall.

[briankleinqiu]

Commented that out just now, should improve at least a little now

[changsiyao]

What is the new_hashList.txt, what data is that for?

[briankleinqiu]

new_hashList.txt is the hashlist for ds005 that includes the mni_func data, so it actually makes the original hashlist redundant for now. but should keep the old hashlist in until we finalize things more

[changsiyao]

and mni_hashlist is for the templates?
Also, what does the templates do? I'm kinda confused.

[briankleinqiu]

Yup, mni_hashlist is for the templates.

Templates is just the folder:

http://nipy.bic.berkeley.edu/rcsds/mni_icbm152_nlin_asym_09c_2mm/

I renamed mni_icbm152_nlin_asym_09c_2mm to templates for clarity and convenience so we don't have to bother with that complicated name.

That's also why I renamed mni_icbm152_t1_tal_nlin_asym_09c_2mm.nii inside that folder to mni_standard.nii. This nii file is the standard brain template we should use to display functional patterns on, according to the Readme updated by Matthew.

Also we'll probably end up using the mask template in there, but haven't renamed that yet.

As for the travis ci, i think combining all the hashlists and doing only one check_hashes might help with the remaining -2.4%

[changsiyao]

Travis won't cover the main function, last time, we also ignored the non-coverage of the last few lines. So I suppose if we combine everything into 1 check hashes, it's going to help, but it might also be confusing to people reading our codes. I'm not so sure about if we should combine all the hash lists...

[BenjaminHsieh]

To avoid coverage on data.py entirely:

[BenjaminHsieh]

You can let coverage check only the functions inside by creating a new python script with only functions
Then go to the main project directory and change the make coverage line:
--cover-package=data/thenewfunction.py

[BenjaminHsieh]

How did this go?

[briankleinqiu]

Is there any other reason not to combine the hashlists? I'm not sure why it'd be confusing since the idea is to check if all the data pulled in matches our hashlist, and it shouldn't matter if there's three hashlists or one?

I combined mni_hash.txt and newhashlist.txt into total_hash.txt, so it checks for everything in ds005 and the mni templates folders. It looks like it'll pass the travis so is it okay if we go with this?

If so I'll remove all the extraneous hashlists and code and then merge the pull request

[BenjaminHsieh]

I think the main thing is to make sure the hashes are checked in the right order as they are generated, did you try downloading the data and validating it? if it works then this should be fine

[briankleinqiu]

Wait are you sure they need to be checked in a certain order? I did the make validate and that was fine.

[BenjaminHsieh]

well, i assumed the for loop in check_hashes requires a certain order, if download and validate work fine then we can go ahead and merge

[briankleinqiu]

Added a readme

[BenjaminHsieh]

lol next time probably should wait for travis