New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remarks about vcs.download_sample_data_files #143

Closed
jypeter opened this Issue Mar 10, 2017 · 7 comments

Comments

Projects
None yet
3 participants
@jypeter

jypeter commented Mar 10, 2017

It took me some time to find how to download the traditional test data files and I eventually found vcs.download_sample_data_files in one of the tutorials

Some remarks about it:

  • is it documented somewhere else than in the tutorials?
  • can somebody add a docstring to it? At least add a note that the path to the downloaded data is specified in vcs.sample_data
  • should there be a similar function in cdms2? Or maybe this function should be in cdms2 to avoid duplicate? Because anyway you will use cdms2 to open the data files
  • is there a way to make a conda package with the sample data, a bit like there is a conda package for the high resolution basemap data? What you get with: conda install --c conda-forge basemap-data-hires
  • I thought the following ways to easily install the test data would be equivalent, but the first one does not work
(cdatm14) [jypmce@ciclad-ng ~]$ python -m vcs -c 'vcs.download_sample_data_files()'
/data/jypmce/cdat/miniconda2/envs/cdatm14/bin/python: No module named vcs.__main__; 'vcs' is a package and cannot be directly executed

(cdatm14) [jypmce@ciclad-ng ~]$ python -c 'import vcs; vcs.download_sample_data_files()'
Downloading: sftlf_visus.nc in /data/jypmce/cdat/miniconda2/envs/cdatm14/share/uvcdat/sample_data/sftlf_visus.nc
Downloading: so_Omon_ACCESS1-0_historical_r1i1p1_185001-185412_2timesteps.nc in /data/jypmce/cdat/miniconda2/envs/cdatm14/share/uvcdat/sample_data/so_Omon_ACCESS1-0_historical_r1i1p1_185001-185412_2timesteps.nc
[...]

@doutriaux1 doutriaux1 self-assigned this Mar 13, 2017

@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Mar 13, 2017

@jypeter you're right it should probably be split between vcs and vcs. Funny thing is that cdms test suite is the one requiring the bigger datasets.
There's no way to have a conda with this. It would suck up all of our data storage.
I will also tweak it to specify where the data is downloaded, to avoid having one copy per conda envirnoment.

@jypeter

This comment has been minimized.

jypeter commented Mar 13, 2017

It seems there is already an option for specifying a custom path. Except that without a docstring, you can't know for sure what the parameter is for

Help on function download_sample_data_files in module vcs.utils:

download_sample_data_files(path=None)
(END)

It's a good thing to minimize storage, and in that case I suggest you tell people to install vcdat with Miniconda, rather than Anaconda. My figures show a reduction of 1.9G, if you install with Miniconda. Mostly because if you do a full install with Ananconda, then you duplicate everything when you install CDAT with conda-forge
https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:uvcdat:cdat_conda:cdat_2_8_0

@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Mar 13, 2017

good point. Although these days I would think most people already have some flavor of conda on their systems already.

@jypeter

This comment has been minimized.

jypeter commented Mar 13, 2017

I'm not so sure about most people already having conda. I did not, that's why I spent some time experimenting. The installation instructions on the uvcdat wiki rely a lot on people being already familiar with conda. I have put my own notes on the following page:
https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:uvcdat:conda_notes

@durack1

This comment has been minimized.

Member

durack1 commented Mar 13, 2017

@doutriaux1 @jypeter I would agree with the statement above, anaconda is not installed by default on any system that I know about, so I don't think making any assumptions about a user knowledge of conda, or that it's installed is reasonable.. Explicit start to end docs would be the best path forward

@jypeter

This comment has been minimized.

jypeter commented Mar 14, 2017

My own notes and install instructions are available below. Beware, it's pretty detailed (with lotta links)

https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:uvcdat:cdat_conda:cdat_2_8_0

My end-users on multiple servers should theoretically mostly have to read

https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:python:starting#uv-cdat

@doutriaux1 doutriaux1 modified the milestone: 3.0 May 5, 2017

@doutriaux1

This comment has been minimized.

Member

doutriaux1 commented Jun 1, 2017

test download splitted now

@doutriaux1 doutriaux1 closed this Jun 1, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment