How to specify data directory

[xnchu]

I am loading OMNI data, and wish to set local_data_dir to D:/data/omni.
In pyspedas/omni/config.py, it seems that local _data_dir can be set using os.environ['OMNI_DATA_DIR']
However, the following code will still download OMNI data in the current directory instead of D:/data/omni.
This was because os.environ is not changed in config.py
The following code will reproduce the problem.
Thanks for your attention.

import pyspedas
import os

os.environ['OMNI_DATA_DIR'] = "C:/data/omni/"
omni_vars = pyspedas.omni.data(trange=['2013-11-5', '2013-11-6'])

[ericthewizard]

Hello!

The preferred way to set the environment variable is to set it in the operating system before opening Python, e.g., on Windows, in the command-line, use:

setx OMNI_DATA_DIR D:/data/omni/

Then restart your terminal and open python - your OMNI_DATA_DIR should be saved. You can also set it temporarily in the terminal using 'set' instead of setx with:

set OMNI_DATA_DIR=D:/data/omni/

Then open python. With 'set', you won't have to restart your terminal, but it won't persist through terminal sessions.

If you prefer to set the data directory in python before making the load routine call, you can update the OMNI CONFIG table directly with:

import pyspedas
pyspedas.omni.config.CONFIG['local_data_dir'] = 'D:/data/omni/'
omni_vars = pyspedas.omni.data(trange=['2013-11-5', '2013-11-6'])

Hope this helps!

[xnchu]

I've tested the two solutions and both worked. Thanks a lot.

[donglai96]

Hi,

I tried this way to download the data in my jupyter notebook

from pyspedas.utilities.download import download

files = download(remote_file=remote_names, remote_path=remote_data_dir,
                         local_path=local_data_dir, no_download=no_update)

I set local path to D disk

And I found it created a lot of temporary files in my C disk and I couldn't locate them. Really curious why that happened

[ericthewizard]

The download function stores the files as temporary files before copying to the local data directory - this is to avoid problems with partial downloads. e.g., if the internet connection drops before a download completes and you only get 75% of a CDF, the load routines would crash trying to load that invalid file.

I think the temporary files could be added to the local data directory instead of the temp directory (just with a temp file name instead of .cdf); I'll look into changing this when I get a chance.

[donglai96]

That's really helpful! Thank you!