Where do the files downloaded from ESGF go?

[rswamina]

I need some help understanding how to set the path for downloads from ESGF. I have the following set in various configuration files:

My conf.yaml file in the intake-esgf directory has the following in it but the data gets downloaded to ~/.esgf regardless. I have set the esg_dataroot and local_cache to a group work space on JASMIN but that doesn't see m enough? I cannot see where it defaults to my home directory location. Not sure what I should do to fix this. @bouweandela or @valeriupredoi - do you have a quick answer perhaps?

additional_df_cols: []
break_on_error: true
confirm_download: false
download_db: ~/.config/intake-esgf/download.db
esg_dataroot:
- /gws/ssde/j25b/terrafirma/ranjinis/climate_data/
- /p/css03/esgf_publish
- /eagle/projects/ESGF2/esg_dataroot
- /global/cfs/projectdirs/m3522/cmip6/
- /glade/campaign/collections/cmip.mirror
globus_indices:
  ESGF2-US-1.5-Catalog: true
  anl-dev: false
  ornl-dev: false
local_cache:
- /gws/ssde/j25b/terrafirma/ranjinis/climate_data/
logfile: ~/.config/intake-esgf/esgf.log
num_threads: 6
print_log_on_error: false
requests_cache:
  cache_name: intake-esgf/requests-cache.sqlite
  expire_after: 3600
  use_cache_dir: true
slow_download_threshold: 0.5
solr_indices:
  esg-dn1.nsc.liu.se: false
  esgf-data.dkrz.de: false
  esgf-node.ipsl.upmc.fr: false
  esgf-node.llnl.gov: false
  esgf-node.ornl.gov: false
  esgf.ceda.ac.uk: false
  esgf.nci.org.au: false
stac_indices:
  api.stac.ceda.ac.uk: false

[bouweandela]

Last time someone was struggling with this, they used the wrong path for the configuration file. Can you run the command

python -c 'import intake_esgf; print("Settings are stored here:", intake_esgf.conf.filename); print("Data will be stored here:", intake_esgf.conf["local_cache"][0])'

and check that this is what you expect?

[valeriupredoi]

well it looks at its default location, if you set it via Python or runtime variable then it's gonna pick that up

[rswamina]

Ok. If I understand correctly then, ~/.config/intake-esgf is the default location whether I use my locally installed environment or the default JASMIN module. How do I set this at run time or with an environment variable to a different location? Thanks!

[valeriupredoi]

this way #4501 (reply in thread)

[rswamina]

Ok..that's in the Python script then?

[bouweandela]

To avoid downloading the same data multiple times, we use the default intake-esgf configuration file location so intake-esgf always uses the same configuration with the same paths, regardless of whether you're using it standalone or from ESMValTool and regardless of where it is installed.

[valeriupredoi]

As @bouweandela says and to change it:

import intake_esgf

intake_esgf.conf.set(
    local_cache="/path/to/your/custom/directory"
)

There is also a runtime variable associated with it, but better to have a scripted change so you don't forget where you changed it via ariable setting.

[bouweandela]

You'd need to save it too:

import intake_esgf

intake_esgf.conf.set(
    local_cache="/path/to/your/custom/directory"
)
intake_esgf.conf.save()

[valeriupredoi]

ah good call! Forgot about save 😁

[rswamina]

I am not running it inside ipynb. I have my default config location as follows:

~/.config/esmvaltool/intake-esgf/conf.yaml

[rswamina]

I see my mistake...I seem to have two different intake-esgf files. Let me fix that and see if it works.

[valeriupredoi]

also if you are using JASMIN central installation of esmvaltool, you may not be able to reset that location bc you don't have permissions to edit the configuration