Creating data products for distribution

Each data product has a major, a minor and a revision qualifier. Please follow the naming conventions described in VALOR/SBN README.md to create new data products for distributions. If in doubt, drop a message in the VALOR Slack.

Once the data product name is established, following conventions, follow the next few steps

• Upload the new data product in one of the standard VALOR data resources.

  See here for a list of data resources.

  Your data product files (and all its sub-directories, if any) should go under under a DATA_RESOURCE_PATH/data/major/minor/revision sub-directory.

  For example, for the LIV1 data resource (see link above), the DATA_RESOURCE_PATH is /hepstore/neutrino1/data_archive/ and, therefore, the files of a data product with (major, minor, revision) qualifiers = (sbn_numu_disappearance, dsts, 2019b2) should go under /hepstore/neutrino1/data_archive/data/sbn_numu_disappearance/dsts/2019b2

• Generate checksums for the newly created (or updated) data product.

  Checksums can be generated using the valor_data-generate_checksums script.

  Example command:

  valor_data-generate_checksums --data-major sbn_nue --data-minor dsts --data-revis 2020a1 --archive-path /hepstore/neutrino1/data_archive/

  The output checksums are stored in DATA_RESOURCE_PATH/checksums/major_minor_revision.md5

• Add an entry for your new data product in the list of SBN data products.

• Add the analysis label in the last column of the above list of data products, provided this is an essential and mandatory data product (e.g. dsts, responses, prior_errors etc) for running a VALOR/SBN analysis. Not all data products are mandatory, and some products (e.g. universes) may be used only in the preparation of other, mandatory but reduced data products.

• Test fetching your new data product using the VALOR valor_data-fetch script.

• If an analysis label was added next, use the VALOR valor_data-fetch_set script, and confirm that your new data product is included in the list of those essential data products fetched when the analysis label is passed to the above script via the --data-set argument.