This repository contains the products of the various HELP Data Management Units (DMU). The Github repository contains everything except the actual data, while its mirror on HeDaM contains everything. The reason not to store the data files in Github is that we have such huge files, that git is not designed for working with them. The directory structures on HeDaM and GitHub are identical. If you plan to use HELP data extensively, we recommend cloning the GitHub repository and downloading data to the corresponding data folders on your computer so that relative links continue to work properly. As well as accessing the binary data files on HeDaM:
it is also possible to query the data from the Virtual Observatory database:
The repository is organised as follows:
- Each DMU (see list below) has its own directory (e.g. DMU24/ contains photometric redshifts).
- Each DMU product (e.g. photometric redshifts in the XMM-LSS field) lives in a separate sub-directory. These contain
readme.mdfile describing the data product. If you want to keep your data in the clone of this repository on your computer, you can put it in a
data/folder inside the DMU product directory. These folders are automatically ignored by git.
Here is the list of the DMU numbers with their associated responsibility. Some of these involve ongoing work and are not currently on GitHub. dmu32 contains the main HELP data product which joins together data from dmu0 (optical surveys), dmu1 (masterlist), dmu24 (photometric redshifts), dmu26 (XID+ fluxes), dmu28 (CIGALE SED fits).
|3||Morphologies (Shapes & Sizes) of Objects|
|4||Bright Star Mask|
|5||Known Star Flag|
|6||Optical photometry validation|
|8||Radio data - LOFAR & FIRST/NVSS/TGSS|
|9||Radio data - JVLA-DEEP & GMRT-DEEP|
|11||Cross matching MIPS/PACS/SPIRE|
|12||Cross Matching LOFAR & FIRST/NVSS/TGSS|
|13||Cross Matching JVLA-DEEP & GMRT-DEEP|
|20||MIPS blind photometry|
|21||PACS blind photometry|
|22||SPIRE blind photometry|
|27||Empirical models / templates|
|28||SED fitting / CIGALE|
|29||Radiative transfer models|
|30||Missing (supplementary) Sources|
Running the code
Most of the code used to generate and manipulate data here is run through Jupyter notebooks and makes extensive use of the Python package
herschelhelp_internal written for the purpose. Installation instructions are available on the GitHub download page:
If you don't wish to rerun or adapt the code here but just to use the data, you may be interested in the Python package
herschelhelp_python. We anticipate most users of HELP data using this package.
Procedure to add a new DMU data product
Here is the procedure to add a new DMU data product:
- Create a
- Create a
readme.mdfile describing the product. Indicate what pristine data was used and all the processes that were applied to it.
- You may add some scripts in the folder or create another DMU product containing the programme that you are using in various products.
- Add a
meta_main.ymland one or more
meta_survey.yml(if there are several, name them
meta_survey_1.ymland so on) taken from the
TEMPLATESdirectory. These files are important as they allow us to trace and acknowledge the data we use.
- Don't put the actual data on Github, instead explain how to get it in
get_data.mdfile, or contact Yannick.
When this is done, Yannick will:
- Synchronise the DMU product repository on HeDaM and add the actual data in the
- Add a
changelog.mdfile describing the history of the data product.
- Add a
dmu2_validationreport with technical checks to the data.
Procedure for validators
When you have to validate a product, you can download it from
HeDaM. Then you need to add
validation_report.md file inside the product folder and sync it back to
Github. You can also update the
For a full description of the database and how it is organised in to
dmu_products please see the top level readme.
The Herschel Extragalactic Legacy Project, (HELP), is a European Commission Research Executive Agency funded project under the SP1-Cooperation, Collaborative project, Small or medium-scale focused research project, FP7-SPACE-2013-1 scheme, Grant Agreement Number 607254.