Release v1 (#16)

* mv models from github to zenodo * update * add script option * usage * rename templates to add gp * update * rm * update with descriptions * update * tools readme * models readme * gammapy readme * paragraphs * split datasets and code use * update * rename typo * typo * increase log level * move env to mamba * activate mamba env * update readme * update function name as sag rel2 * mamba env * fix paths and clean unused * add tutorial notebook * update * rm irfs saves * fix path * update * rm prints * rm unused
ambra-dipiano · May 7, 2024 · 11b72e3 · 11b72e3
1 parent 6752c34
commit 11b72e3
Show file tree

Hide file tree

Showing 53 changed files with 1,698 additions and 1,854 deletions.
diff --git a/.gitignore b/.gitignore
@@ -4,13 +4,20 @@ logs
 img
 conf
 checkpoints
+crta_models
+cnn_cleaner
+cnn_regressor
+slurms
+tutorial_data
 
 # files
 *.out
 *.sh
 *.ll
+*.zip
 *regressor*1e4*z20*
-!conf/template*yml
+!/astroai/conf/template_cnn.yml
+!astroai/conf/template_gp.yml
 
 # Byte-compiled / optimized / DLL files
 __pycache__/

diff --git a/README.md b/README.md
@@ -14,19 +14,19 @@ cd astroAI
 
 ## Environment
 
-This software uses `venv` virtual environment. You are free to use a different solution but requirements compatibility is not guaranteed. 
+This software uses [mamba](https://mamba.readthedocs.io/en/latest/installation/mamba-installation.html) for managing environments. You are free to use a different solution though compatibility is not guaranteed. Once you have installed and activated `mamba` you can proceed with creating the environment.
 
 Step1: 
 ```bash 
-python -m venv astroai
+mamba create -n astroai
 ```
 Step2: 
 ```bash
-source astroai/bin/activate
+mamba activate astroai
 ```
 Step3:
 ```bash
-python -m pip install -r requirements.txt
+mamba install -f environment.yml
 ```
 
 ## Installation
@@ -43,26 +43,20 @@ Alternatively you may install in editable mode.
 pip install -e .
 ```
 
-## Configuration file
-How to configure the software.
+## Datasets
 
-## Preprocessing pipeline
-A single pipeline to manage preprocessing for all different tools.
+To run this code you are required to provide DL3 simulations in a compatible FITS format:
+- simulation with gammapy
+- simulation with ctools
+- provided sample datasets [simulations_dataset_v1](https://zenodo.org/)
 
-## Models
-An ensamble of deep learning models, with different architctures and goals.
+## Using this code
 
-### CNN-based models
-A collection of CNN-based models providing a number of different tools for high level analysis.
+- You can preprocess data using the instructions in the dedicated [README](./astroai/tools/README.md).
+- You can train models anew following the instructions in the dedicated [README](./astroai/models/README.md) or use the provided [cnn_models_v1](https://zenodo.org/).
+- You can compare the resuts with the reference real-time analysis pipeline following the instructions in the dedicated [README](./astroai/pipes/README.md).
 
-#### Binary classifier
-This model aims to classifying counts maps containing only background signal (class 0) and counts maps containing source and background signal.
+## Configurations
 
-#### Image cleaner
-This model aims to remove background noise from a counts map.
-
-#### Hotspots regressor
-This model aims to localise the coordinates of hotspots in a counts maps, whereas the hotspots have a given minimum Gaussian significance.
-
-#### Background classifier
-This model aims to classify various levels of background in a counts map, in order to associate the most appropriate Instrument Response Function.
+- A commented configuration template for preprocessing and model training is provided here: [template_cnn.yml](./astroai/conf/template_cnn.yml).
+- A commented configuration template to execute the reference rtal-time analysis pipeline is provided here: [template_gp.yml](./astroai/conf/template_gp.yml).
diff --git a/astroai/conf/template.yml → astroai/conf/template_cnn.yml b/astroai/conf/template.yml → astroai/conf/template_cnn.yml
diff --git a/astroai/conf/template_gp.yml b/astroai/conf/template_gp.yml
@@ -0,0 +1,62 @@
+# GENERIC CONFIGURATION
+samples:                           # number of samples
+start_seed:                        # starting seed of sample
+
+# SIMULATIONS PARAMETERS
+simulation:                        
+  directory:                       # path to data directory
+  datfile:                         # name of infotable in data directory
+  id:                              # empty to process full dataset - individual observation id to process single dl3
+  caldb:                           # name of calibration database production
+  irf:                             # name of single IRF file or 'random'
+  caldb_path:                      # path to calibration database folder "caldb"
+  point_ra:                        # empty to process full dataset - individual RA of observation pointing to process single dl3
+  point_dec:                       # empty to process full dataset - individual DEC of observation pointing to process single dl3
+  skyframeref:                     # observation sky frame, typically icrs or fk5
+  skyframeunitref:                 # observation sky frame unit, typically deg
+  timesys:                         # observation time system, typically tt
+  timeref:                         # observation time reference, typically 51544.5
+  timeunit:                        # observation time unit, typically s
+
+# DATA SELECTION OPTIONS
+selection:
+  emin:                            # minimum energy, tipycally in TeV
+  emax:                            # maximum energy, typically in TeV
+  energy_bins_per_decade:          # bins per energy decade
+  roi_ringrad:                     # radius of ROI, typically in TeV
+  tmin:                            # minimum time, tipycally in seconds
+  tmax:                            # maximum time, tipycally in seconds
+
+# EXECUTION OPTIONS
+execute:
+  pixel_size:                      # pixel size, typically in deg
+  exclusion:                       # path to regions exlusion file
+  outdir:                          # path to output directory
+  reducedirfdir:                   # path to reduced IRF (to read or create)
+  savefits:                        # option to save fits products (yes/no)
+  plotfullfov:                     # option to plot full FOV plots (yes/no)
+  plotirfs:                        # option to plot IRF (yes/no)
+  plotts:                          # option to plot TS map (yes/no)
+  makemap:                         # option to compute maps (yes/no)
+  mapreg:                          # option to save regions file (yes/no)
+  maproi:                          # map region radius, typically in deg
+  computeph:                       # option to perform aperture photometry (yes/no)
+  blindsearch:                     # option to perform blind search (yes/no)
+
+# LOGGING OPTIONS
+logging:
+  loglevel:                        # logging level: debug, info, warning, error, critical
+  logfile:                         # name of file for saving logs
+
+# PHOTOMETRY PARAMETERS
+photometry:
+  onoff_radius:                    # radius of regions, typically in deg
+  onoff_method:                    # off regions method, typically reflection
+
+# BLINDSEARCH PARAMETERS
+blindsearch:
+  sigmathresh:                     # sigma acceptance threshold
+  maxoffset:                       # maximum acceptable offset, tyically in deg
+  maxsrc:                          # maximum number of candidates
+  plotzoom:                        # option to plot a candidate zoom-in (yes/no)
+  corr_rad:                        # correlation radius
diff --git a/astroai/models/README.md b/astroai/models/README.md
@@ -0,0 +1,20 @@
+# Model training
+
+After you have compiled the configuration file following the commentary in [template_cnn.yml](../conf/template_cnn.yml) an preprocess your data following the [README](../tools/REDME.md) you can train your models.
+
+```bash
+python cnn.py -f ../conf/your_conf_file.yml
+```
+
+Alternatively you can submit a job to [Slurm](https://slurm.schedmd.com/documentation.html) if available to you.
+
+```bash
+python slurmjobs.py -f ../conf/your_conf_file.yml  -a cnn -m clean
+```
+
+or 
+
+```bash
+python slurmjobs.py -f ../conf/your_conf_file.yml -a cnn -m localise
+```
+
diff --git a/astroai/models/cnn.py b/astroai/models/cnn.py
@@ -10,7 +10,7 @@
 import tensorflow as tf
 from os.path import join, isfile
 from astroai.tools.utils import *
-TF_CPP_MIN_LOG_LEVEL="1"
+TF_CPP_MIN_LOG_LEVEL="3"
 
 # ---- CREATE MODELS -----
 

diff --git a/astroai/models/cnn_cleaner/cleaner_5sgm_filter12_z20.keras b/astroai/models/cnn_cleaner/cleaner_5sgm_filter12_z20.keras
diff --git a/astroai/models/cnn_cleaner/cleaner_5sgm_filter12_z20_history.npy b/astroai/models/cnn_cleaner/cleaner_5sgm_filter12_z20_history.npy
diff --git a/astroai/models/cnn_cleaner/cleaner_5sgm_filter12_zALL.keras b/astroai/models/cnn_cleaner/cleaner_5sgm_filter12_zALL.keras
diff --git a/astroai/models/cnn_cleaner/cleaner_5sgm_filter12_zALL_history.npy b/astroai/models/cnn_cleaner/cleaner_5sgm_filter12_zALL_history.npy
diff --git a/astroai/models/cnn_cleaner/cleaner_5sgm_filter2_zALL.keras b/astroai/models/cnn_cleaner/cleaner_5sgm_filter2_zALL.keras
diff --git a/astroai/models/cnn_cleaner/cleaner_5sgm_filter2_zALL_history.npy b/astroai/models/cnn_cleaner/cleaner_5sgm_filter2_zALL_history.npy
diff --git a/astroai/models/cnn_cleaner/cleaner_5sgm_filter6_zALL.keras b/astroai/models/cnn_cleaner/cleaner_5sgm_filter6_zALL.keras
diff --git a/astroai/models/cnn_cleaner/cleaner_5sgm_filter6_zALL_history.npy b/astroai/models/cnn_cleaner/cleaner_5sgm_filter6_zALL_history.npy
diff --git a/astroai/models/cnn_cleaner/cleaner_NOsgm_filter12_zALL.keras b/astroai/models/cnn_cleaner/cleaner_NOsgm_filter12_zALL.keras
diff --git a/astroai/models/cnn_cleaner/cleaner_NOsgm_filter12_zALL_history.npy b/astroai/models/cnn_cleaner/cleaner_NOsgm_filter12_zALL_history.npy
diff --git a/astroai/models/cnn_regressor/regressor_5sgm_filter2_2convs_dense10.keras b/astroai/models/cnn_regressor/regressor_5sgm_filter2_2convs_dense10.keras
diff --git a/astroai/models/cnn_regressor/regressor_5sgm_filter2_2convs_dense10_CLEAN.keras b/astroai/models/cnn_regressor/regressor_5sgm_filter2_2convs_dense10_CLEAN.keras
diff --git a/astroai/models/cnn_regressor/regressor_5sgm_filter2_2convs_dense10_CLEAN_history.npy b/astroai/models/cnn_regressor/regressor_5sgm_filter2_2convs_dense10_CLEAN_history.npy
diff --git a/astroai/models/cnn_regressor/regressor_5sgm_filter2_2convs_dense10_history.npy b/astroai/models/cnn_regressor/regressor_5sgm_filter2_2convs_dense10_history.npy
diff --git a/astroai/models/cnn_regressor/regressor_5sgm_filter2_4convs_dense10.keras b/astroai/models/cnn_regressor/regressor_5sgm_filter2_4convs_dense10.keras
diff --git a/astroai/models/cnn_regressor/regressor_5sgm_filter2_4convs_dense10_CLEAN.keras b/astroai/models/cnn_regressor/regressor_5sgm_filter2_4convs_dense10_CLEAN.keras
diff --git a/astroai/models/cnn_regressor/regressor_5sgm_filter2_4convs_dense10_CLEAN_history.npy b/astroai/models/cnn_regressor/regressor_5sgm_filter2_4convs_dense10_CLEAN_history.npy
diff --git a/astroai/models/cnn_regressor/regressor_5sgm_filter2_4convs_dense10_history.npy b/astroai/models/cnn_regressor/regressor_5sgm_filter2_4convs_dense10_history.npy
diff --git a/astroai/models/slurmjobs.py b/astroai/models/slurmjobs.py
@@ -19,7 +19,7 @@ def main(architecture, filename, mode):
     sh_outname = join(dirname(abspath(__file__)), 'slurms', f'{job_name}.sh')
     with open(sh_outname, 'w+') as f:
         f. write("#!/bin/bash\n")
-        f.write(f"\nsource {join(expandvars('$HOME'), 'venvs/astroai/bin/activate')}")
+        f.write(f"\mamba activate astroai")
         f.write(f"\n\tpython {join(dirname(abspath(__file__)), architecture)}.py -f {filename}\n")
 
     # write job