Machine Learning Module

[maxnoe]

At least for particle classification, energy regression and for the mono case also for direction reconstruction, we need to implement Components / Tools to train and apply machine learning models.

This is currently not implemented in ctapipe, but in several tools including protopipe.mva, aict-tools, lstchain.reco and probably more.

We should implement this in ctapipe, combining what we learned from those above, using the ctapipe configuration system.

A first step should probably focus on sklearn models on image parameters, a second step could implement an abstract API for training and applying deeplearning models, for which we could use a plugin system similar to IO so that for example the deep learning people can try new architectures / frameworks inside ctapipe.

Needed targets:

• Energy Regression
• Particle Classification
• Origin using the disp method
• Combination of mono predictions

Needed Tools:

• Training

Components:

• Application of each of the models / combination of mono predictions (to be used inside ShowerReconstructor and/or using tables.)

Most of this can probably be taken directly from aict-tools / protopipe.mva adapted to Components the config system.

[kosack]

To add to that list:

• gamma-like event quality (type) classification and/or reconstruction merging

[kosack]

this is related to #1744

[maxnoe]

Plan for things to be done until we meet in Annecy:

• Add train classifier tool (Similar to what is done in Draft: ML Energy Regressor Component/Tool #1877 for energy) (@nbiederbeck )
• Add component to calculate (weighted) mean prediction for stereo (@LukasNickel )
• Add support for reading machine learning predictions using TableLoader (@maxnoe)
• Add application of models to ctapipe-process (@maxnoe)
• Models per telescope type (tbd)
• ctapipe-ml-apply-models (rename apply energy, add other models) (tbd)
• Chunked reading for TableLoader? (@maxnoe)
• Handle units (in features and predictions) (@maxnoe)
• Performance plots
• Write out predictions from cross-validation?
• QualityQuery for training / application

[maxnoe]

Since most of this is now implemented, I think it makes sense to close this large issue and open smaller ones for the remaining tasks.