Refactor simulation tools configuration

[GernotMaier]

Simplifies the configuration of simulation tools.

This is a major refactoring PR touching almost all modules relevant for simulation configuration and production. This addresses especially simulate_prod.

Main changes are:

• removal of configuration by yaml file (as those listed in data/parameters); replaced by configuration via command line (or using configuration files for all command line parameters). There are two parameter files left (camera-efficiency_parameters.yml, corsika_parameters.yml), which will be removed in upcoming PRs
• re-organiation of all 'runners': there is now a dedicated runner directory to collect all of them
• introduce module runner_services to collect common functionality
• removed a lot of obsolete code, unused code; simplified many models (e.g., instead replaced get_output_file, get_log_file, get_histogram_file, etc by get_file(file_type="output").
• removed application simulate_showers_for_trigger_rates - this is a duplication of simulate_prod.
• removed a couple of debug statements which were part of loops (to many output statements makes it hard to read)
• trying to replace module calls like _check_run_result(self, **kwargs) by def _check_run_result(self, run_number=None): - this makes it much clearer
• rigorous testing of all new functions / modules

Closes #982
Closes #792 (configuration is now completely different and this does not apply anymore)
Closes #911 (configuration files are now handled through the command line and argparser; no metaschema required)

[tobiaskleiner]

All the removed arguments still appear in the docstring above. Why did you remove the config.parser.add_arguments?

[GernotMaier]

No, they are moved to the command_line module, as we use this type of parameters for several applications. So the docstring is still fine (as e.g., you want to configure simulate_prod with azimuth or number of runs).

[tobiaskleiner]

In the docstring the parameters are under command_line_parameters. Maybe you can make clear that the values come from the config file.

[GernotMaier]

This is correct, they are command line parameters:

• any argument defined through Configurator can be set on the command line, by environmental variable or using the --config option and given them all in a configuration file.

This allows us to have let the user decide to configure tools with super long command lines or using a configuration tool.

[tobiaskleiner]

I would say they are converted into command line parameters if they are passed via a configuration file? Maybe you can add this sentence at the beginning somewhere.

[GernotMaier]

This has been changed, as discussed in the chat.

[tobiaskleiner]

Thanks @GernotMaier , see few comments.

[GernotMaier]

@tobiaskleiner - thanks a lot for the review. I am waiting for one clarification from your side (on the run numbers / divergent pointing issue). Let me know if there is anything else.

[ctao-dpps-sonarqube]

Analysis Details

0 Issues

• 0 Bugs
• 0 Vulnerabilities
• 0 Code Smells

Coverage and Duplications

• 96.10% Coverage (89.00% Estimated after merge)
• 0.00% Duplicated Code (0.00% Estimated after merge)

Project ID: gammasim_simtools_AY_ssha9WiFxsX-2oy_w

View in SonarQube

[GernotMaier]

Thanks again @tobiaskleiner . Let me know if there is anything else.

[tobiaskleiner]

All good now. Thanks again @GernotMaier.