Merge pull request #173 from Rykath/master

refactor run system API

.git-blame-ignore-revs

@@ -0,0 +1,2 @@
+# initial reformat of old code
+302a06382dfd68133acb91ee81f39708d0ef98c5

.github/workflows/install-and-test.yml

@@ -32,10 +32,9 @@ jobs:
       run: |
         python -m pip install --upgrade pip
         python -m pip install setuptools wheel
-        python -m pip install pytest pytest-cov
     - name: Install package
       run: |
-        USE_FORTRAN=1 python -m pip install -e .
+        USE_FORTRAN=1 python -m pip install -e .[dev]
     - name: Test with pytest
       run: |
         pytest

.pre-commit-config.yaml

@@ -4,15 +4,20 @@ repos:
   - id: jupyter-clear-output
     name: jupyter-clear-output
     files: \.ipynb$
+    exclude: ^draft/
     stages: [commit]
-    language: system
+    language: python
+    additional_dependencies: ['nbconvert']
     entry: python3 -m nbconvert --ClearOutputPreprocessor.enabled=True --inplace
 - repo: https://github.com/pre-commit/pre-commit-hooks
   rev: v2.3.0
   hooks:
   - id: end-of-file-fixer
+    exclude: ^draft/
   - id: trailing-whitespace
+    exclude: ^draft/
 - repo: https://github.com/psf/black
   rev: 22.3.0
   hooks:
   - id: black-jupyter
+    exclude: ^draft/

.readthedocs.yaml

@@ -12,7 +12,7 @@ sphinx:
 # Optionally build your docs in additional formats such as PDF
 formats:
   - pdf
-  
+
 # Optionally set the version of Python and requirements required to build your docs
 python:
   version: 3.8

.zenodo.json

@@ -50,4 +50,3 @@
       "Active Learning"
    ]
 }
-

CONTRIBUTING.md

@@ -5,12 +5,12 @@ Contributions to proFit are always welcome.
 * repository on [github](https://github.com/redmod-team/profit)
   * [issue tracker](https://github.com/redmod-team/profit/issues)
   * pull requests
-  * feature planning via *Projects*
+  * feature planning via *Projects* and *Discussions*
   * automation via *Actions*
   * managing releases
 * documentation on [readthedocs.io](https://profit.readthedocs.io/en/latest)
 * meeting log on [HedgeDoc](https://pad.gwdg.de/lOiz56TIS4e5E9-92q-2MQ?view)
-* archived on [zenodo.org](https://zenodo.org/record/6563463#.Yo9PjiNBxQo) with DOI `10.5281/zenodo.3580488`
+* archived on [zenodo.org](https://zenodo.org/record/3580488) with DOI [`10.5281/zenodo.3580488`](https://doi.org/10.5281/zenodo.3580488)
 * python package published on [PyPI](https://pypi.org/project/profit/)
 * test coverage on [coveralls.io](https://coveralls.io/github/redmod-team/profit)
 
@@ -52,6 +52,8 @@ Starting with the development for `v0.6`, proFit uses *pre-commit* to ensure con
 [black](https://github.com/psf/black) and clean jupyter notebooks.
 Pre-commit is configured with `.pre-commit-config.yaml` and needs to be activated with `pre-commit install`.
 To run the hooks on all files (e.g. after adding new hooks), use `pre-commit run --all-files`.
+The [pre-commit ci](https://pre-commit.ci/) is used to enforce the hooks for all pull requests.
+Currently pre-commit is configured to ignore everything in `draft`.
 
 ## Installing
 Install proFit from your git repository using the editable install: `pip install -e .[docs,dev]`.
@@ -79,7 +81,7 @@ The release candidate should be tagged as *pre-release* in GitHub. It will not b
 
 Releases are created with the GitHub interface and trigger workflows to automate the publishing and packaging. In particular:
 * A python package is created an uploaded to [PyPI](https://pypi.org/project/profit/)
-* The repository is archived and a new version added to [zenodo](https://zenodo.org/record/6563463#.Yo9PjiNBxQo)
+* The repository is archived and a new version added to [zenodo](https://zenodo.org/record/3580488)
 * A new version of the documentation is created on [readthedocs](https://profit.readthedocs.io)
 
 Before creating a version, check the metadata in `setup.cfg` (for the python package) and `.zenodo.json` (for zenodo).

LICENSE.txt

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2018-2019 The RedMod Team
+Copyright (c) 2018-2022 The RedMod Team
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.md

@@ -1,38 +1,40 @@
 [![DOI](https://zenodo.org/badge/168945305.svg)](https://zenodo.org/badge/latestdoi/168945305)
 [![PyPI](https://img.shields.io/pypi/v/profit)](https://pypi.org/project/profit/)
 [![Python Versions](https://img.shields.io/pypi/pyversions/profit)](https://pypi.org/project/profit/)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 [![Coverage Status](https://coveralls.io/repos/github/redmod-team/profit/badge.svg)](https://coveralls.io/github/redmod-team/profit)
 
 [![Documentation Status](https://readthedocs.org/projects/profit/badge/?version=latest)](https://profit.readthedocs.io/en/latest/?badge=latest)
 [![Install & Test Status](https://github.com/redmod-team/profit/actions/workflows/install-and-test.yml/badge.svg?)](https://github.com/redmod-team/profit/actions/workflows/install-and-test.yml)
+[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/redmod-team/profit/master.svg)](https://results.pre-commit.ci/latest/github/redmod-team/profit/master)
 [![Publish to PyPI Status](https://github.com/redmod-team/profit/actions/workflows/publish-to-pypi.yml/badge.svg)](https://github.com/redmod-team/profit/actions/workflows/publish-to-pypi.yml)
 
 <img src="https://raw.githubusercontent.com/redmod-team/profit/master/logo.png" width="208.5px">
 
 # Probabilistic Response Model Fitting with Interactive Tools
 
-This is a collection of tools for studying parametric dependencies of 
-black-box simulation codes or experiments and construction of reduced 
-order response models over input parameter space. 
+This is a collection of tools for studying parametric dependencies of
+black-box simulation codes or experiments and construction of reduced
+order response models over input parameter space.
 
-proFit can be fed with a number of data points consisting of different 
-input parameter combinations and the resulting output of the simulation under 
-investigation. It then fits a response-surface through the point cloud 
+proFit can be fed with a number of data points consisting of different
+input parameter combinations and the resulting output of the simulation under
+investigation. It then fits a response-surface through the point cloud
 using Gaussian process regression (GPR) models.
-This probabilistic response model allows to predict ("interpolate") the output 
-at yet unexplored parameter combinations including uncertainty estimates. 
-It can also tell you where to put more training points to gain maximum new 
+This probabilistic response model allows to predict ("interpolate") the output
+at yet unexplored parameter combinations including uncertainty estimates.
+It can also tell you where to put more training points to gain maximum new
 information (experimental design) and automatically generate and start
-new simulation runs locally or on a cluster. Results can be explored and checked 
+new simulation runs locally or on a cluster. Results can be explored and checked
 visually in a web frontend.
 
 Telling proFit how to interact with your existing simulations is easy
-and requires no changes in your existing code. Current functionality covers 
-starting simulations locally or on a cluster via [Slurm](https://slurm.schedmd.com), subsequent 
-surrogate modelling using [GPy](https://github.com/SheffieldML/GPy), 
-[scikit-learn](https://github.com/scikit-learn/scikit-learn), 
+and requires no changes in your existing code. Current functionality covers
+starting simulations locally or on a cluster via [Slurm](https://slurm.schedmd.com), subsequent
+surrogate modelling using [GPy](https://github.com/SheffieldML/GPy),
+[scikit-learn](https://github.com/scikit-learn/scikit-learn),
 as well as an active learning algorithm to iteratively sample at interesting
-points and a Markov-Chain-Monte-Carlo (MCMC) algorithm. The web frontend to interactively explore the point cloud 
+points and a Markov-Chain-Monte-Carlo (MCMC) algorithm. The web frontend to interactively explore the point cloud
 and surrogate is based on [plotly/dash](https://github.com/plotly/dash).
 
 ## Features
@@ -48,7 +50,7 @@ and surrogate is based on [plotly/dash](https://github.com/plotly/dash).
 
 ## Installation
 
-Currently, the code is under heavy development, so it should be cloned 
+Currently, the code is under heavy development, so it should be cloned
 from GitHub via Git and pulled regularly.
 
 ### Requirements
@@ -76,7 +78,7 @@ All dependencies are configured in `setup.cfg` and should be installed automatic
 Automatic tests use `pytest`.
 
 ### Windows 10
-To install proFit under Windows 10 we recommend using *Windows Subsystem 
+To install proFit under Windows 10 we recommend using *Windows Subsystem
 for Linux (WSL2)* with the Ubuntu 20.04 LTS distribution ([install guide](https://docs.microsoft.com/en-us/windows/wsl/install-win10)).
 
 After the installation of WSL2 execute the following steps in your Linux terminal (when asked press `y` to continue):
@@ -125,14 +127,19 @@ Certain surrogates require a compiled Fortran backend. To enable compilation of
 ```bash
 python3 -m pip install -e . --user
 ```
-3. If pip warns you about PATH or proFit is not found close and reopen the terminal 
+3. If pip warns you about PATH or proFit is not found close and reopen the terminal
    and type `profit --help` to check if the installation was successful.
 
 
 ### Documentation using *Sphinx*
 Install requirements for building the documentation using `sphinx`
 
-    pip install .[docs] 
+    pip install .[docs]
+
+Additionally `pandoc` is required on a system level:
+
+    sudo apt install pandoc
+
 
 ## HowTo
 
@@ -158,27 +165,27 @@ Also, the integration tests under `tests/integration_tests/` may be informative
 ### Steps
 
 1. Create and enter a directory (e.g. `study`) containing `profit.yaml` for your run.
-    If your code is based on text configuration files for each run, copy the according directory to `template` and 
+    If your code is based on text configuration files for each run, copy the according directory to `template` and
     replace values of parameters to be varied within UQ/surrogate models by placeholders `{param}`.
-  
-2. Running the simulations: 
+
+2. Running the simulations:
    ```bash
    profit run
    ```
-   to start simulations at all the points. Per default the generated input variables are written to `input.txt` and the 
+   to start simulations at all the points. Per default the generated input variables are written to `input.txt` and the
    output data is collected in `output.txt`.
-   
+
    For each run of the simulation, proFit creates a run directory, fills the templates with the generated input data and
-   collects the results. Each step can be customized with the 
+   collects the results. Each step can be customized with the
    [configuration file](https://profit.readthedocs.io/en/latest/config.html).
 
 3. To fit the model:
    ```bash
    profit fit
    ```
    Customization can be done with `profit.yaml` again.
-   
-4. Explore data graphically: 
+
+4. Explore data graphically:
    ```bash
    profit ui
    ```
@@ -194,15 +201,15 @@ proFit supports scheduling the runs on a cluster using *slurm*. This is done ent
 the usage doesn't change.
 
 `profit ui` starts a *dash* server and it is possible to remotely connect to it (e.g. via *ssh port forwarding*)
-  
+
 ## User-supplied files
 
 * a [configuration file](https://profit.readthedocs.io/en/latest/config.html): (default: `profit.yaml`)
   * Add parameters and their distributions via `variables`
   * Set paths and filenames
   * Configure the run backend (how to interact with the simulation)
   * Configure the fit / surrogate model
-  
+
 * the `template` directory
   * containing everything a simulation run needs (scripts, links to executables, input files, etc)
   * input files use a template format where `{variable_name}` is substituted with the generated values

doc/Makefile

@@ -16,4 +16,4 @@ help:
 # Catch-all target: route all unknown targets to Sphinx using the new
 # "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
 %: Makefile
-	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

doc/cluster.rst

@@ -50,6 +50,5 @@ For more information and to submit the bugs you encountered visit the
 Available Options
 -----------------
 
-.. autoraw:: profit.run.slurm.SlurmRunner.handle_config
-
-.. autoraw:: profit.run.zeromq.ZeroMQRunnerInterface.handle_config
+* :py:class:`profit.run.slurm.SlurmRunner`
+* :py:class:`profit.run.zeromq.ZeroMQRunnerInterface`

doc/components.rst

@@ -10,6 +10,7 @@ different modules of proFit.
 
     variables
     run_system
+    dev_run_system
     surrogates
     active_learning
     mcmc