Skip to content

Move examples to separate folder #73

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Apr 23, 2025
Merged

Move examples to separate folder #73

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
28b6ac4
Move examples to separate folder
jan-janssen Apr 23, 2025
b61ba76
update environment path
jan-janssen Apr 23, 2025
b0b97df
fix import
jan-janssen Apr 23, 2025
b9074ea
Merge remote-tracking branch 'origin/main' into refactor
jan-janssen Apr 23, 2025
db20b82
update python module path
jan-janssen Apr 23, 2025
d67d371
update postbuild script
jan-janssen Apr 23, 2025
4e4853c
move mybinder related parts
jan-janssen Apr 23, 2025
09fa6fd
rename notebooks
jan-janssen Apr 23, 2025
780fe63
Merge remote-tracking branch 'origin/main' into refactor
jan-janssen Apr 23, 2025
5e17bab
rename python modules
jan-janssen Apr 23, 2025
e17fcb0
Merge remote-tracking branch 'origin/main' into refactor
jan-janssen Apr 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 26 additions & 22 deletions .github/workflows/pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ jobs:
- uses: psf/black@stable
with:
options: "--check --diff"
src: ./qe_xml_parser/src/qe_xml_parser
src: ./example_workflows/quantum_espresso/qe_xml_parser/src/qe_xml_parser

nfdi4ing:
runs-on: ubuntu-22.04
Expand All @@ -27,57 +27,60 @@ jobs:
with:
auto-update-conda: true
python-version: "3.12"
environment-file: environment.yml
environment-file: binder/environment.yml
auto-activate-base: false
- name: Installation and setup
shell: bash -l {0}
run: |
pip install -e python_workflow_definition
conda install -c conda-forge jupyter papermill
verdi presto --profile-name pwd
sudo apt-get install -y $(cat apt.txt)
sudo apt-get install -y $(cat binder/apt.txt)
- name: Create Additional Conda Environments
shell: bash -l {0}
run: |
conda env create -n preprocessing -f source/envs/preprocessing.yaml -y
conda env create -n processing -f source/envs/processing.yaml -y
conda env create -n postprocessing -f source/envs/postprocessing.yaml -y
conda env create -n preprocessing -f example_workflows/nfdi/source/envs/preprocessing.yaml -y
conda env create -n processing -f example_workflows/nfdi/source/envs/processing.yaml -y
conda env create -n postprocessing -f example_workflows/nfdi/source/envs/postprocessing.yaml -y
- name: Tests
shell: bash -l {0}
run: |
papermill aiida_nfdi.ipynb aiida_nfdi_out.ipynb -k "python3"
papermill jobflow_nfdi.ipynb jobflow_nfdi_out.ipynb -k "python3"
papermill pyiron_base_nfdi.ipynb pyiron_base_nfdi_out.ipynb -k "python3"
papermill universal_workflow_nfdi.ipynb universal_workflow_nfdi_out.ipynb -k "python3"
cd example_workflows/nfdi/
papermill aiida.ipynb aiida_out.ipynb -k "python3"
papermill jobflow.ipynb jobflow_out.ipynb -k "python3"
papermill pyiron_base.ipynb pyiron_base_out.ipynb -k "python3"
papermill universal_workflow.ipynb universal_workflow_out.ipynb -k "python3"

quantumespresso:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup environment
shell: bash -l {0}
run: tail --lines=+4 environment_qe.yml >> environment.yml
run: tail --lines=+4 example_workflows/quantum_espresso/environment.yml >> binder/environment.yml
- uses: conda-incubator/setup-miniconda@v3
with:
auto-update-conda: true
python-version: "3.12"
environment-file: environment.yml
environment-file: binder/environment.yml
auto-activate-base: false
- name: Installation and setup
shell: bash -l {0}
run: |
pip install -e qe_xml_parser
pip install -e example_workflows/quantum_espresso/qe_xml_parser
pip install -e python_workflow_definition
conda install -c conda-forge jupyter papermill
verdi presto --profile-name pwd
- name: Tests
shell: bash -l {0}
run: |
export ESPRESSO_PSEUDO=$(pwd)/espresso/pseudo
papermill aiida_qe.ipynb aiida_qe_out.ipynb -k "python3"
papermill jobflow_qe.ipynb jobflow_qe_out.ipynb -k "python3"
papermill pyiron_base_qe.ipynb pyiron_base_qe_out.ipynb -k "python3"
papermill universal_workflow_qe.ipynb universal_workflow_qe_out.ipynb -k "python3"
cp -r example_workflows/quantum_espresso/espresso .
cd example_workflows/quantum_espresso
papermill aiida.ipynb aiida_out.ipynb -k "python3"
papermill jobflow.ipynb jobflow_out.ipynb -k "python3"
papermill pyiron_base.ipynb pyiron_base_out.ipynb -k "python3"
papermill universal_workflow.ipynb universal_workflow_out.ipynb -k "python3"

arithmetic:
runs-on: ubuntu-latest
Expand All @@ -87,7 +90,7 @@ jobs:
with:
auto-update-conda: true
python-version: "3.12"
environment-file: environment.yml
environment-file: binder/environment.yml
auto-activate-base: false
- name: Installation and setup
shell: bash -l {0}
Expand All @@ -98,10 +101,11 @@ jobs:
- name: Tests
shell: bash -l {0}
run: |
papermill aiida_arithmetic.ipynb aiida_arithmetic_out.ipynb -k "python3"
papermill jobflow_arithmetic.ipynb jobflow_arithmetic_out.ipynb -k "python3"
papermill pyiron_base_arithmetic.ipynb pyiron_base_arithmetic_out.ipynb -k "python3"
papermill universal_workflow_arithmetic.ipynb universal_workflow_arithmetic_out.ipynb -k "python3"
cd example_workflows/arithmetic
papermill aiida.ipynb aiida_out.ipynb -k "python3"
papermill jobflow.ipynb jobflow_out.ipynb -k "python3"
papermill pyiron_base.ipynb pyiron_base_out.ipynb -k "python3"
papermill universal_workflow.ipynb universal_workflow_out.ipynb -k "python3"

book:
runs-on: ubuntu-latest
Expand Down
56 changes: 28 additions & 28 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,8 @@ These two Python functions are combined in the following example workflow:
tmp_dict = get_prod_and_div(x=1, y=2)
result = get_sum(x=tmp_dict["prod"], y=tmp_dict["div"])
```
For the workflow representation of these Python functions the Python functions are stored in the [arithmetic_workflow.py](arithmetic_workflow.py)
Python module. The connection of the Python functions are stored in the [workflow_arithmetic.json](workflow_arithmetic.json)
For the workflow representation of these Python functions the Python functions are stored in the [arithmetic_workflow.py](example_workflows/arithmetic/workflow.py)
Python module. The connection of the Python functions are stored in the [workflow_arithmetic.json](example_workflows/arithmetic/workflow.json)
JSON file:
```
{
Expand Down Expand Up @@ -59,45 +59,45 @@ The corresponding Jupyter notebooks demonstrate this functionality:

| Example | Explanation |
|----------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------|
| [aiida_arithmetic.ipynb](aiida_arithmetic.ipynb) | Define Workflow with aiida and execute it with jobflow and pyiron_base. |
| [jobflow_arithmetic.ipynb](jobflow_arithmetic.ipynb) | Define Workflow with jobflow and execute it with aiida and pyiron_base. |
| [pyiron_base_arithmetic.ipynb](pyiron_base_arithmetic.ipynb) | Define Workflow with pyiron_base and execute it with aiida and jobflow. |
| [universal_workflow_arithmetic.ipynb](universal_workflow_arithmetic.ipynb) | Execute workflow defined in the Python Workflow Definition with aiida, executorlib, jobflow, pyiron_base and pure Python. |
| [aiida_arithmetic.ipynb](example_workflows/arithmetic/aiida.ipynb) | Define Workflow with aiida and execute it with jobflow and pyiron_base. |
| [jobflow_arithmetic.ipynb](example_workflows/arithmetic/jobflow.ipynb) | Define Workflow with jobflow and execute it with aiida and pyiron_base. |
| [pyiron_base_arithmetic.ipynb](example_workflows/arithmetic/pyiron_base.ipynb) | Define Workflow with pyiron_base and execute it with aiida and jobflow. |
| [universal_workflow_arithmetic.ipynb](example_workflows/arithmetic/universal_workflow.ipynb) | Execute workflow defined in the Python Workflow Definition with aiida, executorlib, jobflow, pyiron_base and pure Python. |

### Quantum Espresso Workflow
The second workflow example is the calculation of an energy volume curve with Quantum Espresso. In the first step the
initial structure is relaxed, afterward it is strained and the total energy is calculated.
* [quantum_espresso_workflow.py](quantum_espresso_workflow.py) Python functions
* [workflow_qe.json](workflow_qe.json) Workflow definition in the Python Workflow Definition.
* [environment_qe.yml](environment_qe.yml) Conda environment
* [quantum_espresso_workflow.py](example_workflows/quantum_espresso/workflow.py) Python functions
* [workflow_qe.json](example_workflows/quantum_espresso/workflow.json) Workflow definition in the Python Workflow Definition.
* [environment_qe.yml](example_workflows/quantum_espresso/environment.yml) Conda environment

| Example | Explanation |
|------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------|
| [aiida_qe.ipynb](aiida_qe.ipynb) | Define Workflow with aiida and execute it with jobflow and pyiron_base. |
| [jobflow_qe.ipynb](jobflow_qe.ipynb) | Define Workflow with jobflow and execute it with aiida and pyiron_base. |
| [pyiron_base_qe.ipynb](pyiron_base_qe.ipynb) | Define Workflow with pyiron_base and execute it with aiida and jobflow. |
| [universal_workflow_qe.ipynb](universal_workflow_qe.ipynb) | Execute workflow defined in the Python Workflow Definition with aiida, executorlib, jobflow, pyiron_base and pure Python. |
| [aiida_qe.ipynb](example_workflows/quantum_espresso/aiida.ipynb) | Define Workflow with aiida and execute it with jobflow and pyiron_base. |
| [jobflow_qe.ipynb](example_workflows/quantum_espresso/jobflow.ipynb) | Define Workflow with jobflow and execute it with aiida and pyiron_base. |
| [pyiron_base_qe.ipynb](example_workflows/quantum_espresso/pyiron_base.ipynb) | Define Workflow with pyiron_base and execute it with aiida and jobflow. |
| [universal_workflow_qe.ipynb](example_workflows/quantum_espresso/universal_workflow.ipynb) | Execute workflow defined in the Python Workflow Definition with aiida, executorlib, jobflow, pyiron_base and pure Python. |

### NFDI4Ing Scientific Workflow Requirements
To demonstrate the compatibility of the Python Workflow Definition to file based workflows, the workflow benchmark developed as part of [NFDI4Ing](https://github.com/BAMresearch/NFDI4IngScientificWorkflowRequirements)
is implemented for all three simulation codes based on a shared workflow definition.
* [nfdi_ing_workflow.py](nfdi_ing_workflow.py) Python functions
* [workflow_nfdi.json](workflow_nfdi.json) Workflow definition in the Python Workflow Definition.
* [nfdi_ing_workflow.py](example_workflows/nfdi/workflow.py) Python functions
* [workflow_nfdi.json](example_workflows/nfdi/workflow.json) Workflow definition in the Python Workflow Definition.

Additional source files provided with the workflow benchmark:
* [source/envs/preprocessing.yaml](source/envs/preprocessing.yaml) Conda environment for preprocessing
* [source/envs/processing.yaml](source/envs/processing.yaml) Conda environment for processing
* [source/envs/postprocessing.yaml](source/envs/postprocessing.yaml) Conda environment for postprocessing
* [source/macros.tex.template](source/macros.tex.template) LaTeX module template
* [source/paper.tex](source/paper.tex) LaTeX paper template
* [source/poisson.py](source/poisson.py) Poisson Python script
* [source/postprocessing.py](source/postprocessing.py) Postprocessing Python script
* [source/prepare_paper_macros.py](source/prepare_paper_macros.py) LaTeX preprocessing Python script
* [source/unit_square.geo](source/unit_square.geo) Input structure
* [source/envs/preprocessing.yaml](example_workflows/nfdi/source/envs/preprocessing.yaml) Conda environment for preprocessing
* [source/envs/processing.yaml](example_workflows/nfdi/source/envs/processing.yaml) Conda environment for processing
* [source/envs/postprocessing.yaml](example_workflows/nfdi/source/envs/postprocessing.yaml) Conda environment for postprocessing
* [source/macros.tex.template](example_workflows/nfdi/source/macros.tex.template) LaTeX module template
* [source/paper.tex](example_workflows/nfdi/source/paper.tex) LaTeX paper template
* [source/poisson.py](example_workflows/nfdi/source/poisson.py) Poisson Python script
* [source/postprocessing.py](example_workflows/nfdi/source/postprocessing.py) Postprocessing Python script
* [source/prepare_paper_macros.py](example_workflows/nfdi/source/prepare_paper_macros.py) LaTeX preprocessing Python script
* [source/unit_square.geo](example_workflows/nfdi/source/unit_square.geo) Input structure

| Example | Explanation |
|----------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------|
| [aiida_nfdi.ipynb](aiida_nfdi.ipynb) | Define Workflow with aiida and execute it with jobflow and pyiron_base. |
| [jobflow_nfdi.ipynb](jobflow_nfdi.ipynb) | Define Workflow with jobflow and execute it with aiida and pyiron_base. |
| [pyiron_base_nfdi.ipynb](pyiron_base_nfdi.ipynb) | Define Workflow with pyiron_base and execute it with aiida and jobflow. |
| [universal_workflow_nfdi.ipynb](universal_workflow_nfdi.ipynb) | Execute workflow defined in the Python Workflow Definition with aiida, executorlib, jobflow, pyiron_base and pure Python. |
| [aiida_nfdi.ipynb](example_workflows/nfdi/aiida.ipynb) | Define Workflow with aiida and execute it with jobflow and pyiron_base. |
| [jobflow_nfdi.ipynb](example_workflows/nfdi/jobflow.ipynb) | Define Workflow with jobflow and execute it with aiida and pyiron_base. |
| [pyiron_base_nfdi.ipynb](example_workflows/nfdi/pyiron_base.ipynb) | Define Workflow with pyiron_base and execute it with aiida and jobflow. |
| [universal_workflow_nfdi.ipynb](example_workflows/nfdi/universal_workflow.ipynb) | Execute workflow defined in the Python Workflow Definition with aiida, executorlib, jobflow, pyiron_base and pure Python. |
Loading
Loading