Skip to content

Fix integration notebook #832

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 12 commits into from
Closed

Fix integration notebook #832

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
eff11ef
Fix integration notebook
jan-janssen Oct 3, 2025
328ced4
limit maximum number of workers
jan-janssen Oct 3, 2025
4839ffa
fix cluster
jan-janssen Oct 3, 2025
9856539
restrict threads
jan-janssen Oct 4, 2025
3ecb8f2
inside function
jan-janssen Oct 4, 2025
25da5b5
list available resources
jan-janssen Oct 4, 2025
e70c1e4
restrict number of cores
jan-janssen Oct 4, 2025
c1aa2d3
mpi again
jan-janssen Oct 4, 2025
be3870b
use world communicator
jan-janssen Oct 4, 2025
241a4ca
set mpi4py flag
jan-janssen Oct 4, 2025
b28a413
import mpi4py
jan-janssen Oct 4, 2025
240165b
just environmet variable
jan-janssen Oct 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,7 @@ jobs:
shell: bash -l {0}
timeout-minutes: 20
run: |
flux start flux resource list
flux start papermill notebooks/5-1-gpaw.ipynb notebooks/5-1-gpaw-out.ipynb -k python3
flux start papermill notebooks/5-2-quantum-espresso.ipynb notebooks/5-2-quantum-espresso-out.ipynb -k python3

Expand Down
124 changes: 74 additions & 50 deletions notebooks/5-1-gpaw.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,16 +11,16 @@
},
{
"cell_type": "code",
"execution_count": 1,
"execution_count": null,
"id": "21852c01-efda-43fe-add9-d08123a82995",
"metadata": {},
"outputs": [],
"source": [
"import subprocess\n",
"from ase.build import bulk\n",
"from atomistics.workflows.evcurve.helper import (\n",
" analyse_structures_helper as evcurve_analyse_structures,\n",
" generate_structures_helper as evcurve_generate_structures,\n",
"from atomistics.workflows import (\n",
" analyse_results_for_energy_volume_curve,\n",
" get_tasks_for_energy_volume_curve,\n",
")\n",
"import matplotlib.pyplot as plt\n",
"import pprint\n",
Expand All @@ -29,19 +29,25 @@
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "The only function which is executed using [mpi4py](https://mpi4py.readthedocs.io) is the `evaluate_with_gpaw()` function:",
"id": "e15eed0da3c6547b"
"id": "e15eed0da3c6547b",
"metadata": {},
"source": [
"The only function which is executed using [mpi4py](https://mpi4py.readthedocs.io) is the `evaluate_with_gpaw()` function:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": null,
"id": "aa435760-390f-4eff-88f4-96b15777de3e",
"metadata": {},
"outputs": [],
"source": [
"def evaluate_with_gpaw(task_dict, kpts, encut):\n",
" import os\n",
" os.environ[\"OMP_NUM_THREADS\"] = \"1\" # Ensures that each task uses only one thread\n",
" os.environ[\"GPAW_MPI4PY\"] = \"1\" # Ensures that GPAW uses MPI via mpi4py\n",
"\n",
" from gpaw import GPAW, PW\n",
"\n",
" structure = task_dict[\"calc_energy\"].copy()\n",
Expand All @@ -54,19 +60,21 @@
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "As a first step of the workflow the equilibrium structure of four Aluminium atoms is strained by 5%.",
"id": "e9936ca22151d490"
"id": "e9936ca22151d490",
"metadata": {},
"source": [
"As a first step of the workflow the equilibrium structure of four Aluminium atoms is strained by 5%."
]
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": null,
"id": "a1d4e9a1-7275-4d9a-b038-db27185f00ae",
"metadata": {},
"outputs": [],
"source": [
"structure_dict = evcurve_generate_structures(\n",
"task_dict = get_tasks_for_energy_volume_curve(\n",
" structure=bulk(\"Al\", a=4.05, cubic=True),\n",
" num_points=7,\n",
" vol_range=0.05,\n",
Expand All @@ -75,26 +83,30 @@
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "The resulting dictionary of structures `structure_dict` is transformed to simplify the parallel execution:",
"id": "8bf60b7b0f5af31a"
"id": "8bf60b7b0f5af31a",
"metadata": {},
"source": [
"The resulting dictionary of structures `task_dict` is transformed to simplify the parallel execution:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": null,
"id": "73ad1043-eacd-4758-a7b4-f64969beabab",
"metadata": {},
"outputs": [],
"source": [
"task_loop_dict = {k: {\"calc_energy\": v} for k, v in structure_dict.items()}"
"task_loop_dict = {k: {\"calc_energy\": v} for k, v in task_dict[\"calc_energy\"].items()}"
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "The status of the flux cluster is validated using the `flux resource list` command and the `flux jobs -a` command, just to highlight flux was initialized correctly and has access to the available resources.",
"id": "af5ba456f3fafd39"
"id": "af5ba456f3fafd39",
"metadata": {},
"source": [
"The status of the flux cluster is validated using the `flux resource list` command and the `flux jobs -a` command, just to highlight flux was initialized correctly and has access to the available resources."
]
},
{
"cell_type": "code",
Expand Down Expand Up @@ -157,14 +169,16 @@
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "The for each strained structure a calculation task is submitted to the `FluxClusterExecutor`. After the successful submission the current status of the flux queue is printed using `flux jobs -a`. Finally, the results are collected by gathering the [concurrent.futures.Future](https://docs.python.org/3/library/concurrent.futures.html#future-objects) objects.",
"id": "2670c218fbde5b02"
"id": "2670c218fbde5b02",
"metadata": {},
"source": [
"The for each strained structure a calculation task is submitted to the `FluxClusterExecutor`. After the successful submission the current status of the flux queue is printed using `flux jobs -a`. Finally, the results are collected by gathering the [concurrent.futures.Future](https://docs.python.org/3/library/concurrent.futures.html#future-objects) objects."
]
},
{
"cell_type": "code",
"execution_count": 8,
"execution_count": null,
"id": "11d7a7f3-5fbf-4377-ac98-71cd29caf433",
"metadata": {},
"outputs": [
Expand Down Expand Up @@ -224,39 +238,43 @@
" sleep(1)\n",
" pprint.pp(subprocess.check_output([\"flux\", \"jobs\", \"-a\"], universal_newlines=True).split(\"\\n\"))\n",
" result_dict = {\n",
" k: f.result()[-1] \n",
" k: f.result() \n",
" for k, f in tqdm(future_dict.items())\n",
" }\n",
" sleep(1)\n",
" pprint.pp(subprocess.check_output([\"flux\", \"jobs\", \"-a\"], universal_newlines=True).split(\"\\n\"))"
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "The resulting energies for the different volumes are fitted using a 3rd order polynomial to derive the bulk modulus as second derivative multiplied by the equilibrium volume.",
"id": "8c80e3bcb483b069"
"id": "8c80e3bcb483b069",
"metadata": {},
"source": [
"The resulting energies for the different volumes are fitted using a 3rd order polynomial to derive the bulk modulus as second derivative multiplied by the equilibrium volume."
]
},
{
"cell_type": "code",
"execution_count": 9,
"execution_count": null,
"id": "5a5822c7-5ab6-483b-a3ab-8cef5fcaf269",
"metadata": {},
"outputs": [],
"source": [
"fit_dict = evcurve_analyse_structures(\n",
"fit_dict = analyse_results_for_energy_volume_curve(\n",
" output_dict={\"energy\": result_dict},\n",
" structure_dict=structure_dict,\n",
" task_dict=task_dict,\n",
" fit_type=\"polynomial\",\n",
" fit_order=3,\n",
")"
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "The final energy volume curve plot summarizes the results of this calculation.",
"id": "571b7c25dcca8bb"
"id": "571b7c25dcca8bb",
"metadata": {},
"source": [
"The final energy volume curve plot summarizes the results of this calculation."
]
},
{
"cell_type": "code",
Expand Down Expand Up @@ -322,14 +340,16 @@
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "The for each strained structure a calculation task is submitted to the `FluxJobExecutor`. After the successful submission the current status of the flux queue is printed using `flux jobs -a`. Finally, the results are collected by gathering the [concurrent.futures.Future](https://docs.python.org/3/library/concurrent.futures.html#future-objects) objects.",
"id": "21bdd0d42f9806e4"
"id": "21bdd0d42f9806e4",
"metadata": {},
"source": [
"The for each strained structure a calculation task is submitted to the `FluxJobExecutor`. After the successful submission the current status of the flux queue is printed using `flux jobs -a`. Finally, the results are collected by gathering the [concurrent.futures.Future](https://docs.python.org/3/library/concurrent.futures.html#future-objects) objects."
]
},
{
"cell_type": "code",
"execution_count": 12,
"execution_count": null,
"id": "2b0c559b-a117-4766-b679-314ab9e65e97",
"metadata": {},
"outputs": [
Expand Down Expand Up @@ -409,7 +429,7 @@
],
"source": [
"future_dict = {}\n",
"with FluxJobExecutor() as exe:\n",
"with FluxJobExecutor(max_workers=2) as exe:\n",
" for k, v in task_loop_dict.items():\n",
" future_dict[k] = exe.submit(\n",
" evaluate_with_gpaw, \n",
Expand All @@ -421,39 +441,43 @@
" sleep(1)\n",
" pprint.pp(subprocess.check_output([\"flux\", \"jobs\", \"-a\"], universal_newlines=True).split(\"\\n\"))\n",
" result_dict = {\n",
" k: f.result()[-1] \n",
" k: f.result() \n",
" for k, f in tqdm(future_dict.items())\n",
" }\n",
" sleep(1)\n",
" pprint.pp(subprocess.check_output([\"flux\", \"jobs\", \"-a\"], universal_newlines=True).split(\"\\n\"))"
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "The resulting energies for the different volumes are fitted using a 3rd order polynomial to derive the bulk modulus as second derivative multiplied by the equilibrium volume.",
"id": "ca353da1941c1c7c"
"id": "ca353da1941c1c7c",
"metadata": {},
"source": [
"The resulting energies for the different volumes are fitted using a 3rd order polynomial to derive the bulk modulus as second derivative multiplied by the equilibrium volume."
]
},
{
"cell_type": "code",
"execution_count": 13,
"execution_count": null,
"id": "8bb73863-bfde-4e96-9323-5330b2051632",
"metadata": {},
"outputs": [],
"source": [
"fit_dict = evcurve_analyse_structures(\n",
"fit_dict = analyse_results_for_energy_volume_curve(\n",
" output_dict={\"energy\": result_dict},\n",
" structure_dict=structure_dict,\n",
" task_dict=task_dict,\n",
" fit_type=\"polynomial\",\n",
" fit_order=3,\n",
")"
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "The final energy volume curve plot summarizes the results of this calculation.",
"id": "3d4189b5c74ba2a2"
"id": "3d4189b5c74ba2a2",
"metadata": {},
"source": [
"The final energy volume curve plot summarizes the results of this calculation."
]
},
{
"cell_type": "code",
Expand Down
Loading
Loading