Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bugfix/sof 5578 wip - Make Colab Great Again #71

Merged
merged 90 commits into from
Jun 21, 2023
Merged
Show file tree
Hide file tree
Changes from 84 commits
Commits
Show all changes
90 commits
Select commit Hold shift + click to select a range
8d3c755
Update notebooks to load the remote code in a more IPython friendly m…
mrakitin Jun 12, 2023
cb8dcbc
Do not clone repo via examples/utils/initialize_settings.py
mrakitin Jun 12, 2023
e9e5602
Ease pins in `requirements-colab.txt`
mrakitin Jun 12, 2023
e43818b
CLN: do not bother with `condacolab`
mrakitin Jun 12, 2023
4312d14
Update steps in the auth notebook
mrakitin Jun 12, 2023
b2860df
Fix colab URL in the auth notebook
mrakitin Jun 12, 2023
b405772
BLD: Reduce the number of requirements and use concrete requirements
mrakitin Jun 12, 2023
ea849a6
BLD: pymatgen 2023.5.31
mrakitin Jun 13, 2023
b472fda
Temp install of api-client from GH
mrakitin Jun 13, 2023
ebe0228
pandas v1.5.3
mrakitin Jun 13, 2023
cdd033c
Use released `exabyte-api-client`
mrakitin Jun 13, 2023
598a4c8
Convert more notebooks to the new approach
mrakitin Jun 13, 2023
2b5b773
Fix a link to Google Colab in `import_materials_from_materialsproject…
mrakitin Jun 13, 2023
ffcc17e
`get_authentication_params.ipynb`: change to the parent dir before ru…
mrakitin Jun 13, 2023
085dcd8
REF: refactor how we get the env in Colab
mrakitin Jun 13, 2023
2e5ca3c
Update the auth notebook to use a single script to set up the env
mrakitin Jun 13, 2023
82c8d3d
Move `settings.*` to `utils/` and update `setup.py` with the respecti…
mrakitin Jun 13, 2023
3665287
Fix imports for `settings` as a part of a package
mrakitin Jun 13, 2023
589f1e0
CLN: stripout notebooks (auth and import from MP)
mrakitin Jun 13, 2023
4ff58ef
REF: better prioritization of the env vars vs. JSON input
mrakitin Jun 13, 2023
a551185
ENH: allow to rerun the `env.sh` script without failures
mrakitin Jun 13, 2023
4fb2328
CLN: clean up debug prints and make the cells consistent
mrakitin Jun 13, 2023
87fbaa5
DOC: fix the link to `settings.json`
mrakitin Jun 13, 2023
0a1d7d0
Update the branch name for notebooks
mrakitin Jun 13, 2023
27d624a
CLN: silence the output of curl and init bash script in notebooks cells
mrakitin Jun 13, 2023
b53016e
CLN: rename references to the repo from `exabyte-api-examples` -> `ex…
mrakitin Jun 13, 2023
fddf86c
CLN: stripout outputs from `material/get_materials_by_formula.ipynb`
mrakitin Jun 13, 2023
f9c7cac
Enable running with a local Jupyter instance
mrakitin Jun 14, 2023
af6243d
MNT: tweak verbosity conditions
mrakitin Jun 14, 2023
8d80bf8
FIX: `examples/workflow/get_workflows.ipynb`
mrakitin Jun 15, 2023
33d05fd
FIX: `examples/material/create_material.ipynb`
mrakitin Jun 15, 2023
13761f0
ENH: enable git-lfs support for some notebooks
mrakitin Jun 15, 2023
8fac4d5
FIX: `examples/material/import_materials_from_poscar.ipynb`
mrakitin Jun 15, 2023
296876d
MNT: do not execute code on import from `initialize_settings.py`
mrakitin Jun 15, 2023
9ebb472
ENH: silence apt/git commands output from `env.sh`
mrakitin Jun 15, 2023
c47329a
ENH: entry point to return the notebook path only
mrakitin Jun 15, 2023
52250bd
MNT: rearrange `env.sh` script
mrakitin Jun 15, 2023
0c6be1b
FIX: `return` -> `print` in the `cli()` func
mrakitin Jun 15, 2023
6e52a41
MNT: back to `return`, but clean up another stray print
mrakitin Jun 15, 2023
0358017
FIX: `print`, not `return` for the entry point
mrakitin Jun 15, 2023
3c2062b
FIX: parens, not curly braces for shell cmd execution
mrakitin Jun 15, 2023
7a931b4
ENH: better output of the ready env
mrakitin Jun 15, 2023
33b1f5a
ENH: export notebook dir from the `env.sh` script
mrakitin Jun 15, 2023
7ff1f2e
MNT: change to dir in cells only works with Python (`%cd ...`, `os.ch…
mrakitin Jun 15, 2023
6567868
CLN: stripout output from `examples/material/import_materials_from_po…
mrakitin Jun 15, 2023
5a4038d
FIX: `examples/job/create_and_submit_job.ipynb`
mrakitin Jun 15, 2023
bcd653d
ENH: minor print improvement
mrakitin Jun 15, 2023
d04df04
FIX: `examples/job/get-file-from-job.ipynb` (updated initialization a…
mrakitin Jun 15, 2023
57e6fe5
FIX: add a missing import of `urllib` in `examples/job/get-file-from-…
mrakitin Jun 15, 2023
8fdcf0f
FIX: `examples/job/run-simulations-and-extract-properties.ipynb` (upd…
mrakitin Jun 15, 2023
a6e8626
FIX: `examples/job/ml-train-model-predict-properties.ipynb` (updated …
mrakitin Jun 15, 2023
2a0fd52
DOC: update instructions to also install utils in the dev mode (so th…
mrakitin Jun 15, 2023
7529e85
DOC: fix references to `settings.json`
mrakitin Jun 15, 2023
5bdaf6f
MNT: better handling of Colab vs. local Jupyter
mrakitin Jun 15, 2023
b6eb23a
MNT: address review comments (env vars names, descriptions, etc.)
mrakitin Jun 16, 2023
d86ee9b
MNT: fix typo in username placeholder
mrakitin Jun 16, 2023
3f59826
FIX: correct entry point `notebook-path`
mrakitin Jun 16, 2023
3409c52
DEV: add a convenience script to update branch names in the notebooks
mrakitin Jun 16, 2023
0edef7a
MNT: `dev` -> `bugfix/SOF-5578-WIP` branches in Colab URLs
mrakitin Jun 16, 2023
e9d9cb1
PKG: move `setup.py` to the root of the repo and make corresponding a…
mrakitin Jun 16, 2023
a4259e5
MNT: docs.exabyte.io -> docs.mat3ra.com
mrakitin Jun 16, 2023
66c945e
FIX: fix examples imports in the notebooks
mrakitin Jun 16, 2023
f83f195
FIX: fix imports
mrakitin Jun 16, 2023
45b85e2
MNT: platform.exabyte.io -> platform.mat3ra.com
mrakitin Jun 16, 2023
6e21782
FIX: fix import of settings in utils/generic.py
mrakitin Jun 16, 2023
6f7fbff
FIX: install api-examples in a non-editable mode
mrakitin Jun 16, 2023
de247b7
MNT: absolute import of settings
mrakitin Jun 16, 2023
7520dce
ENH: add a helper script to render Jupyter notebook files (.ipynb)
mrakitin Jun 17, 2023
8ad98fb
MNT: ignore generated artifacts after notebooks runs
mrakitin Jun 17, 2023
c4688bf
FIX: use `BANK_WORKFLOW_ID="56xDyXsPMNJ7cF9nv"` for run-simulations-a…
mrakitin Jun 17, 2023
d4d23f4
DOC: update the reference to the bank id in the text cell of run-simu…
mrakitin Jun 17, 2023
ce25d72
MNT: configure the rendering script to ignore run-simulations-and-ext…
mrakitin Jun 17, 2023
f76dcb1
[AUTO] automatically updated notebooks with the rendering script
mrakitin Jun 17, 2023
e382600
FIX: update references from Exabyte/.io -> Mat3ra/.com per PR review
mrakitin Jun 19, 2023
0179594
FIX: update references from Exabyte/.io -> Mat3ra/.com per PR review
mrakitin Jun 19, 2023
ee685be
FIX: update references from Exabyte/.io -> Mat3ra/.com per PR review
mrakitin Jun 19, 2023
6078b14
FIX: address review comments regarding the names and docstrings in `i…
mrakitin Jun 19, 2023
8ab5651
FIX: update references from Exabyte/.io -> Mat3ra/.com per PR review
mrakitin Jun 19, 2023
62fba13
TST: test `api_interoperability_showcase.ipynb` in Colab
mrakitin Jun 19, 2023
ff1cc93
FIX: resolve pymatgen's import and stripout notebook's outputs
mrakitin Jun 20, 2023
57c4439
ENH: add support in the rendering script for individual notebooks pas…
mrakitin Jun 20, 2023
4c3a4c2
MNT: fix the `api_interoperability_showcase.ipynb` notebook and rende…
mrakitin Jun 20, 2023
ecdf5df
DOC: add API interoperability notebook to the README
mrakitin Jun 20, 2023
6ca2143
[AUTO] automatically updated notebooks with the rendering script
mrakitin Jun 20, 2023
923e74b
MNT: rename `initialize_settings.py` -> `notebook.py` and clean it up…
mrakitin Jun 21, 2023
30d1f43
DOC: update the description of the variables in `settings.py` (moved …
mrakitin Jun 21, 2023
d7b330b
MNT: run export to python scripts first before executing the .ipynb f…
mrakitin Jun 21, 2023
20dd73c
FIX: import of `get_notebook_info`
mrakitin Jun 21, 2023
e7c7a8c
[AUTO] automatically updated notebooks with the rendering script
mrakitin Jun 21, 2023
a957f08
MNT: branch update `bugfix/SOF-5578-WIP` -> `dev`
mrakitin Jun 21, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -107,3 +107,4 @@ venv.bak/

# Miscellaneous
examples/assets/Molecules_Dataset_Collection
examples/job/pw_scf.*
16 changes: 9 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,20 @@
# Exabyte API Examples
# Mat3ra API Examples

## Contents of this Repository

Below, we list the contents of this repository, in roughly the order that a user might want to go through it in order to learn how our API works.

| Folder | Notebook | Description |
| ------------------|-----------------------------------------| ----------- |
| [Examples/System](examples/system/) | [Get Authentication Params](examples/system/get_authentication_params.ipynb) | Demonstrates how to programatically find your user ID and access token, which is to [authenticate](https://docs.mat3ra.com/rest-api/authentication/) for many portions of the Exabyte API.
| [Examples/Workflow](examples/workflow/) | [Get Workflows](examples/workflow/get_workflows.ipynb) | Walks through how to [query](https://docs.mat3ra.com/rest-api/query-structure/) the Exabyte API to programatically search for workflows. In this example, we search for workflows that calculate the total energy of a material.
| [Examples/System](examples/system/) | [Get Authentication Params](examples/system/get_authentication_params.ipynb) | Demonstrates how to programatically find your user ID and access token, which is to [authenticate](https://docs.mat3ra.com/rest-api/authentication/) for many portions of the Mat3ra API.
| [Examples/Workflow](examples/workflow/) | [Get Workflows](examples/workflow/get_workflows.ipynb) | Walks through how to [query](https://docs.mat3ra.com/rest-api/query-structure/) the Mat3ra API to programatically search for workflows. In this example, we search for workflows that calculate the total energy of a material.
| [Examples/Material](examples/material/) | [Get Materials by Formula](examples/material/get_materials_by_formula.ipynb) | Shows how [queries](https://docs.mat3ra.com/rest-api/query-structure/) can be made to search for materials stored on your account by their formula. In this example, we search for a system containing Si.
| [Examples/Material](examples/material/) | [Create Material](examples/material/create_material.ipynb) | Gives an overview of how materials can be generated in [JSON format](https://docs.mat3ra.com/materials/data/) and uploaded to your user account. In this example, we create an FCC Si crystal and upload it.
| [Examples/Material](examples/material/) | [Import Materials from Materials Project](examples/material/import_materials_from_materialsproject.ipynb) | Demonstrates how materials can be imported from [Materials Project](https://materialsproject.org/about), if their Materials Project ID is known. In this example, we import monoclinic and hexagonal SiGe cells.
| [Examples/Material](examples/material/) | [Import Materials from Poscar](examples/material/import_materials_from_poscar.ipynb) | Provides an example of how materials can be imported directly from Poscar files (a common chemical file format best-known [for its use in VASP](https://www.vasp.at/wiki/index.php/Input)). In this example, we import the unit cell of SiGe.
| [Examples/Job](examples/job/) | [Create and Submit Job](examples/job/create_and_submit_job.ipynb) | Shows how to use the Exabyte API to [create jobs](https://docs.mat3ra.com/jobs/data/) and run them on our cluster. In this example, we run a DFT calculation to get the total energy of an FCC Si unit cell using Quantum Espresso.
| [Examples/Job](examples/job/) | [Get File from Job](examples/job/get-file-from-job.ipynb) | Guides you through using the Exabyte API to query for a list of files produced by a job, describes the metadata assigned to each file, and ends by demonstrating how to download any remote file generated by a job to the local disk.
| [Examples/Material](examples/material/) | [Interoperability between Mat3ra and Materials Project](examples/material/api_interoperability_showcase.ipynb) | In this notebook, we showcase a major advantage of APIs: interoperability. We begin by performing a query using the [Materials Project](https://materialsproject.org) API for all systems containing Iron and Oxygen. We then filter our results (for demonstraiton purposes, we keep only the first 10 materials found). Finally, we upload our results to the Mat3ra platform, where further calculations could be performed to characterize these materials.
| [Examples/Job](examples/job/) | [Create and Submit Job](examples/job/create_and_submit_job.ipynb) | Shows how to use the Mat3ra API to [create jobs](https://docs.mat3ra.com/jobs/data/) and run them on our cluster. In this example, we run a DFT calculation to get the total energy of an FCC Si unit cell using Quantum Espresso.
| [Examples/Job](examples/job/) | [Get File from Job](examples/job/get-file-from-job.ipynb) | Guides you through using the Mat3ra API to query for a list of files produced by a job, describes the metadata assigned to each file, and ends by demonstrating how to download any remote file generated by a job to the local disk.
| [Examples/Job](examples/job/) | [Run Simulations and Extract Properties](examples/job/run-simulations-and-extract-properties.ipynb) | Leads you through the process of copying a [bank workflow](https://docs.mat3ra.com/workflows/bank/) to your account and using it to automatically calculate the [properties](https://docs.mat3ra.com/properties/overview/) of multiple materials. In this example, we determine the [band gap](https://docs.mat3ra.com/properties-directory/non-scalar/band-gaps/) of Si and Ge.
| [Examples/Job](examples/job/) | [ML - Train Model Predict Properties](examples/job/ml-train-model-predict-properties.ipynb) | Walks you through automated dataset generation and the training/prediction of material properties using [machine learning](https://docs.mat3ra.com/software-directory/overview/#machine-learning). In this example, we calculate the band gaps of Si and SiGe, and using various materials properties as descriptors, train a model to predict their band gaps. Finally, we use this trained model to predict the band gap of Ge.

Expand Down Expand Up @@ -50,6 +51,7 @@ Follow the steps below in order to setup and view the Jupyter notebooks:
virtualenv .env
source .env/bin/activate
pip install --no-deps -r requirements.txt
pip install --no-deps -e .
```

4. Run Jupyter and open a notebook in a browser. In order for the post-save hook feature to work properly, one must launch their Jupyter Notebook environment in the folder that contains the file `config.py`, which is the `examples` folder shown below:
Expand All @@ -63,7 +65,7 @@ Follow the steps below in order to setup and view the Jupyter notebooks:

In order to run or edit the examples:

1. Assert an existing Exabyte.io account. Examples require an account to run. New users can register [here](https://platform.mat3ra.com/register) to obtain one.
1. Assert an existing Mat3ra.com account. Examples require an account to run. New users can register [here](https://platform.mat3ra.com/register) to obtain one.

2. Open [settings](examples/settings.json) and adjust it to provide the API authentication parameters. See the [corresponding example](examples/system/get_authentication_params.ipynb) to learn how to obtain the authentication parameters.

Expand All @@ -90,7 +92,7 @@ If you would like to add new examples or adjust existing ones, please consider t

## Links

1. Exabyte.io RESTful API, description in the online documentation: [link](https://docs.mat3ra.com/rest-api/overview/)
1. Mat3ra.com RESTful API, description in the online documentation: [link](https://docs.mat3ra.com/rest-api/overview/)
2. Jupyter.org, official website: [link](https://jupyter.org/)
3. Git Large File Storage, official website: [link](https://git-lfs.github.com/)
4. GitHub Standard Fork & Pull Request Workflow, online explanation: [link](https://gist.github.com/Chaser324/ce0505fbed06b947d962)
Empty file added examples/__init__.py
Empty file.
Loading