Repository navigation
Tutorial Y
A job is an individual unit of computing.
A worflow is a series of jobs that must be performed to recieve a desired output. These are also called pipelines. In order to be truly high-throughput we need automated workflows.
There are many workflow management tools available. Aiida, Joblib, AFlow, ASE etc.
These are used to build automated databases, such as the Materials Project, OQMD, Materials Cloud, Jarvis etc.
JobFlow is the successor to Fireworks, which is what was used to build the Materials Project.
Workflows are executed by Workflow Engines. Workflow libraries are available that encode the process of the workflows.
Pymatgen is a transformation, processing or preparation tool. Custodian is an error correction in realtime tool.
A job can be any process. But it must return some kind of serialisable output object. (such as int, float, bool. list dict).
MSONable types too, the underlying datastructure of pymatgen.
Jobs are defiined using decorators.
JSON is JavaScript Object Notation. Designed to be both human and machine readable.
PyDantic used for validation and serialisation through type annotations.
"TaskDocuments" are Pydantic models with a function that initialises them from a directory or similar.
Model, schema and document are considered interchangable. Workflow and pipeline are considered interchangable.
Job flow is a language to construct work flows. Job flow Remote is to execute them. It is comprised of User, Runner and Worker.
To begin this kind of workflow on POWER system, we have our own database server. On this server is the software MongoDB that can be accessed by the command "mongosh" short for Mongo Shell. Unfortunately this can't be installed on the log in node so once you are on POWER you will need to set up passwordless access to our compute node. Currently these are compute-0-422 (132.66.112.243)