Scientist performs a recorded code run

[krystophny]

User story

1. (optional) I pull required input and templates from DATA to RUN.
   dat pull <path>
2. I start the run.
   slurm <run_code>
3. I commit the started run to the database.
   dat commit -m <message>
4. The run has finished. I push results from RUN to DATA.
   dat push

Code story

1. Unidirectional sync of subdirectory DATA->RUN. DATA access from file system or cloud. No metadata.
2. Code writes (optional) specific metadata in metadata.json. Careful with parallelization.
3. Dat writes message (title and optional body) and automatically retrievable information to metadata.json and creates an according database entry with status running.
4. Dat updates local metadata.json and database and sets status finished. Unidirectional sync of current directory from RUN to DATA.

Implementation

1. Via rclone.
2. JSON in eLabFTW standard. Python: json in standard library, Fortran: json-fortran.
3. Mimick git and use eLabFTW API. Credentials have been stored as environment variables.
4. As above.

Open questions

• What if a run fails? Still push and/or store metadata as failed?
• dat search via eLabFTW API?
• dat remote add to store urls and keys?

[krystophny]

Automated triage pass (Claude), grounded in the repository at fe6c66f. Verify before acting.

Verdict: keep open, as a tracker. Nothing in this story has been implemented, and the repo has not moved since it was filed — fe6c66f is still HEAD, the last of 9 commits all dated 13–15 Feb 2024. The usual "the code moved on" heuristic is inverted here: this issue is the specification for work that stopped one day after it was written. One thing has changed underneath it, though, and it invalidates part of the Implementation section — see Blocking dependency change below.

Purpose

Every ITPcp production run currently loses its provenance the moment it finishes: which inputs it consumed, which code revision produced it, whether it completed or crashed, and where the outputs landed exist only in the operator's memory and in shell history. This story makes a run self-describing — inputs pulled from a canonical store, a machine-written metadata.json beside the results, and a matching eLabFTW entry that is searchable months later. That is the minimum needed for results in a paper to be traceable back to the run that produced them, and it is what the FAIR principles (Wilkinson et al. 2016) call findability and provenance. The scientist-facing cost must stay at four commands, or it will not be used.

Blocking dependency change

The Implementation section says "use eLabFTW API" and pyproject.toml:17 pins elabapy>=1.1.0, imported at src/dat/journal.py:6. That library is now deprecated by its own authors. The upstream README states verbatim:

WARNING: DEPRECATED LIBRARY — Because this package is for APIv1 and APIv1 is deprecated from eLab, this library is therefore also deprecated and should not be used.

elabapy last released 1.1.0 (8 Aug 2023, per PyPI); the supported client is elabapi-python, currently 5.6.0, targeting REST API v2. Any work on steps 3–4 must migrate first, and the migration is not cosmetic. I unpacked the 5.6.0 wheel to check: ExperimentsApi.post_experiment() (elabapi_python/api/experiments_api.py:348) is declared :return: None, and the response headers are only returned when _return_http_data_only is false — elabapi_python/api_client.py:164-168, inside the private ApiClient.__call_api that both call_api and every *_with_http_info wrapper delegate to:

if _return_http_data_only:
    return (return_data)
else:
    return (return_data, response_data.status,
            response_data.getheaders())

So the pattern at journal.py:45-51 —

    def create(self):
        experiment = self.manager.create_experiment()
        if experiment["result"] == "success":
            entry = create_entry("Untitled")
            entry["id"] = experiment["id"]
            return entry
        raise ValueError("Failed to create experiment on eLabFTW server.")

— has no v2 equivalent and must be replaced by parsing the Location response header from post_experiment_with_http_info(body=...). Method names verified by unpacking the 5.6.0 wheel: post_experiment and patch_experiment(body, id) and read_experiments in elabapi_python/api/experiments_api.py (lines 348, 243, 542), and post_upload(entity_type, id, file=..., comment=...) in elabapi_python/api/uploads_api.py:263. Auth is ApiClient.set_default_header("Authorization", ELAB_KEY); the host must include the API path (Configuration.host defaults to https://elab.local:3148/api/v2).

Current state

Four source files, 145 lines. Of the four story steps, step 3 is ~40% done and steps 1, 2 and 4 do not exist.

src/dat/dat.py:9-16 — dispatch is globals()[name], and commit is the only command function at module scope:

def main():
    print("This is Dat.")
    command = get_function_by_name(sys.argv[1])
    command(sys.argv[2:])

def get_function_by_name(name):
    return globals()[name]

src/dat/dat.py:19-32 — commit(args) accepts args and never reads it, so -m <message> from step 3 is silently discarded. Line 27 is missing its f prefix. Line 30 is where the eLabFTW write belongs and is a bare print:

    print("Committing changes to {local_storage.filename}...")
    local_storage.save(entry)

    print("Committing changes to eLabFTW...")

src/dat/dat.py:6-7 — entry and local_storage are built at import time with a CWD-relative hard-coded path, so the storage target cannot be redirected.

src/dat/journal.py:36-57 — ElabStorage is fully written but never instantiated anywhere in the tree. HDF5Storage (journal.py:60-68) raises NotImplementedError in both methods. copy_fields (journal.py:15-17) is never called. src/dat/draft.py is dead — it references argv, json and manager with no imports — but lines 13-20 are the only surviving record of the intended eLabFTW extra_fields payload and should be salvaged, not deleted blind.

There is no status field anywhere, no rclone invocation, and no notion of a run identity. There are no tests, no CI, and no .github/ directory.

Reproduction / evidence

Real run against fe6c66f, in a clean venv. Reproduce with:

git clone https://github.com/itpplasma/dat && cd dat && git checkout fe6c66f
python3 -m venv venv && ./venv/bin/pip install .
mkdir run && cd run

Step 1 of the story, dat pull:

$ ../venv/bin/dat pull /proj/plasma/foo
This is Dat.
  File ".../dat/dat.py", line 16, in get_function_by_name
    return globals()[name]
KeyError: 'pull'

Step 3, dat commit -m <message>:

$ printf 'my run title\n' | ../venv/bin/dat commit -m "sync test run"
This is Dat.
Title: Committing changes to {local_storage.filename}...
Committing changes to eLabFTW...
Changes committed.
$ cat metadata.json
{"title": "my run title"}

Observed: -m "sync test run" is discarded, the tool prompts interactively anyway, the literal {local_storage.filename} is printed, no status is recorded, and nothing reaches eLabFTW. Expected: no prompt, metadata.json containing the message and "status": "running", and an eLabFTW entry id.

The prompt is fatal in a batch job, where stdin is not a tty — same command in a fresh directory with stdin closed:

$ ../venv/bin/dat commit -m "x" < /dev/null
This is Dat.
  File ".../dat/dat.py", line 25, in commit
    entry["title"] = input("Title: ")
EOFError: EOF when reading a line

Two further defects fall out of the same dispatch design — dat with no argument raises IndexError at dat.py:11, and dat os reaches the imported os module and dies with TypeError: 'module' object is not callable, because globals() exposes every module-level name as a command.

Behaviour (BDD)

Scenario: Pull inputs without clobbering local edits
  Given DAT_DATA is "/proj/plasma/runs" and the local run directory is empty
  And /proj/plasma/runs/case42/input.namelist exists
  When I run "dat pull case42"
  Then ./input.namelist exists and is byte-identical to the source
  And rclone was invoked with argv exactly
    ["rclone", "copy", "--ignore-existing", "--immutable",
     "/proj/plasma/runs/case42", "."]
  And the argv contains no "sync", "move", "delete" or "--delete-*" token
  And no file in /proj/plasma/runs was created, modified or deleted

Scenario: Commit a started run non-interactively
  Given a run directory with no metadata.json
  And stdin is not a tty
  When I run "dat commit -m 'KNOSOS scan, beta=1.5%'"
  Then metadata.json contains title "KNOSOS scan, beta=1.5%" and status "running"
  And metadata.json contains run_id, created_at (ISO 8601 UTC), host, and code_revision
  And the process exits 0 without prompting

Scenario: Push marks the run finished
  Given metadata.json exists with status "running" and an eLabFTW id
  When I run "dat push"
  Then metadata.json status becomes "finished" and finished_at is set
  And the eLabFTW entry with that id is patched to status "finished"
  And results are copied RUN -> DATA with no deletion at the destination

Scenario: A failed run is still recorded
  Given metadata.json exists with status "running"
  When I run "dat push --failed"
  Then metadata.json status becomes "failed"
  And the results are still copied to DATA

Scenario: Unknown command fails cleanly
  When I run "dat os"
  Then stderr contains "unknown command: os" and lists the valid commands
  And the exit code is 2 with no traceback

Implementation plan

Do these in order; steps 1–3 are prerequisites for everything else.

1. src/dat/cli.py (new). Replace globals() dispatch with argparse subparsers: pull, push, commit, archive, status. Explicit COMMANDS dict, dict.get() with a clean SystemExit(2) on miss. Handles the bare-dat IndexError and the dat os leak in one change.
2. src/dat/dat.py — reduce to a thin main() delegating to cli.main(). Delete the module-level entry/local_storage at lines 6-7; construct them inside each command from a config object. Fix the missing f prefix at line 27.
3. src/dat/config.py (new). Read DAT_DATA (canonical store root), DAT_ARCHIVE, ELAB_URL, ELAB_KEY. Raise a named error listing the missing variable rather than passing None into the API client, which is what journal.py:38-40 does today.
4. src/dat/journal.py — swap import elabapy for elabapi_python. Rewrite ElabStorage.__init__ to the v2 pattern (Configuration(), configuration.host = f"{ELAB_URL}/api/v2", api_client.set_default_header("Authorization", ELAB_KEY)). Rewrite create() to call post_experiment_with_http_info(body=...) and parse the trailing integer of the Location header. Rewrite save() to patch_experiment(body, id). Give Storage.load a consistent zero-arg signature — Storage.load(self, entry) at line 29 contradicts JSONStorage.load(self) at line 75.
5. pyproject.toml / requirements.txt — replace elabapy>=1.1.0 with elabapi-python>=5.0. Fix the three placeholder [project.urls] entries pointing at example.com and at the upstream elabapy repo.
6. src/dat/metadata.py (new). Define the schema and a SCHEMA_VERSION constant: run_id (uuid4), title, body, status ∈ {running, finished, failed}, created_at, finished_at, host, user, code_revision, slurm_job_id, data_path. Auto-populate slurm_job_id from SLURM_JOB_ID and code_revision from git rev-parse HEAD in the run directory, both optional. Write via tempfile.NamedTemporaryFile + os.replace for atomicity — the issue's "careful with parallelization" note; also document that only MPI rank 0 may call dat commit. Freeze this exact shape:

{
  "schema_version": 1,
  "run_id": "0f9c1e3a-7b52-4f0e-9c2a-1d8e6b4a5c31",
  "title": "KNOSOS scan, beta=1.5%",
  "body": "",
  "status": "running",
  "created_at": "2024-02-15T09:31:04Z",
  "finished_at": null,
  "host": "itpcp01",
  "user": "albert",
  "code_revision": "fe6c66f6d6eeaa5c830c6a703032258e473fc7c9",
  "slurm_job_id": "184213",
  "data_path": "/proj/plasma/runs/case42",
  "elab_id": 231
}

finished_at, slurm_job_id, code_revision and elab_id may be null; every other key is required.
7. src/dat/sync.py (new). Wrap rclone via subprocess.run with a checked exit code. Use rclone copy, never rclone sync — copy "doesn't delete files from the destination", which is exactly issue #1's no-delete rule; add --ignore-existing ("Skip all files that exist on destination") and --immutable ("Do not modify files, fail if existing files have been modified") for the no-overwrite rule. Detect a missing rclone binary up front with a clear message.
8. src/dat/commands/ (new) — pull.py, push.py, commit.py, archive.py. push takes --failed; archive moves DATA → DAT_ARCHIVE and patches the record, closing issue #1's fourth bullet.
9. src/dat/draft.py — delete, after moving the extra_fields payload shape from lines 13-20 into journal.py as the eLabFTW metadata builder.
10. tests/ (new, greenfield) — test_cli.py (dispatch, unknown command, bare invocation), test_metadata.py (schema, status transitions, atomic write), test_sync.py (assert the exact rclone argv via a stub binary on PATH — do not hit the network), test_journal.py (ElabStorage against a mocked ApiClient, including Location-header id parsing). Add [project.optional-dependencies] test = ["pytest"] to pyproject.toml.
11. .github/workflows/ci.yml (new) — pip install -e .[test] and pytest on 3.10–3.12. The repo has no CI at all today.

No sibling-repo changes are required; dat has no submodules and no coupling to libneo. The Fortran side named in the issue (json-fortran) only needs to agree on the metadata.py schema — freeze it in step 6 before anyone writes the Fortran writer.

Acceptance criteria

• dat with no arguments prints usage and exits 2 — no IndexError
• dat os prints unknown command: os and exits 2 — no TypeError
• dat commit -m "msg" with stdin closed writes title = "msg" and status = "running" and exits 0 without prompting
• dat push flips status to finished; dat push --failed flips it to failed; both sync RUN → DATA
• dat pull and dat push invoke rclone copy --ignore-existing --immutable, asserted on argv in tests/test_sync.py; a pre-existing destination file is never modified
• grep -r elabapy src/ pyproject.toml requirements.txt returns nothing
• pytest passes; all five BDD scenarios have a corresponding test
• pip install -e . && dat --help succeeds on a clean 3.10 venv
• A round trip against https://elabftw.tugraz.at/ creates one experiment and patches it once (manual, credentials required — keep out of CI)

Open questions from the issue body

• Failed runs — yes, push and record. Data from a crashed run is often the diagnostic; status: "failed" is cheaper than a lost run. Covered by scenario 4.
• dat search — defer to a follow-up. ExperimentsApi.read_experiments supports it, but it is not needed for the story to work end to end.
• dat remote add — do not build it. rclone config already owns remote definitions; dat should reference an rclone remote name via DAT_DATA rather than duplicating credential storage.

References

• Wilkinson, M.D. et al. (2016), "The FAIR Guiding Principles for scientific data management and stewardship", Scientific Data 3, 160018, doi:10.1038/sdata.2016.18 — the provenance/findability requirements this tool serves.
• Carpi, N., Minges, A., Piel, M. (2017), "eLabFTW: An open source laboratory notebook for research labs", Journal of Open Source Software 2(12), 146, doi:10.21105/joss.00146 — the backend targeted here.
• Davison, A.P. (2012), "Automated Capture of Experiment Context for Easier Reproducibility in Computational Research", Computing in Science & Engineering 14(4), 48–56 — Sumatra solves precisely this problem (record a simulation run with its parameters, code revision and outputs) and is worth reading before finalising the metadata.py schema in step 6.
• Prior art in the same space, by name: DVC and DataLad (git-annex based) for data-side versioning, MLflow Tracking for the run-record model, Snakemake for workflow-level provenance. None of them writes to an ELN, which is the gap dat fills here.
• eLabFTW REST API v2 reference: https://doc.elabftw.net/api/v2/ ; client https://github.com/elabftw/elabapi-python
• rclone copy semantics: https://rclone.org/commands/rclone_copy/

Scope and risk

This is an epic, not a single PR — it is labelled story and spans four commands, a dependency migration, and a greenfield test suite. Recommend splitting into atomic issues along the plan: (a) CLI dispatch + config + the three dat.py bugs, (b) elabapy → elabapi-python migration, (c) metadata schema + atomic write, (d) rclone pull/push, (e) archive + CI. Keep #2 open as the tracker linking them. itpplasma/dat#1 is the command-level subset of this same design and should be folded into (a), (d) and (e) rather than worked separately.

Risk is low — there are no users and no tests to regress. The one genuine hazard is the sync layer: a mistaken rclone sync in place of rclone copy would delete data in /proj/plasma. Step 7's argv assertion exists specifically to prevent that, and it should land before anyone runs the tool against a real store.