diff --git a/README.md b/README.md
index a699aef..148fcb7 100644
--- a/README.md
+++ b/README.md
@@ -19,6 +19,8 @@ Command-line and Python client for downloading and deploying datasets on DBpedia
- [Deploy](#cli-deploy)
- [Delete](#cli-delete)
- [Manifest](#cli-manifest)
+ - [Replay](#cli-manifest-replay)
+ - [Summary](#cli-manifest-summary)
- [Module Usage](#module-usage)
- [Deploy](#module-deploy)
- [Development & Contributing](#development--contributing)
@@ -149,8 +151,10 @@ Options:
--help Show this message and exit.
Commands:
+ delete Delete a dataset from the databus.
deploy Flexible deploy to Databus command supporting three modes:
download Download datasets from databus, optionally using vault access...
+ manifest Manifest utilities.
```
@@ -609,6 +613,102 @@ The manifest records input parameters, per-file URLs, checksums, byte sizes, tim
Refer [examples/reproducible-download.md](examples/reproducible-download.md) for a full walkthrough.
+
+#### Replay
+
+Any manifest written with `--manifest` can be replayed later using `databusclient manifest replay `. Replay re-executes the original operation using the parameters recorded in the manifest — you don't need to remember or retype the original command.
+
+```bash
+# Python
+databusclient manifest replay [OPTIONS] MANIFEST_PATH
+# Docker
+docker run --rm -v $(pwd):/data dbpedia/databus-python-client manifest replay [OPTIONS] MANIFEST_PATH
+```
+
+**Important:** credentials are never stored in the manifest and must always be supplied fresh at replay time — `--vault-token`, `--databus-key`, and `--apikey` behave exactly as they do on the original commands.
+
+```bash
+databusclient manifest replay --help
+
+# Output:
+Usage: databusclient manifest replay [OPTIONS] MANIFEST_PATH
+
+ Replay a previously recorded manifest operation.
+
+ Currently supports replay of download, delete, and deploy manifests.
+ For delete manifests, an interactive confirmation is required by
+ default -- use --force to skip it for scripted/unattended use, or
+ --dry-run to preview without prompting or deleting.
+
+Options:
+ --localdir TEXT Override local output directory for download replay.
+ --databus TEXT Override Databus endpoint for replay.
+ --vault-token TEXT Vault token file path, required if manifest auth
+ method is vault_token.
+ --databus-key TEXT Databus API key, required if manifest auth method is
+ databus_key. Also required for delete replay.
+ --apikey TEXT Databus API key, required for deploy replay.
+ --force For delete replay: skip the interactive confirmation
+ prompt. Required for unattended/scripted replay.
+ --dry-run For delete replay: force a dry-run preview even if
+ the original operation wasn't one.
+ --help Show this message and exit.
+```
+
+**Replaying a download:**
+```bash
+databusclient manifest replay ./manifests/download-run.jsonld --localdir ./replayed-data
+```
+If `--localdir` is omitted, replay falls back to the same auto-computed folder structure a fresh download would use — this is not necessarily the same folder the original download used, since the original folder location itself is never stored in the manifest.
+
+**Replaying a delete:** by default, replay asks for confirmation before deleting, exactly like a normal `delete` call:
+```bash
+databusclient manifest replay ./manifests/delete-run.jsonld --databus-key YOUR_API_KEY
+# About to replay a DELETE operation for the following 1 URI(s):
+# - https://databus.dbpedia.org/...
+# This is irreversible. Proceed? [y/N]:
+```
+For unattended/scripted use (e.g. CI/CD), skip the prompt with `--force`:
+```bash
+databusclient manifest replay ./manifests/delete-run.jsonld --databus-key YOUR_API_KEY --force
+```
+If the original delete was run with `--dry-run --manifest ...`, replay automatically previews without deleting — no flag needed. You can also force a preview on a manifest that wasn't originally a dry run:
+```bash
+databusclient manifest replay ./manifests/delete-run.jsonld --databus-key YOUR_API_KEY --dry-run
+```
+
+**Replaying a deploy:** supported for classic (distributions-as-arguments) and metadata-file deploys. The manifest stores fully-resolved deployment metadata (checksums, sizes, formats already computed), so replay never re-downloads or re-hashes the original files:
+```bash
+databusclient manifest replay ./manifests/deploy-run.jsonld --apikey YOUR_API_KEY
+```
+Replaying redeploys the same version — if it already exists on Databus, it is updated. WebDAV/Nextcloud deploys cannot be replayed, since the originally uploaded local files may no longer exist at their original paths by the time replay runs.
+
+
+#### Summary
+
+Print a readable summary of any recorded manifest without replaying it:
+
+```bash
+# Python
+databusclient manifest summary ./manifests/download-run.jsonld
+# Docker
+docker run --rm -v $(pwd):/data dbpedia/databus-python-client manifest summary ./manifests/download-run.jsonld
+```
+
+Example output:
+
+```
+Command : download
+Executed : 2024-03-24T10:02:49.500418+00:00
+Endpoint : https://databus.dbpedia.org/sparql
+Auth : vault_token
+Files : 1 succeeded · 0 failed
+Total : 100.0 MB
+Status : completed
+```
+
+Only existing data already stored in the manifest is read — no new files are downloaded or written, and no network access happens.
+
## Module Usage
diff --git a/databusclient/api/deploy.py b/databusclient/api/deploy.py
index 08d9016..2c8cd08 100644
--- a/databusclient/api/deploy.py
+++ b/databusclient/api/deploy.py
@@ -249,7 +249,7 @@ def create_distribution(
return f"{url}|{meta_string}"
-def _create_distributions_from_metadata(
+def create_distributions_from_metadata(
metadata: List[Dict[str, Union[str, int]]],
) -> List[str]:
"""
@@ -524,7 +524,7 @@ def deploy_from_metadata(
Parameters
----------
metadata : List[Dict[str, Union[str, int]]]
- List of file metadata entries (see _create_distributions_from_metadata)
+ List of file metadata entries (see create_distributions_from_metadata)
version_id : str
Dataset version ID in the form $DATABUS_BASE/$ACCOUNT/$GROUP/$ARTIFACT/$VERSION
artifact_version_title : str
@@ -538,7 +538,7 @@ def deploy_from_metadata(
apikey : str
API key for authentication
"""
- distributions = _create_distributions_from_metadata(metadata)
+ distributions = create_distributions_from_metadata(metadata)
dataset = create_dataset(
version_id=version_id,
diff --git a/databusclient/cli.py b/databusclient/cli.py
index 14c03a1..fab57a9 100644
--- a/databusclient/cli.py
+++ b/databusclient/cli.py
@@ -10,6 +10,8 @@
from databusclient.api.download import download as api_download, DownloadAuthError
from databusclient.manifest.context import ManifestContext
from databusclient.manifest.writer import ManifestWriter
+from databusclient.manifest.replay import ManifestReplayError, replay_manifest, load_manifest
+from databusclient.manifest.summary import format_summary
from databusclient.extensions import webdav
@@ -138,6 +140,11 @@ def _write_manifest():
license_url=license_url,
distributions=distributions,
)
+ if manifest_context:
+ manifest_context.replay_params["deploy_mode"] = "classic"
+ manifest_context.replay_params["resolved_distributions"] = (
+ dataid["@graph"][-1].get("distribution", [])
+ )
api_deploy.deploy(dataid=dataid, api_key=apikey)
if manifest_context:
for dist in distributions:
@@ -146,7 +153,7 @@ def _write_manifest():
except Exception as exc:
if manifest_context:
manifest_context.record_operation_error(exc)
- raise
+ raise click.ClickException(str(exc))
finally:
_write_manifest()
return
@@ -155,8 +162,11 @@ def _write_manifest():
if metadata_file:
click.echo(f"[MODE] Deploy from metadata file: {metadata_file}")
try:
- with open(metadata_file, "r") as f:
+ with open(metadata_file, "r", encoding="utf-8-sig") as f:
metadata = json.load(f)
+ if manifest_context:
+ manifest_context.replay_params["deploy_mode"] = "metadata"
+ manifest_context.replay_params["resolved_metadata"] = metadata
api_deploy.deploy_from_metadata(
metadata, version_id, title, abstract, description, license_url, apikey
)
@@ -171,7 +181,7 @@ def _write_manifest():
except Exception as exc:
if manifest_context:
manifest_context.record_operation_error(exc)
- raise
+ raise click.ClickException(str(exc))
finally:
_write_manifest()
return
@@ -189,6 +199,8 @@ def _write_manifest():
)
click.echo("[MODE] Upload & Deploy to DBpedia Databus via Nextcloud")
click.echo(f"→ Uploading to: {remote}:{path}")
+ if manifest_context:
+ manifest_context.replay_params["deploy_mode"] = "webdav"
try:
metadata = webdav.upload_to_webdav(distributions, remote, path, webdav_url)
api_deploy.deploy_from_metadata(
@@ -205,7 +217,7 @@ def _write_manifest():
except Exception as exc:
if manifest_context:
manifest_context.record_operation_error(exc)
- raise
+ raise click.ClickException(str(exc))
finally:
_write_manifest()
return
@@ -444,6 +456,117 @@ def delete(databusuris: List[str], databus_key: str, dry_run: bool, force: bool,
err=True,
)
+@app.group()
+def manifest():
+ """
+ Manifest utilities.
+
+ Includes replay of previously recorded operations from JSON-LD manifests.
+ """
+ pass
+
+
+@manifest.command("replay")
+@click.argument("manifest_path", type=click.Path(exists=True, dir_okay=False))
+@click.option(
+ "--localdir",
+ default=None,
+ help="Override local output directory for download replay.",
+)
+@click.option(
+ "--databus",
+ default=None,
+ help="Override Databus endpoint for replay (example: https://databus.dbpedia.org/sparql).",
+)
+@click.option(
+ "--vault-token",
+ default=None,
+ help="Vault token file path required if manifest auth method is vault_token.",
+)
+@click.option(
+ "--databus-key",
+ default=None,
+ help="Databus API key required if manifest auth method is databus_key. "
+ "Also required for delete replay (never stored in the manifest).",
+)
+@click.option(
+ "--apikey",
+ "apikey",
+ default=None,
+ help="Databus API key required for deploy replay (never stored in the manifest).",
+)
+@click.option(
+ "--force",
+ is_flag=True,
+ default=False,
+ help="For delete replay: skip the interactive confirmation prompt. "
+ "Required for unattended/scripted replay of a delete operation.",
+)
+@click.option(
+ "--dry-run",
+ "dry_run",
+ is_flag=True,
+ default=False,
+ help="Force a dry-run preview even if the original operation wasn't "
+ "one. If the original delete WAS a dry run, replay already "
+ "previews automatically -- this flag cannot turn that off.",
+)
+def manifest_replay(manifest_path, localdir, databus, vault_token, databus_key, force, dry_run, apikey):
+ """
+ Replay a previously recorded manifest operation.
+
+ Currently supports replay of download, delete, and deploy manifests.
+ For delete manifests, an interactive confirmation is required by
+ default -- use --force to skip it for scripted/unattended use, or
+ --dry-run to preview without prompting or deleting. Deploy replay
+ supports classic and metadata-file modes only (not WebDAV).
+ """
+ overrides = {
+ "localDir": localdir,
+ "endpoint": databus,
+ "token": vault_token,
+ "databus_key": databus_key,
+ "api_key": apikey,
+ }
+ # Keep only explicitly provided text overrides
+ overrides = {k: v for k, v in overrides.items() if v is not None}
+ # Flags are always explicit values (False is a real, meaningful default)
+ overrides["force"] = force
+ overrides["dry_run"] = dry_run
+
+ try:
+ replay_info = replay_manifest(manifest_path, overrides=overrides)
+ if replay_info["command"] == "delete" and not replay_info.get("executed", True):
+ if replay_info.get("dry_run"):
+ click.echo("Delete replay: dry run only, nothing was deleted.")
+ else:
+ click.echo("Delete replay cancelled — nothing was deleted.")
+ else:
+ click.echo(f"Replayed command: {replay_info['command']}")
+ click.echo(f"Source manifest: {manifest_path}")
+ except ManifestReplayError as e:
+ raise click.ClickException(str(e))
+ except DownloadAuthError as e:
+ raise click.ClickException(str(e))
+ except ValueError as e:
+ raise click.ClickException(str(e))
+ except Exception as e:
+ raise click.ClickException(str(e))
+
+@manifest.command("summary")
+@click.argument("manifest_path", type=click.Path(exists=True, dir_okay=False))
+def manifest_summary(manifest_path):
+ """
+ Print a readable summary of a recorded manifest operation.
+
+ Reads the existing summary data already stored in the manifest --
+ no new data is collected and no new file is written.
+ """
+ try:
+ manifest = load_manifest(manifest_path)
+ click.echo(format_summary(manifest))
+ except ManifestReplayError as e:
+ raise click.ClickException(str(e))
if __name__ == "__main__":
app()
diff --git a/databusclient/manifest/__init__.py b/databusclient/manifest/__init__.py
index 791429d..3f7e892 100644
--- a/databusclient/manifest/__init__.py
+++ b/databusclient/manifest/__init__.py
@@ -2,4 +2,14 @@
Provides ManifestContext (records operation details in memory)
and ManifestWriter (serializes to JSON-LD on disk).
-"""
\ No newline at end of file
+"""
+from databusclient.manifest.context import ManifestContext
+from databusclient.manifest.writer import ManifestWriter
+from databusclient.manifest.replay import ManifestReplayError, replay_manifest
+
+__all__ = [
+ "ManifestContext",
+ "ManifestWriter",
+ "ManifestReplayError",
+ "replay_manifest",
+]
\ No newline at end of file
diff --git a/databusclient/manifest/replay.py b/databusclient/manifest/replay.py
new file mode 100644
index 0000000..0bb135c
--- /dev/null
+++ b/databusclient/manifest/replay.py
@@ -0,0 +1,313 @@
+from __future__ import annotations
+import json
+from typing import Any, Dict, Optional
+from databusclient.api.download import download as api_download
+from databusclient.api.delete import delete as api_delete
+from databusclient.api.deploy import (
+ create_dataset as api_create_dataset,
+ deploy as api_deploy_call,
+ create_distribution as api_create_distribution,
+ create_distributions_from_metadata,
+)
+
+
+class ManifestReplayError(Exception):
+ """Raised when replay manifest is invalid or cannot be replayed safely."""
+
+
+def load_manifest(path: str) -> Dict[str, Any]:
+ try:
+ with open(path, "r", encoding="utf-8-sig") as f:
+ data = json.load(f)
+ except FileNotFoundError as e:
+ raise ManifestReplayError(f"Manifest file not found: {path}") from e
+ except json.JSONDecodeError as e:
+ raise ManifestReplayError(f"Manifest is not valid JSON: {path}") from e
+ except OSError as e:
+ raise ManifestReplayError(f"Failed to read manifest: {e}") from e
+
+ if not isinstance(data, dict):
+ raise ManifestReplayError("Manifest root must be a JSON object.")
+ return data
+
+
+def _validate_replay_params(params: Any) -> Dict[str, Any]:
+ if not isinstance(params, dict):
+ raise ManifestReplayError("Manifest field dbus:replayParams must be an object.")
+ return params
+
+
+def _build_download_kwargs(
+ manifest: Dict[str, Any],
+ replay_params: Dict[str, Any],
+ overrides: Dict[str, Any],
+) -> Dict[str, Any]:
+ databus_uris = replay_params.get("databusURIs")
+ if not isinstance(databus_uris, list) or not databus_uris:
+ raise ManifestReplayError(
+ "Manifest replay for download requires non-empty replayParams.databusURIs."
+ )
+
+ auth_method = manifest.get("dbus:authMethod")
+ token = overrides.get("token")
+ databus_key = overrides.get("databus_key")
+
+ if auth_method == "vault_token" and not token:
+ raise ManifestReplayError(
+ "Manifest uses vault_token authentication. Provide --vault-token for replay."
+ )
+ if auth_method == "databus_key" and not databus_key:
+ raise ManifestReplayError(
+ "Manifest uses databus_key authentication. Provide --databus-key for replay."
+ )
+
+ endpoint = overrides.get("endpoint", manifest.get("dbus:endpoint"))
+
+ return {
+ "localDir": overrides.get("localDir"),
+ "endpoint": endpoint,
+ "databusURIs": databus_uris,
+ "token": token,
+ "databus_key": databus_key,
+ "all_versions": replay_params.get("all_versions"),
+ "auth_url": replay_params.get(
+ "authurl",
+ "https://auth.dbpedia.org/realms/dbpedia/protocol/openid-connect/token",
+ ),
+ "client_id": replay_params.get("clientid", "vault-token-exchange"),
+ "compression": replay_params.get("compression"),
+ "convert_format": replay_params.get("convert_format"),
+ "graph_name": replay_params.get("graph_name"),
+ "base_uri": replay_params.get("base_uri"),
+ "validate_checksum": bool(replay_params.get("validate_checksum", False)),
+ "manifest_context": None,
+ }
+
+def _build_delete_kwargs(
+ replay_params: Dict[str, Any],
+ overrides: Dict[str, Any],
+) -> Dict[str, Any]:
+ databus_uris = replay_params.get("databusURIs")
+ if not isinstance(databus_uris, list) or not databus_uris:
+ raise ManifestReplayError(
+ "Manifest replay for delete requires non-empty replayParams.databusURIs."
+ )
+
+ databus_key = overrides.get("databus_key")
+ if not databus_key:
+ raise ManifestReplayError(
+ "Delete replay requires --databus-key to be provided explicitly. "
+ "The API key is never stored in the manifest."
+ )
+
+ return {
+ "databusURIs": databus_uris,
+ "databus_key": databus_key,
+ }
+
+
+def _replay_delete(
+ replay_params: Dict[str, Any],
+ overrides: Dict[str, Any],
+ confirm_fn,
+) -> Dict[str, Any]:
+ kwargs = _build_delete_kwargs(replay_params, overrides)
+ databus_uris = kwargs["databusURIs"]
+
+ # dry_run reflects what actually happened in the ORIGINAL operation
+ # (already recorded in replayParams by the plain `delete` command).
+ # A replay-time --dry-run flag may only force it ON for an extra
+ # safety preview -- it can never turn a recorded dry_run=True back off.
+ recorded_dry_run = bool(replay_params.get("dry_run", False))
+ dry_run = recorded_dry_run or bool(overrides.get("dry_run", False))
+
+ # force is CLI-only at replay time, never read from the manifest --
+ # same pattern as vault_token / databus_key.
+ force = bool(overrides.get("force", False))
+
+ if dry_run:
+ api_delete(
+ databusURIs=databus_uris,
+ databus_key=kwargs["databus_key"],
+ dry_run=True,
+ force=True,
+ manifest_context=None,
+ )
+ return {"command": "delete", "executed": False, "dry_run": True}
+
+ if not force:
+ prompt = (
+ "About to replay a DELETE operation for the following "
+ f"{len(databus_uris)} URI(s):\n"
+ + "\n".join(f" - {u}" for u in databus_uris)
+ + "\nThis is irreversible. Proceed? [y/N]: "
+ )
+ answer = confirm_fn(prompt).strip().lower()
+ if answer not in ("y", "yes"):
+ return {"command": "delete", "executed": False, "dry_run": False}
+
+ # Replay-level confirmation already happened above (or --force was
+ # given). Call delete() with force=True so it doesn't also prompt
+ # per-resource internally -- avoids double confirmation.
+ api_delete(
+ databusURIs=databus_uris,
+ databus_key=kwargs["databus_key"],
+ dry_run=False,
+ force=True,
+ manifest_context=None,
+ )
+ return {"command": "delete", "executed": True, "dry_run": False}
+
+def _reconstruct_distribution_strings(resolved_distributions: list) -> list:
+ strings = []
+ for part in resolved_distributions:
+ url = part.get("downloadURL")
+ if not url:
+ raise ManifestReplayError(
+ "A stored distribution entry is missing 'downloadURL'; "
+ "cannot replay this deploy."
+ )
+ cvs = {
+ k[len("dcv:"):]: v
+ for k, v in part.items()
+ if k.startswith("dcv:")
+ }
+ sha256sum = part.get("sha256sum")
+ byte_size = part.get("byteSize")
+ sha_tuple = (
+ (sha256sum, byte_size)
+ if sha256sum and byte_size is not None
+ else None
+ )
+ strings.append(
+ api_create_distribution(
+ url=url,
+ cvs=cvs,
+ file_format=part.get("formatExtension"),
+ compression=part.get("compression"),
+ sha256_length_tuple=sha_tuple,
+ )
+ )
+ return strings
+
+
+def _build_deploy_kwargs(replay_params: Dict[str, Any]) -> Dict[str, Any]:
+ deploy_mode = replay_params.get("deploy_mode")
+
+ if deploy_mode == "webdav":
+ raise ManifestReplayError(
+ "Deploy replay is not supported for WebDAV/Nextcloud uploads. "
+ "The uploaded files may no longer exist at their original "
+ "local paths, so this mode cannot be safely replayed."
+ )
+
+ version_id = replay_params.get("version_id")
+ title = replay_params.get("title")
+ abstract = replay_params.get("abstract")
+ description = replay_params.get("description")
+ license_url = replay_params.get("license_url")
+
+ missing = [
+ name for name, val in [
+ ("version_id", version_id),
+ ("title", title),
+ ("abstract", abstract),
+ ("description", description),
+ ("license_url", license_url),
+ ] if not val
+ ]
+ if missing:
+ raise ManifestReplayError(
+ f"Manifest replay for deploy is missing required field(s): "
+ f"{', '.join(missing)}."
+ )
+
+ if deploy_mode == "classic":
+ resolved = replay_params.get("resolved_distributions")
+ if not resolved:
+ raise ManifestReplayError(
+ "Manifest replay for classic-mode deploy requires "
+ "replayParams.resolved_distributions."
+ )
+ distributions = _reconstruct_distribution_strings(resolved)
+ elif deploy_mode == "metadata":
+ metadata = replay_params.get("resolved_metadata")
+ if not metadata:
+ raise ManifestReplayError(
+ "Manifest replay for metadata-mode deploy requires "
+ "replayParams.resolved_metadata."
+ )
+ distributions = create_distributions_from_metadata(metadata)
+ else:
+ raise ManifestReplayError(
+ f"Manifest replay for deploy has an unknown or missing "
+ f"deploy_mode ('{deploy_mode}'). This manifest may predate "
+ "deploy replay support and cannot be replayed."
+ )
+
+ return {
+ "version_id": version_id,
+ "artifact_version_title": title,
+ "artifact_version_abstract": abstract,
+ "artifact_version_description": description,
+ "license_url": license_url,
+ "distributions": distributions,
+ }
+
+
+def _replay_deploy(
+ replay_params: Dict[str, Any],
+ overrides: Dict[str, Any],
+) -> Dict[str, Any]:
+ api_key = overrides.get("api_key")
+ if not api_key:
+ raise ManifestReplayError(
+ "Deploy replay requires --apikey to be provided explicitly. "
+ "The API key is never stored in the manifest."
+ )
+
+ kwargs = _build_deploy_kwargs(replay_params)
+ dataid = api_create_dataset(**kwargs)
+ api_deploy_call(dataid=dataid, api_key=api_key)
+ return {"command": "deploy", "executed": True}
+
+def replay_manifest(
+ manifest_path: str,
+ overrides: Optional[Dict[str, Any]] = None,
+ confirm_fn=input,
+) -> Dict[str, Any]:
+ """
+ Replay a previously recorded operation from a JSON-LD manifest.
+
+ Currently supported:
+ - download
+ - delete (interactive y/n confirmation by default; overrides "force"
+ skips the prompt, "dry_run" previews without prompting or deleting)
+ - deploy (classic and metadata modes only; WebDAV replay is not supported)
+
+ confirm_fn is injectable for testing (defaults to the built-in input()).
+ """
+ overrides = overrides or {}
+ manifest = load_manifest(manifest_path)
+
+ command = manifest.get("dbus:command")
+ if not command:
+ raise ManifestReplayError("Manifest missing required field dbus:command.")
+
+ replay_params = _validate_replay_params(manifest.get("dbus:replayParams"))
+
+ if command == "download":
+ kwargs = _build_download_kwargs(manifest, replay_params, overrides)
+ api_download(**kwargs)
+ return {"command": "download", "executed": True}
+
+ if command == "delete":
+ return _replay_delete(replay_params, overrides, confirm_fn)
+
+ if command == "deploy":
+ return _replay_deploy(replay_params, overrides)
+
+ raise ManifestReplayError(
+ f"Replay for command '{command}' is not implemented yet. "
+ "Currently supported: download, delete, deploy."
+ )
\ No newline at end of file
diff --git a/databusclient/manifest/summary.py b/databusclient/manifest/summary.py
new file mode 100644
index 0000000..914ebfe
--- /dev/null
+++ b/databusclient/manifest/summary.py
@@ -0,0 +1,84 @@
+"""manifest summary — formats an existing manifest as readable console output.
+
+Reads already-recorded fields from a manifest JSON-LD file and formats
+them for display. No new data is collected, no new file is written, and
+no network access happens here at all.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict
+
+
+def _format_bytes(total_bytes: int) -> str:
+ """Format a byte count as a human-readable MB value."""
+ mb = total_bytes / (1024 * 1024)
+ return f"{mb:.1f} MB"
+
+
+def _derive_status(manifest: Dict[str, Any]) -> str:
+ """Derive an overall status string for the operation.
+
+ - "failed" if a top-level dbus:operationError was recorded.
+ - "completed with errors" if some individual files failed but the
+ operation itself did not raise.
+ - "completed" otherwise.
+ """
+ if manifest.get("dbus:operationError"):
+ return "failed"
+
+ result = manifest.get("dbus:executionResult", {})
+ if result.get("dbus:failed", 0) > 0:
+ return "completed with errors"
+
+ return "completed"
+
+
+def format_summary(manifest: Dict[str, Any]) -> str:
+ """Format a loaded manifest dict as a readable multi-line summary string.
+
+ Args:
+ manifest: A manifest dict, as produced by loading a manifest
+ JSON-LD file (e.g. via replay._load_manifest).
+
+ Returns:
+ A formatted multi-line string ready to print to the console.
+ """
+ lines = []
+
+ command = manifest.get("dbus:command", "unknown")
+ lines.append(f"Command : {command}")
+
+ issued = manifest.get("dcterms:issued", {}).get("@value")
+ if issued:
+ lines.append(f"Executed : {issued}")
+
+ endpoint = manifest.get("dbus:endpoint")
+ if endpoint:
+ lines.append(f"Endpoint : {endpoint}")
+
+ auth_method = manifest.get("dbus:authMethod")
+ if auth_method:
+ lines.append(f"Auth : {auth_method}")
+
+ result = manifest.get("dbus:executionResult", {})
+ succeeded = result.get("dbus:succeeded", 0)
+ failed = result.get("dbus:failed", 0)
+ lines.append(f"Files : {succeeded} succeeded \u00b7 {failed} failed")
+
+ total_bytes = result.get("dbus:totalBytes", 0)
+ if command == "download" and total_bytes:
+ lines.append(f"Total : {_format_bytes(total_bytes)}")
+
+ lines.append(f"Status : {_derive_status(manifest)}")
+
+ operation_error = manifest.get("dbus:operationError")
+ if operation_error:
+ error_type = operation_error.get("dbus:errorType", "")
+ error_message = operation_error.get("dbus:errorMessage", "")
+ if error_type:
+ lines.append(f"Error : {error_type}: {error_message}")
+ else:
+ lines.append(f"Error : {error_message}")
+
+ return "\n".join(lines)
\ No newline at end of file
diff --git a/examples/reproducible-download.md b/examples/reproducible-download.md
index d5e1919..d59d8f6 100644
--- a/examples/reproducible-download.md
+++ b/examples/reproducible-download.md
@@ -36,4 +36,21 @@ Key fields:
Six months later, a colleague can verify the same data was
downloaded by checking the checksums in the manifest against
the files on disk, or by inspecting the `dbus:replayParams`
-to understand exactly what was fetched and when.
\ No newline at end of file
+to understand exactly what was fetched and when.
+
+## Actually reproducing it: replay
+
+Rather than manually re-typing the original command from the
+`dbus:replayParams` you inspected above, replay it directly:
+
+```bash
+databusclient manifest replay ./manifests/labels-download.jsonld --localdir ./data-replayed
+```
+
+This re-runs the download using the exact same parameters that
+were recorded — compression, format conversion, checksum
+validation, and so on — without needing to remember or
+reconstruct the original command by hand. Credentials
+(`--vault-token`/`--databus-key`) are never stored in the
+manifest and, if the original download needed them, must be
+supplied again here.
\ No newline at end of file
diff --git a/tests/test_manifest_replay.py b/tests/test_manifest_replay.py
new file mode 100644
index 0000000..59ed13d
--- /dev/null
+++ b/tests/test_manifest_replay.py
@@ -0,0 +1,448 @@
+import json
+import pytest
+from databusclient.manifest.replay import ManifestReplayError, replay_manifest
+
+def _write_manifest(path, payload):
+ with open(path, "w", encoding="utf-8") as f:
+ json.dump(payload, f)
+
+def test_replay_download_calls_api_download(tmp_path, monkeypatch):
+ captured = {}
+
+ def fake_download(**kwargs):
+ captured.update(kwargs)
+
+ monkeypatch.setattr("databusclient.manifest.replay.api_download", fake_download)
+
+ manifest = {
+ "@type": "dbus:OperationManifest",
+ "dbus:command": "download",
+ "dbus:endpoint": "https://databus.dbpedia.org/sparql",
+ "dbus:replayParams": {
+ "databusURIs": ["https://databus.dbpedia.org/dbpedia/test/artifact/2024.01.01"],
+ "all_versions": False,
+ "compression": "gz",
+ "convert_format": "turtle",
+ "graph_name": None,
+ "base_uri": None,
+ "validate_checksum": True,
+ "authurl": "https://auth.dbpedia.org/realms/dbpedia/protocol/openid-connect/token",
+ "clientid": "vault-token-exchange",
+ },
+ }
+
+ path = tmp_path / "run.jsonld"
+ _write_manifest(path, manifest)
+
+ result = replay_manifest(str(path))
+
+ assert result["command"] == "download"
+ assert captured["databusURIs"] == manifest["dbus:replayParams"]["databusURIs"]
+ assert captured["endpoint"] == "https://databus.dbpedia.org/sparql"
+ assert captured["compression"] == "gz"
+ assert captured["convert_format"] == "turtle"
+ assert captured["validate_checksum"] is True
+ assert captured["manifest_context"] is None
+
+
+def test_replay_missing_command_raises(tmp_path):
+ manifest = {
+ "@type": "dbus:OperationManifest",
+ "dbus:replayParams": {"databusURIs": ["https://example.org/data"]},
+ }
+ path = tmp_path / "missing-command.jsonld"
+ _write_manifest(path, manifest)
+
+ with pytest.raises(ManifestReplayError, match="dbus:command"):
+ replay_manifest(str(path))
+
+
+def test_replay_missing_replay_params_raises(tmp_path):
+ manifest = {
+ "@type": "dbus:OperationManifest",
+ "dbus:command": "download",
+ }
+ path = tmp_path / "missing-replay-params.jsonld"
+ _write_manifest(path, manifest)
+
+ with pytest.raises(ManifestReplayError, match="dbus:replayParams"):
+ replay_manifest(str(path))
+
+
+def test_replay_unsupported_command_raises(tmp_path):
+ manifest = {
+ "@type": "dbus:OperationManifest",
+ "dbus:command": "workflow",
+ "dbus:replayParams": {"version_id": "https://example.org/version"},
+ }
+ path = tmp_path / "unsupported.jsonld"
+ _write_manifest(path, manifest)
+
+ with pytest.raises(ManifestReplayError, match="not implemented"):
+ replay_manifest(str(path))
+
+
+def test_replay_requires_vault_token_when_manifest_auth_method_is_vault(tmp_path):
+ manifest = {
+ "@type": "dbus:OperationManifest",
+ "dbus:command": "download",
+ "dbus:authMethod": "vault_token",
+ "dbus:replayParams": {
+ "databusURIs": ["https://databus.dbpedia.org/dbpedia/test/artifact/2024.01.01"]
+ },
+ }
+ path = tmp_path / "vault-required.jsonld"
+ _write_manifest(path, manifest)
+
+ with pytest.raises(ManifestReplayError, match="--vault-token"):
+ replay_manifest(str(path))
+
+
+def test_replay_overrides_are_applied(tmp_path, monkeypatch):
+ captured = {}
+
+ def fake_download(**kwargs):
+ captured.update(kwargs)
+
+ monkeypatch.setattr("databusclient.manifest.replay.api_download", fake_download)
+
+ manifest = {
+ "@type": "dbus:OperationManifest",
+ "dbus:command": "download",
+ "dbus:endpoint": "https://old.example.org/sparql",
+ "dbus:authMethod": "databus_key",
+ "dbus:replayParams": {
+ "databusURIs": ["https://databus.dbpedia.org/dbpedia/test/artifact/2024.01.01"],
+ "validate_checksum": False,
+ },
+ }
+
+ path = tmp_path / "override.jsonld"
+ _write_manifest(path, manifest)
+
+ replay_manifest(
+ str(path),
+ overrides={
+ "endpoint": "https://databus.dbpedia.org/sparql",
+ "localDir": "./replay-data",
+ "databus_key": "dummy-key",
+ },
+ )
+
+ assert captured["endpoint"] == "https://databus.dbpedia.org/sparql"
+ assert captured["localDir"] == "./replay-data"
+ assert captured["databus_key"] == "dummy-key"
+
+def test_replay_delete_confirmed_calls_api_delete(tmp_path, monkeypatch):
+ captured = {}
+
+ def fake_delete(**kwargs):
+ captured.update(kwargs)
+
+ monkeypatch.setattr("databusclient.manifest.replay.api_delete", fake_delete)
+
+ manifest = {
+ "@type": "dbus:OperationManifest",
+ "dbus:command": "delete",
+ "dbus:replayParams": {
+ "databusURIs": ["https://databus.dbpedia.org/acct/grp/art/1.0"],
+ },
+ }
+ path = tmp_path / "delete-run.jsonld"
+ _write_manifest(path, manifest)
+
+ result = replay_manifest(
+ str(path),
+ overrides={"databus_key": "dummy-key"},
+ confirm_fn=lambda prompt: "y",
+ )
+
+ assert result == {"command": "delete", "executed": True, "dry_run": False}
+ assert captured["databusURIs"] == manifest["dbus:replayParams"]["databusURIs"]
+ assert captured["databus_key"] == "dummy-key"
+ assert captured["dry_run"] is False
+ assert captured["force"] is True # replay already confirmed, skip inner prompt
+
+
+def test_replay_delete_declined_does_not_call_api_delete(tmp_path, monkeypatch):
+ called = {"value": False}
+
+ def fake_delete(**kwargs):
+ called["value"] = True
+
+ monkeypatch.setattr("databusclient.manifest.replay.api_delete", fake_delete)
+
+ manifest = {
+ "@type": "dbus:OperationManifest",
+ "dbus:command": "delete",
+ "dbus:replayParams": {
+ "databusURIs": ["https://databus.dbpedia.org/acct/grp/art/1.0"],
+ },
+ }
+ path = tmp_path / "delete-decline.jsonld"
+ _write_manifest(path, manifest)
+
+ result = replay_manifest(
+ str(path),
+ overrides={"databus_key": "dummy-key"},
+ confirm_fn=lambda prompt: "n",
+ )
+
+ assert result == {"command": "delete", "executed": False, "dry_run": False}
+ assert called["value"] is False
+
+
+def test_replay_delete_force_skips_prompt(tmp_path, monkeypatch):
+ prompt_called = {"value": False}
+
+ def fake_delete(**kwargs):
+ pass
+
+ def fake_confirm(prompt):
+ prompt_called["value"] = True
+ return "n" # should never be reached
+
+ monkeypatch.setattr("databusclient.manifest.replay.api_delete", fake_delete)
+
+ manifest = {
+ "@type": "dbus:OperationManifest",
+ "dbus:command": "delete",
+ "dbus:replayParams": {
+ "databusURIs": ["https://databus.dbpedia.org/acct/grp/art/1.0"],
+ },
+ }
+ path = tmp_path / "delete-force.jsonld"
+ _write_manifest(path, manifest)
+
+ result = replay_manifest(
+ str(path),
+ overrides={"databus_key": "dummy-key", "force": True},
+ confirm_fn=fake_confirm,
+ )
+
+ assert result == {"command": "delete", "executed": True, "dry_run": False}
+ assert prompt_called["value"] is False
+
+
+def test_replay_delete_dry_run_from_manifest_skips_prompt(tmp_path, monkeypatch):
+ """dry_run recorded in the manifest (from the original delete run) is honored automatically."""
+ captured = {}
+ prompt_called = {"value": False}
+
+ def fake_delete(**kwargs):
+ captured.update(kwargs)
+
+ def fake_confirm(prompt):
+ prompt_called["value"] = True
+ return "y"
+
+ monkeypatch.setattr("databusclient.manifest.replay.api_delete", fake_delete)
+
+ manifest = {
+ "@type": "dbus:OperationManifest",
+ "dbus:command": "delete",
+ "dbus:replayParams": {
+ "databusURIs": ["https://databus.dbpedia.org/acct/grp/art/1.0"],
+ "dry_run": True,
+ },
+ }
+ path = tmp_path / "delete-dryrun-manifest.jsonld"
+ _write_manifest(path, manifest)
+
+ result = replay_manifest(
+ str(path),
+ overrides={"databus_key": "dummy-key"},
+ confirm_fn=fake_confirm,
+ )
+
+ assert result == {"command": "delete", "executed": False, "dry_run": True}
+ assert prompt_called["value"] is False
+ assert captured["dry_run"] is True
+
+
+def test_replay_delete_dry_run_override_forces_preview(tmp_path, monkeypatch):
+ """--dry-run at replay time can force a preview even if original wasn't a dry run."""
+ captured = {}
+ prompt_called = {"value": False}
+
+ def fake_delete(**kwargs):
+ captured.update(kwargs)
+
+ def fake_confirm(prompt):
+ prompt_called["value"] = True
+ return "y"
+
+ monkeypatch.setattr("databusclient.manifest.replay.api_delete", fake_delete)
+
+ manifest = {
+ "@type": "dbus:OperationManifest",
+ "dbus:command": "delete",
+ "dbus:replayParams": {
+ "databusURIs": ["https://databus.dbpedia.org/acct/grp/art/1.0"],
+ "dry_run": False,
+ },
+ }
+ path = tmp_path / "delete-dryrun-override.jsonld"
+ _write_manifest(path, manifest)
+
+ result = replay_manifest(
+ str(path),
+ overrides={"databus_key": "dummy-key", "dry_run": True},
+ confirm_fn=fake_confirm,
+ )
+
+ assert result == {"command": "delete", "executed": False, "dry_run": True}
+ assert prompt_called["value"] is False
+ assert captured["dry_run"] is True
+
+
+def test_replay_delete_requires_databus_key(tmp_path):
+ manifest = {
+ "@type": "dbus:OperationManifest",
+ "dbus:command": "delete",
+ "dbus:replayParams": {
+ "databusURIs": ["https://databus.dbpedia.org/acct/grp/art/1.0"],
+ },
+ }
+ path = tmp_path / "delete-no-key.jsonld"
+ _write_manifest(path, manifest)
+
+ with pytest.raises(ManifestReplayError, match="--databus-key"):
+ replay_manifest(str(path), overrides={})
+
+def test_replay_deploy_classic_mode(tmp_path, monkeypatch):
+ captured = {}
+
+ def fake_create_dataset(**kwargs):
+ captured["create_dataset_kwargs"] = kwargs
+ return {"@graph": [{"@id": "fake"}]}
+
+ def fake_deploy(dataid, api_key):
+ captured["deploy_dataid"] = dataid
+ captured["deploy_api_key"] = api_key
+
+ monkeypatch.setattr("databusclient.manifest.replay.api_create_dataset", fake_create_dataset)
+ monkeypatch.setattr("databusclient.manifest.replay.api_deploy_call", fake_deploy)
+
+ manifest = {
+ "@type": "dbus:OperationManifest",
+ "dbus:command": "deploy",
+ "dbus:replayParams": {
+ "version_id": "https://databus.dbpedia.org/acct/grp/art/1.0",
+ "title": "T", "abstract": "A", "description": "D",
+ "license_url": "https://license.example.org",
+ "deploy_mode": "classic",
+ "resolved_distributions": [
+ {
+ "downloadURL": "https://example.org/file.nt",
+ "formatExtension": "nt",
+ "compression": "none",
+ "byteSize": 123,
+ "sha256sum": "a" * 64,
+ }
+ ],
+ },
+ }
+ path = tmp_path / "deploy-classic.jsonld"
+ _write_manifest(path, manifest)
+
+ result = replay_manifest(str(path), overrides={"api_key": "dummy-key"})
+
+ assert result == {"command": "deploy", "executed": True}
+ assert captured["deploy_api_key"] == "dummy-key"
+ assert len(captured["create_dataset_kwargs"]["distributions"]) == 1
+ assert "a" * 64 in captured["create_dataset_kwargs"]["distributions"][0]
+
+
+def test_replay_deploy_metadata_mode(tmp_path, monkeypatch):
+ captured = {}
+
+ def fake_create_dataset(**kwargs):
+ captured["create_dataset_kwargs"] = kwargs
+ return {"@graph": [{"@id": "fake"}]}
+
+ def fake_deploy(dataid, api_key):
+ captured["deploy_api_key"] = api_key
+
+ monkeypatch.setattr("databusclient.manifest.replay.api_create_dataset", fake_create_dataset)
+ monkeypatch.setattr("databusclient.manifest.replay.api_deploy_call", fake_deploy)
+
+ manifest = {
+ "@type": "dbus:OperationManifest",
+ "dbus:command": "deploy",
+ "dbus:replayParams": {
+ "version_id": "https://databus.dbpedia.org/acct/grp/art/1.0",
+ "title": "T", "abstract": "A", "description": "D",
+ "license_url": "https://license.example.org",
+ "deploy_mode": "metadata",
+ "resolved_metadata": [
+ {
+ "checksum": "b" * 64,
+ "size": 456,
+ "url": "https://example.org/data.csv",
+ }
+ ],
+ },
+ }
+ path = tmp_path / "deploy-metadata.jsonld"
+ _write_manifest(path, manifest)
+
+ result = replay_manifest(str(path), overrides={"api_key": "dummy-key"})
+
+ assert result == {"command": "deploy", "executed": True}
+ assert len(captured["create_dataset_kwargs"]["distributions"]) == 1
+
+
+def test_replay_deploy_webdav_mode_raises(tmp_path):
+ manifest = {
+ "@type": "dbus:OperationManifest",
+ "dbus:command": "deploy",
+ "dbus:replayParams": {
+ "version_id": "https://databus.dbpedia.org/acct/grp/art/1.0",
+ "title": "T", "abstract": "A", "description": "D",
+ "license_url": "https://license.example.org",
+ "deploy_mode": "webdav",
+ },
+ }
+ path = tmp_path / "deploy-webdav.jsonld"
+ _write_manifest(path, manifest)
+
+ with pytest.raises(ManifestReplayError, match="WebDAV"):
+ replay_manifest(str(path), overrides={"api_key": "dummy-key"})
+
+
+def test_replay_deploy_requires_apikey(tmp_path):
+ manifest = {
+ "@type": "dbus:OperationManifest",
+ "dbus:command": "deploy",
+ "dbus:replayParams": {
+ "version_id": "https://databus.dbpedia.org/acct/grp/art/1.0",
+ "title": "T", "abstract": "A", "description": "D",
+ "license_url": "https://license.example.org",
+ "deploy_mode": "classic",
+ "resolved_distributions": [],
+ },
+ }
+ path = tmp_path / "deploy-no-key.jsonld"
+ _write_manifest(path, manifest)
+
+ with pytest.raises(ManifestReplayError, match="--apikey"):
+ replay_manifest(str(path), overrides={})
+
+
+def test_replay_deploy_missing_deploy_mode_raises(tmp_path):
+ """Backward-compat: manifests written before deploy replay existed have no deploy_mode."""
+ manifest = {
+ "@type": "dbus:OperationManifest",
+ "dbus:command": "deploy",
+ "dbus:replayParams": {
+ "version_id": "https://databus.dbpedia.org/acct/grp/art/1.0",
+ "title": "T", "abstract": "A", "description": "D",
+ "license_url": "https://license.example.org",
+ },
+ }
+ path = tmp_path / "deploy-old-manifest.jsonld"
+ _write_manifest(path, manifest)
+
+ with pytest.raises(ManifestReplayError, match="predate"):
+ replay_manifest(str(path), overrides={"api_key": "dummy-key"})
\ No newline at end of file
diff --git a/tests/test_manifest_summary.py b/tests/test_manifest_summary.py
new file mode 100644
index 0000000..9407449
--- /dev/null
+++ b/tests/test_manifest_summary.py
@@ -0,0 +1,117 @@
+from databusclient.manifest.summary import format_summary
+
+
+def test_summary_full_download_manifest():
+ manifest = {
+ "dbus:command": "download",
+ "dcterms:issued": {"@value": "2024-03-24T10:00:00Z"},
+ "dbus:endpoint": "https://databus.dbpedia.org",
+ "dbus:authMethod": "vault_token",
+ "dbus:executionResult": {
+ "dbus:succeeded": 1,
+ "dbus:failed": 0,
+ "dbus:totalBytes": 104857600,
+ },
+ }
+ output = format_summary(manifest)
+ assert "Command : download" in output
+ assert "Executed : 2024-03-24T10:00:00Z" in output
+ assert "Endpoint : https://databus.dbpedia.org" in output
+ assert "Auth : vault_token" in output
+ assert "Files : 1 succeeded \u00b7 0 failed" in output
+ assert "Total : 100.0 MB" in output
+ assert "Status : completed" in output
+
+
+def test_summary_omits_missing_optional_fields():
+ """Delete/deploy manifests without endpoint/authMethod omit those lines."""
+ manifest = {
+ "dbus:command": "delete",
+ "dcterms:issued": {"@value": "2024-03-24T10:00:00Z"},
+ "dbus:executionResult": {
+ "dbus:succeeded": 1,
+ "dbus:failed": 0,
+ "dbus:totalBytes": 0,
+ },
+ }
+ output = format_summary(manifest)
+ assert "Endpoint" not in output
+ assert "Auth" not in output
+ assert "Total" not in output
+ assert "Command : delete" in output
+ assert "Status : completed" in output
+
+
+def test_summary_shows_failed_files():
+ manifest = {
+ "dbus:command": "download",
+ "dcterms:issued": {"@value": "2024-03-24T10:00:00Z"},
+ "dbus:executionResult": {
+ "dbus:succeeded": 2,
+ "dbus:failed": 1,
+ "dbus:totalBytes": 2048,
+ },
+ }
+ output = format_summary(manifest)
+ assert "Files : 2 succeeded \u00b7 1 failed" in output
+ assert "Status : completed with errors" in output
+
+
+def test_summary_shows_operation_error_as_failed_status():
+ manifest = {
+ "dbus:command": "deploy",
+ "dcterms:issued": {"@value": "2024-03-24T10:00:00Z"},
+ "dbus:executionResult": {
+ "dbus:succeeded": 0,
+ "dbus:failed": 0,
+ "dbus:totalBytes": 0,
+ },
+ "dbus:operationError": {
+ "dbus:errorType": "DeployError",
+ "dbus:errorMessage": "bad API key",
+ },
+ }
+ output = format_summary(manifest)
+ assert "Status : failed" in output
+
+
+def test_summary_total_omitted_for_non_download_command():
+ manifest = {
+ "dbus:command": "deploy",
+ "dcterms:issued": {"@value": "2024-03-24T10:00:00Z"},
+ "dbus:executionResult": {
+ "dbus:succeeded": 1,
+ "dbus:failed": 0,
+ "dbus:totalBytes": 12345,
+ },
+ }
+ output = format_summary(manifest)
+ assert "Total" not in output
+
+def test_summary_includes_error_message_when_operation_failed():
+ manifest = {
+ "dbus:command": "deploy",
+ "dcterms:issued": {"@value": "2024-03-24T10:00:00Z"},
+ "dbus:executionResult": {
+ "dbus:succeeded": 0,
+ "dbus:failed": 0,
+ "dbus:totalBytes": 0,
+ },
+ "dbus:operationError": {
+ "dbus:errorType": "DeployError",
+ "dbus:errorMessage": "Could not deploy dataset to databus. Reason: 'Invalid API key'",
+ },
+ }
+ output = format_summary(manifest)
+ assert "Status : failed" in output
+ assert "Error : DeployError: Could not deploy dataset to databus. Reason: 'Invalid API key'" in output
+
+
+def test_summary_no_error_line_when_no_operation_error():
+ manifest = {
+ "dbus:command": "download",
+ "dcterms:issued": {"@value": "2024-03-24T10:00:00Z"},
+ "dbus:executionResult": {"dbus:succeeded": 1, "dbus:failed": 0, "dbus:totalBytes": 100},
+ }
+ output = format_summary(manifest)
+ assert "Error" not in output
\ No newline at end of file