Skip to content

GSoC2026‐DhanashreePetare

Jump to bottom
Dhanashree Petare edited this page Aug 31, 2026 · 5 revisions

GSoC 2026 Final Report — DBpedia Databus Python Client

Contributor: Dhanashree Petare (@DhanashreePetare)

Organization: DBpedia

Mentors: Fabian Hofer, Marvin Hofer

Project: DBpedia Databus Python Client - Manifest-Driven Reproducible Workflow Orchestration

Goals

The Databus Python Client is DBpedia's command-line and Python interface for downloading and deploying datasets on the DBpedia Databus. Before this project, the client supported basic download, deploy, and delete operations, but had no way to convert between RDF/tabular formats during download, no way to record what an operation actually did, and no way to chain multiple operations into a repeatable pipeline.

The goal of this project was to close those gaps: add format and mapping conversion during download, build a structured manifest system to record and later replay any operation, and add a declarative workflow engine so multi-step pipelines (download → deploy → delete) could be defined once in YAML and run repeatably and unattended.

Full proposal: [ProjectProposal](will add here link from github itself rather than using drive link after the merge)

What I Did

The work was delivered across five milestones, each shipped as its own reviewed pull request:

  1. Format and Mapping Conversion Layer — added on-the-fly RDF triple, RDF quad, and tabular (CSV/TSV) format conversion during download (--format), alongside the existing compression conversion (--compression), bringing the Python client closer to feature parity with the Java client.

  2. Structured Run Manifest System — added an optional --manifest flag to download, deploy, and delete. When used, the client records a JSON-LD file (using the DataID vocabulary) capturing input parameters, resolved file URLs, checksums, byte sizes, and per-file success/failure status — with credentials never written to disk.

  3. Manifest Replay and Summary — databusclient manifest replay <path> re-executes a previously recorded operation from its saved manifest, for all three commands (download, delete, deploy), with careful handling of destructive operations (delete replay requires interactive confirmation by default) and deploy modes (classic and metadata-file deploys are replayable; WebDAV deploys are not, since their original local files may no longer exist). databusclient manifest summary <path> prints a readable summary of any manifest without touching the network.

  4. Declarative Workflow Engine — added databusclient workflow run <file.yml>, letting a user define a sequence of download/deploy/delete steps in YAML, with outputs from one step referenceable by later steps (${steps.name.output_files} / ${steps.name.output_urls}), and per-step error handling (fail, continue, or retry with configurable attempts and delay).

  5. Workflow-Manifest Integration and Example Workflows — extended the manifest system so a full workflow run produces one unified manifest covering every step (each file entry tagged with which step produced it), added an automatic console summary after every workflow run, and added eight real, tested example workflow files covering reproducible downloads, unattended nightly pipelines, batch deployment with retry, CI/CD-safe pipelines, and failure-debugging output.

A final documentation restructuring pass followed, splitting the previously large root README.md into a slim entry point plus detailed doc/cli-usage.md and doc/module-usage.md pages, per mentor feedback, to keep the project approachable for new users.

Current State

All five milestones are complete, tested, and merged into the gsoc-2026 branch of the upstream repository, which has since been merged into main. Every feature was manually verified against the live DBpedia Databus in addition to the automated test suite (172 tests passing).

What's Left

  • run_all_conversion_tests.py (a manual integration-test script under tests/manual/) has a CLI end-to-end verification section that is intentionally run manually against live Databus data rather than automated, since it exercises real network downloads; this remains a manual step by design, not an oversight.

What Code Got Merged Upstream

All work for this project was merged into dbpedia/databus-python-client:

  • Format and Mapping Conversion Layer — #62, #63, #66
  • Structured Run Manifest System — #67
  • Manifest Replay and Summary — #81
  • Declarative Workflow Engine — #84
  • Workflow-Manifest Integration and Example Workflows — #85
  • Documentation Restructuring — #86
  • Final merge into main — #88

Challenges and What I Learned

Working on an existing, actively-maintained CLI tool taught me a different discipline than building something from scratch: every change had to work alongside code I didn't write, without breaking existing behavior, and every design decision (like what a manifest should and shouldn't store, or how a delete-replay should default to safety) needed to be checked against real mentor input rather than assumed.

One specific bug stood out: a workflow that downloaded data, deployed it, then deleted the original version could silently break the deployed distribution's link, because the recorded URL pointed at a Databus wrapper that resolved via HTTP redirect rather than the real, stable source. Finding and fixing this — by capturing the actually-resolved URL at download time instead of the original input URI — was a good lesson in not trusting an operation "worked" just because it completed without error; I had to independently verify the deployed data itself matched what was expected before trusting the fix.

I also learned the value of getting design decisions confirmed explicitly before implementing them, rather than guessing at intent — several points in the project (deploy-replay behavior, delete-replay safety defaults, how a workflow step should handle locally-modified files) turned out to have real tradeoffs that needed a deliberate decision, not an assumption.

Thank you to my mentors, Fabian Hofer and Marvin Hofer, for their guidance, thorough code review, and patience throughout this project.

Links

Clone this wiki locally