Skip to content

GSoC2026‐DhanashreePetare

Jump to bottom
Dhanashree Petare edited this page Sep 1, 2026 · 5 revisions

GSoC 2026 Final Report: DBpedia Databus Python Client

Contributor: Dhanashree Petare (@DhanashreePetare)

Organization: DBpedia

Mentors: Fabian Hofer, Marvin Hofer

Project: Manifest-Driven Reproducible Workflow Orchestration, DBpedia Databus Python Client

Goals

The Databus Python Client is DBpedia's command-line and Python interface for downloading and deploying datasets on the DBpedia Databus. Before this project, the client supported basic download, deploy, and delete operations, but had no way to convert between RDF/tabular formats during download, no way to record what an operation actually did, and no way to chain multiple operations into a repeatable pipeline.

The goal of this project was to close those gaps: add format and mapping conversion during download, build a structured manifest system to record and later replay any operation, and add a declarative workflow engine so multi-step pipelines (download → deploy → delete) could be defined once in YAML and run repeatably and unattended.

Full proposal: Project Proposal

What I Did

The work was delivered across five milestones, each reviewed and merged as one or more pull requests:

  1. Format and Mapping Conversion Layer: Added on-the-fly RDF triple, RDF quad, and tabular (CSV/TSV) format conversion during download (--format), alongside the existing compression conversion (--compression). This brings the Python client closer to feature parity with the Java client.

  2. Structured Run Manifest System: Added an optional --manifest flag to download, deploy, and delete. When used, the client records a JSON-LD file, using the DataID vocabulary, that captures input parameters, resolved file URLs, checksums, byte sizes, and per-file success/failure status. Credentials are never written to disk.

  3. Manifest Replay and Summary: databusclient manifest replay <path> re-executes a previously recorded operation from its saved manifest, for all three commands. Delete replay requires interactive confirmation by default. Deploy replay works for classic and metadata-file deploys; WebDAV deploys aren't replayable, since their original local files may no longer exist. databusclient manifest summary <path> prints a readable summary of any manifest without touching the network.

  4. Declarative Workflow Engine: Added databusclient workflow run <file.yml>, letting a user define a sequence of download, deploy, and delete steps in YAML. Outputs from one step can be referenced by later steps (${steps.name.output_files} or ${steps.name.output_urls}), and each step can be set to fail, continue, or retry on error, with configurable attempts and delay.

  5. Workflow-Manifest Integration and Example Workflows: Extended the manifest system so a full workflow run produces one unified manifest covering every step, with each file entry tagged by which step produced it. Added an automatic console summary after every workflow run, and eight tested example workflow files covering reproducible downloads, unattended nightly pipelines, batch deployment with retry, CI/CD-safe pipelines, and failure-debugging output.

A final documentation restructuring pass followed, splitting the previously large root README.md into a slim entry point plus detailed doc/cli-usage.md and doc/module-usage.md pages.

Current State

All five milestones are complete, tested, and merged into main (172 tests passing). Every feature was manually verified against the live DBpedia Databus in addition to the automated test suite.

What's Left

  • WebDAV deploy replay. Deploy replay currently supports classic and metadata-file deploys, but not WebDAV/Nextcloud uploads, since the originally uploaded local files may no longer exist by replay time. Extending replay to handle this (for example, by re-uploading from a workflow's still-available local output) is a natural next step.
  • A wider set of official example workflows. The five Milestone 5 example workflows cover the core proposal use cases; a few more real-world scenarios (for example, a full CI/CD pipeline definition, not just a CI-safe workflow file) would make the workflow engine easier to adopt for new users.
  • run_all_conversion_tests.py (a manual integration-test script under tests/manual/) has a CLI end-to-end verification section that is intentionally run manually against live Databus data rather than automated, since it exercises real network downloads; this remains a manual step by design, not an oversight.
  • Community feedback. Now that this is part of the released client, real usage from the DBpedia community may surface edge cases or feature requests the original proposal didn't anticipate.

What Code Got Merged Upstream

All work for this project was merged into dbpedia/databus-python-client:

  • Format and Mapping Conversion Layer: #62, #63, #78
  • Structured Run Manifest System: #67
  • Manifest Replay and Summary: #81
  • Declarative Workflow Engine: #84
  • Workflow-Manifest Integration and Example Workflows: #85
  • Documentation Restructuring: #87
  • Final merge into main: #88

Challenges and What I Learned

Working on an existing, actively-maintained CLI tool taught me a different discipline than building something from scratch: every change had to work alongside code I didn't write, without breaking existing behavior, and every design decision (like what a manifest should and shouldn't store, or how a delete-replay should default to safety) needed to be checked against edge cases rather than assumed.

One specific bug stood out: a workflow that downloaded data, deployed it, then deleted the original version could silently break the deployed distribution's link, because the recorded URL pointed at a Databus wrapper that resolved via HTTP redirect rather than the real, stable source. Finding and fixing this, by capturing the actually-resolved URL at download time instead of the original input URI — was a good lesson in not trusting an operation "worked" just because it completed without error; I had to independently verify the deployed data itself matched what was expected before trusting the fix.

I also learned the value of getting design decisions confirmed explicitly before implementing them, rather than guessing at intent — several points in the project (deploy-replay behavior, delete-replay safety defaults, how a workflow step should handle locally-modified files) turned out to have real tradeoffs that needed a deliberate decision and not an assumption.

Thank you to my mentors, Fabian Hofer and Marvin Hofer, for their guidance, thorough code reviews, and patience throughout this project.

Links

Clone this wiki locally