Skip to content

feat: add documentation site and clean up public API namespace#82

Merged
eywalker merged 22 commits intodevfrom
eywalker/plt-911-add-documentation-for-orcapod-python
Mar 14, 2026
Merged

feat: add documentation site and clean up public API namespace#82
eywalker merged 22 commits intodevfrom
eywalker/plt-911-add-documentation-for-orcapod-python

Conversation

@eywalker
Copy link
Copy Markdown
Contributor

@eywalker eywalker commented Mar 13, 2026

Summary

  • Public API refactor: Slimmed root orcapod namespace to 3 core exports (FunctionPod, function_pod, Pipeline); everything else now lives in clean subpackage re-exports (orcapod.sources, orcapod.operators, orcapod.nodes, orcapod.streams, orcapod.databases, orcapod.types)
  • MkDocs documentation site: Built with Material for MkDocs + mkdocstrings. Includes home page with PyPI/GitHub install instructions, getting started guide, 5 concept pages, and auto-generated API reference from docstrings
  • Concept pages (reviewed & confirmed accurate):
    • Sources — SourceStreamBuilder-based architecture, all concrete source types
    • Streams — immutable (Tag, Packet) model, schema inspection, iteration
    • Operators — Join (commutative), MergeJoin (commutative), SemiJoin (non-commutative), Batch, column selection/renaming, PolarsFilter
    • Function Pods@function_pod decorator, FunctionNode DB-backed caching, __call__ shorthand
    • Identity & Hashingcontent_hash() as recursive source-inclusive identity (not raw data), pipeline_hash() as schema+topology only, Merkle chain, resolver pattern
  • Project rules: Added Linear issue template, sys.modules manipulation prohibition, and branch naming conventions to CLAUDE.md and .zed/rules
  • Housekeeping: .gitignore additions (superpowers/, /*.json), removed stale pipeline/nodes.py stub

Test plan

  • All 2044 tests pass (uv run pytest tests/)
  • Doc site builds cleanly (uv run mkdocs build)
  • Imports via new subpackage paths work (from orcapod.sources import DictSource, etc.)
  • Concept pages reviewed for technical accuracy in browser (uv run mkdocs serve)
  • API reference renders correctly

Resolves PLT-911

🤖 Generated with Claude Code

eywalker and others added 15 commits March 13, 2026 19:09
Only export FunctionPod, function_pod, and Pipeline at the root level.
All other classes are now accessible via subpackage imports (e.g.
orcapod.sources.ArrowTableSource, orcapod.databases.InMemoryArrowDatabase).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Register orcapod.sources/operators/nodes/streams in sys.modules so
`from orcapod.sources import X` works, and update all consumer files
to use the new import style.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Create orcapod/sources/, orcapod/operators/, orcapod/nodes/, and
orcapod/streams/ as thin re-export packages so that
`from orcapod.sources import X` works through standard Python
import resolution.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…t rules

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings March 13, 2026 21:53
@codecov-commenter
Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

eywalker and others added 2 commits March 13, 2026 21:55
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a MkDocs-based documentation site while restructuring the public API so the root orcapod namespace is slim and functionality is accessed via dedicated subpackages (e.g., orcapod.sources, orcapod.operators).

Changes:

  • Refactors public imports to prefer subpackages; updates tests/examples accordingly.
  • Introduces a Material for MkDocs site with concept docs and mkdocstrings-driven API reference.
  • Updates dev tooling/deps and project contributor rules; removes older pipeline design/plan documents.

Reviewed changes

Copilot reviewed 33 out of 35 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
uv.lock Updates locked dependencies to include MkDocs tooling and remove Sphinx-related packages.
pyproject.toml Replaces Sphinx dev dependency with MkDocs Material + mkdocstrings.
mkdocs.yml Adds MkDocs configuration (theme, plugins, nav).
src/orcapod/init.py Slims root exports and exposes subpackages for the new public import surface.
src/orcapod/sources/init.py Adds public re-export package for orcapod.core.sources.
src/orcapod/operators/init.py Adds public re-export package for orcapod.core.operators.
src/orcapod/nodes/init.py Adds public re-export package for orcapod.core.nodes.
src/orcapod/streams/init.py Adds public re-export package for orcapod.core.streams.
tests/test_channels/test_pipeline_async_integration.py Updates imports to use the new orcapod.sources location.
examples/save_and_load_pipelines.py Adds an example demonstrating pipeline save/load with the new import style.
examples/async_vs_sync_pipeline.py Updates example imports and formats long print lines.
docs/index.md Adds documentation home page and quick-start example.
docs/getting-started.md Adds a getting started guide covering sources/streams/function pods/operators.
docs/concepts/sources.md Adds “Sources” concept doc describing source types and provenance.
docs/concepts/streams.md Adds “Streams” concept doc describing core stream model and inspection APIs.
docs/concepts/operators.md Adds “Operators” concept doc describing available structural operators.
docs/concepts/function-pods.md Adds “Function Pods” concept doc describing decorator usage and FunctionNode.
docs/concepts/identity.md Adds “Identity & Hashing” concept doc explaining content vs pipeline hashes.
docs/api/index.md Adds API reference landing page describing package structure.
docs/api/sources.md Adds mkdocstrings-based API reference page for sources.
docs/api/streams.md Adds mkdocstrings-based API reference page for streams.
docs/api/operators.md Adds mkdocstrings-based API reference page for operators.
docs/api/function-pods.md Adds mkdocstrings-based API reference page for function pods / packet functions.
docs/api/databases.md Adds mkdocstrings-based API reference page for databases.
docs/api/nodes.md Adds mkdocstrings-based API reference page for nodes.
docs/api/pipeline.md Adds mkdocstrings-based API reference page for Pipeline.
docs/api/types.md Adds mkdocstrings-based API reference page for core types.
docs/superpowers/specs/2026-03-12-pipeline-simplification-design.md Removes large internal design spec document.
docs/superpowers/specs/2026-03-12-pipeline-serialization-design.md Removes large internal design spec document.
docs/superpowers/plans/2026-03-12-pipeline-simplification.md Removes large internal implementation plan document.
docs/superpowers/plans/2026-03-12-pipeline-serialization.md Removes large internal implementation plan document.
CLAUDE.md Adds contributor rules (e.g., prohibit sys.modules hacks; Linear issue template).
.zed/rules Mirrors contributor rules for Zed editor configuration.
.gitignore Removes Sphinx build ignore entries.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread docs/concepts/sources.md Outdated
eywalker and others added 5 commits March 14, 2026 02:08
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All standard pods and operators support __call__ as a shorthand for
.process(), giving a more ergonomic feel. Updated examples throughout
the docs to use this syntax.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…t raw data

content_hash includes the identity of sources recursively, not
necessarily raw data values. In-memory sources hash data directly;
storage-backed sources hash canonical identity. pipeline_hash ignores
source identity entirely, caring only about schemas and topology.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sources use SourceStreamBuilder for enrichment, not delegation to
ArrowTableSource. Updated concept page and API reference to reflect
the builder-based architecture.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@eywalker eywalker merged commit 4fcb63e into dev Mar 14, 2026
4 checks passed
@eywalker eywalker deleted the eywalker/plt-911-add-documentation-for-orcapod-python branch March 14, 2026 02:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants