Design: Pure Pony Git Implementation

[SeanTAllen]

Design companion to the motivation document. This is the high-level approach and phased implementation plan. Each phase will get its own detailed plan as we go.

Approach

Replace GitVCS internals with a pure Pony git library. The VCS, RepoOperation, Repo, and TagListReceiver interfaces stay unchanged. A new PonyGitVCS implements the same three operations — sync, tag_query, checkout — using pure Pony code instead of shelling out to git.

During implementation, a hybrid GitVCS delegates replaced operations to pure Pony and unreplaced operations to the existing process-based implementation. Operations are replaced one at a time in order of increasing complexity:

1. tag_query (ref reading — file I/O only)
2. checkout (object reading + tree walking + file extraction)
3. sync (HTTP clone/fetch — the hardest)

Architecture

New code lives under corral/git/:

corral/git/
  inflate/      — Pure Pony RFC 1951 decompression
  sha1/         — SHA-1 wrapper (thin layer over ssl's Digest)
  objects/      — Git object types + reading (loose + packed)
  pack/         — Packfile + pack index parsing, delta resolution
  refs/         — Ref reading (packed-refs + refs/ directory)
  checkout/     — Tree walking + blob extraction to filesystem
  protocol/     — Git smart HTTP client (using vendored courier)

Vendored courier stack (courier, lori, ssl, logger — 4 packages, ~131 source files) lives alongside the existing source. Exact vendoring location TBD in Phase 1 detailed plan.

The VCS interface is the integration boundary. PonyGitVCS implements it, returning RepoOperation objects that do pure Pony git work. The _Updater actor, command layer, and all non-git VCS stubs remain unchanged.

What we're not building

• No git index: Corral never commits to repos. Checkout walks commit → tree → blobs directly, bypassing the index file entirely.
• No SSH or git:// transport: HTTPS only, matching current behavior.
• No deflate (compression): All operations are read-only. We decompress objects but never create them.
• No pure Pony SHA-1: The vendored ssl package provides Digest.sha1() via OpenSSL/LibreSSL EVP. Since ssl is already required for HTTPS, using it for SHA-1 adds no new dependency.

Phases

Phase 1: Vendor Courier Stack + Foundations

Vendor courier, lori, ssl, and logger into the corral repo. Update the build system so corral compiles with the vendored packages. Existing tests must continue to pass — this phase adds dependencies but changes no behavior.

Then implement the two foundational components:

• Pure Pony inflate: RFC 1951 DEFLATE decompression, RFC 1950 zlib stream framing, Adler-32 checksum. ~800-1200 lines. This is the biggest single component and the foundation for reading any git object.
• SHA-1 wrapper: Thin API over ssl's Digest class providing sha1(data): Array[U8] val and sha1_hex(data): String val. Isolates the ssl dependency to one place.

Deliverable: Vendored packages building cleanly. Inflate and SHA-1 with comprehensive test suites. No git integration yet.

Risk: Inflate is the highest-risk component in the entire project. Huffman table decoding, LZ77 back-references, fixed vs. dynamic code tables — subtle bugs here produce silent data corruption downstream. This must be thoroughly tested before proceeding.

Phase 2: Git Object Layer

Read local git data:

• Loose objects: Decompress .git/objects/xx/yyyy... files, verify SHA-1, parse type size\0content header.
• Packfiles: Parse PACK format header and entries. Resolve OFS_DELTA and REF_DELTA objects (recursive delta chain application). Parse .idx index files for object lookup.
• Refs: Read .git/packed-refs (text format) and .git/refs/ directory tree. Resolve tag names to SHA-1 hashes.
• Object types: Parse commit objects (tree hash, parent hashes, author/committer, message), tree objects (mode + name + SHA-1 entries), blob objects (raw content), tag objects.

Deliverable: A GitObjectDB that, given a .git directory path, can resolve any ref to an object and read any object by SHA-1. Testable against real git repositories.

Risk: Packfile delta resolution is the second highest-risk component. Delta chains can be deep, and incorrect application produces silent corruption. Need test vectors covering OFS_DELTA, REF_DELTA, and multi-level chains.

Phase 3: Checkout + Hybrid Integration

Two pieces: the tree walker and the first integration point.

• Tree walking + extraction: Given a commit SHA-1 (or ref name), read its tree, recursively walk it, read blob contents, write files to the workspace directory with correct paths. This replaces the git reset --mixed + git checkout-index two-step.
• Hybrid integration: Modify GitVCS so that tag_query_op reads refs directly via Phase 2's ref reader (instead of git show-ref), and checkout_op uses the tree walker (instead of git reset + git checkout-index). sync_op continues to use the git CLI.

Deliverable: tag_query_op and checkout_op running pure Pony. sync_op still shells out to git. Integration tests pass.

Phase 4: Smart HTTP Protocol + Full Integration

Git smart HTTP client using vendored courier:

• Reference discovery: GET /info/refs?service=git-upload-pack, parse capability advertisement and ref list.
• Pack negotiation: POST /git-upload-pack with want/have lines. For initial clone: want all refs, have nothing. For fetch: want new refs, report existing refs as haves.
• Pack data reception: Receive packfile stream with sideband demultiplexing (data, progress, error channels). Write packfile + build index.
• Ref update: Write fetched refs to local .git/refs/ or .git/packed-refs.

Replace sync_op: PonyGitSync uses HTTP to clone (create local .git + fetch all refs + download pack) and fetch (negotiate + download incremental pack + update refs). Remove git CLI dependency. GitVCS becomes PonyGitVCS with no process-based operations.

Deliverable: Git CLI no longer required. All integration tests pass against real GitHub repos.

Risk: Highest-complexity phase. The smart HTTP protocol has subtle details (sideband multiplexing, capability negotiation, packfile framing). This is also where authentication for private repos becomes necessary.

Phase Summary

Phase	Delivers	Replaces	Risk
1	Vendored deps, inflate, SHA-1	—	High (inflate correctness)
2	Object DB, packfiles, refs	—	High (delta resolution)
3	Tree walk + file extraction	tag_query + checkout	Medium
4	Smart HTTP clone/fetch	sync	High (protocol complexity)

Phases are sequential (each depends on the prior). Estimated new code: ~2500-4000 lines across the git packages, plus ~131 files of vendored courier/lori/ssl/logger.

Design Notes

• Error reporting via ActionResult.fail: The existing ActionResult class already has a fail(errmsg) constructor used for non-process errors (e.g., GitCheckoutRepo uses it when directory creation fails). Pure Pony operations use this same path — no interface changes needed.
• Remote URL construction: repo.remote is a bare path (e.g., github.com/ponylang/corral.git). The current _clone prepends https:// — the pure Pony HTTP client must do the same. _fetch doesn't use the remote URL at all (git remembers it), but pure Pony fetch must always receive it from repo.remote.
• Local .git directory creation: Currently git clone creates the .git structure. Pure Pony clone (Phase 4) must create .git/, .git/objects/pack/, .git/refs/, etc. explicitly.
• Tag fetching: _fetch uses --tags to fetch all tags. Pure Pony fetch must replicate this — tags are the primary data corral needs for semver resolution.

Open Questions

These don't need answers now — each phase's detailed plan will address them:

• Authentication: Git CLI handles credentials transparently (credential helpers, .netrc, etc.). Pure Pony HTTP needs its own auth story. At minimum, HTTPS basic auth via environment variable or config. Details deferred to Phase 4.
• Logger name collision: Corral has its own corral/logger/ package. Courier depends on a separate logger package. These will collide. Resolution options: rename the vendored logger, restructure the vendor path, or adapt courier to use corral's logger. Addressed in Phase 1 detailed plan.
• Vendoring mechanics: Where vendored packages live in the repo, how the Makefile passes package paths to ponyc (-p flag or placement within the source tree). Addressed in Phase 1 detailed plan.

What Stays Unchanged

• VCS / RepoOperation / Repo / TagListReceiver / RepoOperationResultReceiver interfaces
• _Updater actor and command dispatch layer
• Runner / Program / Action / ActionResult infrastructure (still needed for PostFetchScript)
• VCSBuilder interface and other VCS stubs (hg/bzr/svn/none)
• Bundle model, locator parsing, semver constraint solving
• Integration test structure (tests run against real repos)

What Changes Beyond the Git Library

• CorralVCSBuilder implementation: switches from constructing GitVCS to PonyGitVCS (or hybrid during transition). The VCSBuilder interface itself doesn't change.
• corral.json packages list: new corral/git/ sub-packages and vendored packages must be registered.
• CLAUDE.md source layout documentation: updated to reflect the new corral/git/ tree and vendored packages.
• Makefile: vendored package paths, potentially new build/test targets per phase.

[SeanTAllen]

Phase 1 detailed plan posted: #309