Skip to content

Design: Indexing pipeline deployment on Cloudflare + Fly.io#4

Merged
planeyang merged 1 commit into
masterfrom
planeyang/indexing-pipeline-design
Feb 26, 2026
Merged

Design: Indexing pipeline deployment on Cloudflare + Fly.io#4
planeyang merged 1 commit into
masterfrom
planeyang/indexing-pipeline-design

Conversation

@planeyang
Copy link
Copy Markdown
Collaborator

Summary

  • Design docs for separating the annotation pipeline (indexing) from the web viewer (presentation)
  • Architecture: Cloudflare Pages + Workers + D1 + R2 for the viewer, Fly.io Machine for the Python pipeline
  • Storage design: D1 for queryable metadata (repos, files, jobs, costs), R2 for large JSON blobs (annotations, analysis)
  • 4-phase implementation plan: storage layer → Worker + Pages → queue + runner → progress UI

Documents

  • docs/features/indexing-pipeline/README.md — overview
  • docs/features/indexing-pipeline/architecture.md — system diagram, components, data flows, security
  • docs/features/indexing-pipeline/storage.md — D1 schema, R2 key layout, access patterns
  • docs/features/indexing-pipeline/implementation.md — phased rollout plan

Test plan

  • Review architecture diagram and component responsibilities
  • Review D1 schema and R2 key layout for completeness
  • Review implementation phases and dependencies
  • Validate Cloudflare free tier limits against expected usage

🤖 Generated with Claude Code

Design for separating the annotation pipeline from the viewer,
deploying on Cloudflare (Pages + Workers + D1 + R2) with a
Fly.io machine for heavy Python pipeline work.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@planeyang planeyang merged commit c10abd7 into master Feb 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant