Skip to content

feat(workspace): InsForge + S3 migration toolkit#363

Closed
baryhuang wants to merge 1 commit into
openagents-org:developfrom
baryhuang:feat/insforge-s3-migration-toolkit
Closed

feat(workspace): InsForge + S3 migration toolkit#363
baryhuang wants to merge 1 commit into
openagents-org:developfrom
baryhuang:feat/insforge-s3-migration-toolkit

Conversation

@baryhuang
Copy link
Copy Markdown
Collaborator

@baryhuang baryhuang commented Apr 26, 2026

Summary

Self-contained scripts and runbook for cloning a workspace from the hosted OpenAgents backend into a self-hosted InsForge Postgres + AWS S3 stack. No backend code changes required — point the existing app at the new DB/S3 and it just works.

  • 0001_initial_schema.sql mirrors workspace/backend/app/models.py 1:1 and stamps alembic_version='007' so entrypoint.sh's alembic upgrade head is a no-op against pre-seeded DBs (works on managed InsForge tiers that don't run migrations).
  • migrate.py pumps workspace + members + collaborators + channels + channel_members + events + files via the source's existing /v1 API, upserts into InsForge via PostgREST, and uploads blobs to S3 using the exact key shape S3FileStore.save() expects. Resumable via .migration-state.json; tolerates phantom channels (404 on detail) and orphaned source blobs (5xx on download).
  • MIGRATION_GUIDE.md is the executable runbook covering the gotchas: existing agents stop working until reconnected, alembic stamping, ECS task vs execution role split, silent CloudWatch logs, PostgREST upsert headers, and the no-managed-migrations constraint on cloud InsForge.
  • CONVERSATION.md is the short list of natural-language prompts that drove the InsForge side end-to-end — bootstrap, project sanity-check, plan, password lookup, phase approvals.

All new files live under workspace/scripts/insforge-migration/ — zero impact on the running backend, frontend, or CI.

Test plan

  • Run migrate.py end-to-end against a hosted source workspace into a fresh InsForge + S3 target, then point a backend instance at the new DB/S3 and verify the workspace loads.
  • Re-run migrate.py after partial completion and confirm .migration-state.json resumes correctly without duplicating rows or re-uploading blobs.
  • Apply 0001_initial_schema.sql to an empty Postgres, then start the backend and confirm alembic upgrade head is a no-op (alembic_version already '007').
  • Verify migration tolerates a workspace containing at least one phantom channel (404) and one orphaned blob (5xx) without aborting.
  • Reconnect at least one agent against the new endpoint and confirm new messages land in the migrated stack.

@vercel
Copy link
Copy Markdown

vercel Bot commented Apr 26, 2026

@baryhuang is attempting to deploy a commit to the Raphael's projects Team on Vercel.

A member of the Team first needs to authorize it.

@baryhuang baryhuang force-pushed the feat/insforge-s3-migration-toolkit branch 4 times, most recently from 7141ee1 to 4a66a85 Compare April 26, 2026 06:12
Self-contained scripts and runbook for cloning a workspace from the hosted
OpenAgents backend into your own InsForge Postgres + AWS S3 stack.

- 0001_initial_schema.sql mirrors workspace/backend/app/models.py 1:1 so the
  backend can be repointed at the new DB with no code changes; stamps
  alembic_version='007' so entrypoint.sh's `alembic upgrade head` no-ops.
- migrate.py pumps workspace + members + collaborators + channels +
  channel_members + events + files via the source's existing /v1 API,
  upserting into InsForge through PostgREST and uploading blobs to S3 with
  the exact key shape S3FileStore.save() reads from. Resumable via
  .migration-state.json; tolerates phantom channels (404 on detail) and
  orphaned source blobs (5xx on download).
- MIGRATION_GUIDE.md is the executable runbook with all the gotchas
  (existing agents stop working until reconnected, alembic stamp, task
  role separate from execution role, silent CloudWatch logs, PostgREST
  upsert headers, no managed migrations on cloud InsForge tier).
- CONVERSATION.md is the short list of natural-language prompts that drove
  the InsForge side end-to-end — bootstrap, project sanity-check, plan,
  password lookup, phase approvals.
@baryhuang baryhuang force-pushed the feat/insforge-s3-migration-toolkit branch from 4a66a85 to 74fc3ba Compare April 26, 2026 06:13
@baryhuang
Copy link
Copy Markdown
Collaborator Author

This is a migration documentation PR for migrating from openagents to insforge + AWS s3. Ready to review

@zomux
Copy link
Copy Markdown
Contributor

zomux commented May 4, 2026

Thanks @baryhuang — the toolkit landed on develop in 6cfc575. Closing this PR.

Heads up: the bundling into 6cfc575 was incidental (a local working-tree mistake on our end while reviewing — the commit message describes only the claude-adapter fix that shipped alongside it). Your authorship credit didn't make it onto the merged commit; sorry about that. The next time you push to this area we'll make sure your name is on the commit.

A few small follow-ups we'd appreciate as a separate, tiny PR (non-blocking — happy to do these ourselves if you'd rather):

  • .env.example line 2: replace the real workspace UUID 4e8162ec-… with <your-source-workspace-uuid>
  • .env.example line 7: S3_BUCKET=openagents-files-peakmojo → generic placeholder
  • MIGRATION_GUIDE.md line 11 + line 363 reference customer names ("Peakmojo Team", caremojo.app) — worth genericizing for a public runbook
  • MIGRATION_GUIDE.md line 297 links to baryhuang/openagents#5 (your fork) — should point to the upstream issue if there is one

We reviewed the code carefully (schema diffed against live Postgres, SQL applied to a fresh DB, Python parsed clean) and verified zero impact on the running backend. Great runbook — the gotchas section is going to save someone a lot of pain.

@zomux zomux closed this May 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants