feat(checkpoint): remote-checkpoint indexing → load → deploy pipeline#2329
Merged
Conversation
Productionize indexing a release candidate on disposable Cherry Servers hardware and loading the resulting schemas into a target ENSDb, cutting a full index from ~1 week to ~6 hours. The deployed indexers resume the warm schema rather than re-indexing. - ensdb-cli: dump/load ENSIndexer schemas + their ENSNode metadata - ensindexer-checkpoint: Cherry/R2 box orchestration (full-backfill + end-block modes) - workflows ([next]-prefixed): index_checkpoint_and_deploy, checkpoint (dev), deploy_ensapi_hotfix, reaper - environment-defaults: alpha → searchlight/1 + unigraph; committed configs/*.env parity contract - deploy_ensnode_blue_green: optional image_tag input (decouple image tag from schema version)
🦋 Changeset detectedLatest commit: 7220f4f The changes in this PR will be included in the next version bump. This PR includes changesets to release 23 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
Warning Review limit reached
Next review available in: 53 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThis PR updates alpha environment defaults (adds ChangesAlpha Environment Defaults
ensdb-sdk public API expansion and ensdb-cli TypeScript package
ensindexer-checkpoint orchestration package and CI workflows
Sequence Diagram(s)sequenceDiagram
participant GHA as GitHub Actions
participant checkpoint_sh as checkpoint.sh
participant CherryAPI as Cherry Servers API
participant remote_checkpoint as remote-checkpoint.sh
participant R2 as Cloudflare R2
participant ponder as ponder indexer
participant ensdb_cli as ensdb-cli
GHA->>checkpoint_sh: trigger (CONFIGS, SHA, MODE, TARGET_URL)
checkpoint_sh->>CherryAPI: POST /projects/:id/servers
CherryAPI-->>checkpoint_sh: server id + IP
checkpoint_sh->>remote_checkpoint: ssh on_box source .run-env && remote-checkpoint.sh
remote_checkpoint->>R2: acquire_lock + check existing checkpoint dumps
remote_checkpoint->>ponder: rehydrate ponder_sync from R2 seed, start per-config indexers
ponder-->>remote_checkpoint: is_ready=1
remote_checkpoint->>ensdb_cli: dump schema + write .metadata.json sidecar
remote_checkpoint->>R2: upload dump + metadata sidecar
remote_checkpoint->>ensdb_cli: load dump into TARGET_URL (DO_LOAD=1)
remote_checkpoint->>R2: refresh canonical seed (DO_SEED=1)
remote_checkpoint->>R2: release_lock
remote_checkpoint-->>checkpoint_sh: CHECKPOINT_DONE_OK
checkpoint_sh->>CherryAPI: DELETE /servers/:id (teardown)
Estimated code review effort🎯 5 (Critical) | ⏱️ ~120 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a production-oriented “remote checkpoint” pipeline that indexes a release-candidate on disposable Cherry Servers, exports the resulting ENSIndexer schema + ENSNode metadata to R2, optionally loads it into a target ENSDb, and then deploys the stack so alpha/mainnet resume from a warm schema rather than re-indexing.
Changes:
- Introduces two new private packages:
@ensnode/ensdb-cli(schema dump/load + ENSNode metadata) and@ensnode/ensindexer-checkpoint(Cherry/R2-based orchestration scripts). - Adds committed named ENSIndexer identity configs (
apps/ensindexer/configs/*.env) plus drift-guard tests; updates alpha defaults to includeunigraphandsearchlight/1. - Adds
[next]GitHub workflows for checkpointing/indexing + load + deploy, a dev checkpoint workflow, a reaper safety net, and extends blue/green deploy to accept an optionalimage_tag.
Reviewed changes
Copilot reviewed 40 out of 41 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| pnpm-lock.yaml | Locks dependencies for the new packages and updated dependency graph. |
| packages/ensindexer-checkpoint/scripts/remote-seed-export.sh | Dumps ponder_sync and refreshes the canonical R2 seed (plus purges older seed objects). |
| packages/ensindexer-checkpoint/scripts/remote-run.sh | Rehydrates seed, checks out a commit, runs ENSRainbow + ENSIndexer to is_ready=1, then graceful-stops. |
| packages/ensindexer-checkpoint/scripts/remote-resolve-end-blocks.sh | Resolves deterministic per-chain end blocks from ponder_sync for dev checkpoints. |
| packages/ensindexer-checkpoint/scripts/remote-rehydrate.sh | Initializes/mounts box storage, starts a tuned local Postgres cluster, restores R2 seed. |
| packages/ensindexer-checkpoint/scripts/remote-provision.sh | Provisions the box toolchain (Node/pnpm/pg/rclone, etc.). |
| packages/ensindexer-checkpoint/scripts/remote-checkpoint.sh | On-box orchestrator: lock → produce/reuse checkpoint → optional load → optional seed refresh. |
| packages/ensindexer-checkpoint/scripts/lib.sh | Shared helpers for SSH/scp, R2 paths/locking, and Postgres lifecycle. |
| packages/ensindexer-checkpoint/scripts/detect-done.sh | Reads Ponder readiness (_ponder_meta.app.is_ready) for completion detection. |
| packages/ensindexer-checkpoint/scripts/config.example.sh | Environment-driven config template for CI/manual runs (no committed secrets). |
| packages/ensindexer-checkpoint/scripts/cherry-up.sh | Provisions Cherry box, records id/ip, arms self-destruct watchdog. |
| packages/ensindexer-checkpoint/scripts/cherry-down.sh | Terminates the Cherry box and cleans up local state. |
| packages/ensindexer-checkpoint/scripts/checkpoint.sh | Manual end-to-end runner (up → ship → provision → run → down). |
| packages/ensindexer-checkpoint/scripts/.shellcheckrc | Shellcheck configuration for these scripts. |
| packages/ensindexer-checkpoint/README.md | Package documentation and operational overview of the checkpoint pipeline. |
| packages/ensindexer-checkpoint/package.json | Declares the private package and its lint scripts. |
| packages/ensindexer-checkpoint/.gitignore | Ignores secrets/state files (config.sh, .box-id, .box-host). |
| packages/ensdb-cli/vitest.config.ts | Vitest config for the CLI package. |
| packages/ensdb-cli/tsup.config.ts | Bundling config for the CLI binary output. |
| packages/ensdb-cli/tsconfig.json | TypeScript config for the CLI package. |
| packages/ensdb-cli/src/main.ts | Defines the CLI’s top-level command and subcommands. |
| packages/ensdb-cli/src/lib/pgtools.ts | pg_dump/pg_restore helpers including identifier quoting and TOC parsing. |
| packages/ensdb-cli/src/lib/pgtools.test.ts | Unit tests for pgtools quoting/arg building/TOC parsing. |
| packages/ensdb-cli/src/lib/exec.ts | spawn() wrapper to run external tools and capture output/errors. |
| packages/ensdb-cli/src/lib/ensdb.ts | ENSDb SDK helpers for schema ops and metadata dump/load. |
| packages/ensdb-cli/src/commands/load.ts | Restores a dump, optional schema rename, optional metadata upsert. |
| packages/ensdb-cli/src/commands/dump.ts | Dumps a schema and optional ENSNode metadata sidecar JSON. |
| packages/ensdb-cli/src/cli.ts | Node shebang entrypoint that runs the citty command. |
| packages/ensdb-cli/package.json | Declares the private CLI package, binary, scripts, deps. |
| apps/ensindexer/src/config/named-configs.test.ts | Drift-guard tests to ensure committed configs match EnvironmentDefaults. |
| apps/ensindexer/src/config/environment-defaults.ts | Updates alpha defaults (adds unigraph + searchlight/1 label set). |
| apps/ensindexer/src/config/config.test.ts | Updates expected default client label set values. |
| apps/ensindexer/configs/README.md | Documents “named config” identity env files and build_id parity contract. |
| apps/ensindexer/configs/mainnet.env | Committed identity env vars for mainnet (subgraph-compatible). |
| apps/ensindexer/configs/alpha.env | Committed identity env vars for alpha (mainnet namespace + efp, searchlight/1). |
| .github/workflows/index_checkpoint_and_deploy.yml | New workflow: resolve RC → build images → checkpoint+load → dispatch blue/green deploy. |
| .github/workflows/deploy_ensnode_blue_green.yml | Adds optional image_tag input to decouple image tag from schema version. |
| .github/workflows/deploy_ensapi_hotfix.yml | New workflow for ENSApi-only hotfix deploy without reindex. |
| .github/workflows/checkpoint.yml | New workflow to produce point-in-time dev checkpoints (end-block mode). |
| .github/workflows/checkpoint_reaper.yml | Scheduled safety workflow to terminate stale checkpoint boxes. |
| .changeset/alpha-defaults-unigraph-searchlight.md | Changeset for updated alpha defaults behavior. |
Files not reviewed (1)
- pnpm-lock.yaml: Generated file
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
Contributor
Greptile SummaryThis PR productionizes the ENSIndexer checkpoint pipeline: instead of re-indexing from scratch on Railway (~1 week), it indexes
Confidence Score: 5/5Safe to merge. All manual-trigger-only workflows, no auto-run paths, and the single scheduled job (GC reaper) only ever deletes ensindexer-checkpoint-* boxes — never production infrastructure. Every previously flagged finding has been addressed in the current HEAD: ENSRainbow gate requires both DB marker and live health; end-block resolution uses a temp file with explicit exit-code propagation; the Cherry API token ships via a 600 file; redeploy_service uses --fail --show-error; set_shared_variable inspects the response body for GraphQL errors; load.ts fails fast on an unresolvable dump schema. The remaining comments are diagnostic display nits, a masked watchdog startup guard, and a curl-pipe-bash supply chain note — none affect the functional correctness of the checkpoint-index-load-deploy flow. No files require blocking attention. cherry-up.sh (watchdog || warn guard masked by &) and remote-provision.sh (curl | bash for Node/rclone) are worth a second look for hardening but do not affect correctness. Important Files Changed
Sequence Diagram%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant GH as GitHub Actions
participant Cherry as Cherry Servers
participant R2 as Cloudflare R2
participant Target as Target ENSDb
participant Railway as Railway
GH->>GH: resolve RC commit + version + image_tag
GH->>GH: build-images 4x sha-tagged to GHCR
par Phase 1 deploy-light
GH->>Railway: set sepolia schema vars
GH->>Railway: deploy ENSRainbow + sepolia indexers + ENSApi
and Phase 2 index-checkpoint on Cherry
GH->>Cherry: cherry-up provision box
GH->>Cherry: ship scripts + configs + rclone.conf + run-env 600
Cherry->>R2: download ponder_sync seed rehydrate Postgres
par alpha
Cherry->>Cherry: ensrainbow searchlight ponder alpha
Cherry-->>Cherry: "is_ready=1"
and mainnet
Cherry->>Cherry: ensrainbow subgraph ponder mainnet
Cherry-->>Cherry: "is_ready=1"
end
Cherry->>Cherry: pg_dump schemas
Cherry->>R2: upload sha-keyed dumps + sidecars
Cherry->>R2: upload enriched ponder_sync seed
GH->>Cherry: cherry-down terminate box
end
GH->>GH: Phase 2b load on plain runner
GH->>R2: download checkpoints
GH->>Target: ensdb-cli load alphaSchema + mainnetSchema
GH->>Railway: Phase 3 set ALPHA MAINNET schema vars
GH->>Railway: deploy heavy indexers
Railway->>Target: resume from warm schemas
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant GH as GitHub Actions
participant Cherry as Cherry Servers
participant R2 as Cloudflare R2
participant Target as Target ENSDb
participant Railway as Railway
GH->>GH: resolve RC commit + version + image_tag
GH->>GH: build-images 4x sha-tagged to GHCR
par Phase 1 deploy-light
GH->>Railway: set sepolia schema vars
GH->>Railway: deploy ENSRainbow + sepolia indexers + ENSApi
and Phase 2 index-checkpoint on Cherry
GH->>Cherry: cherry-up provision box
GH->>Cherry: ship scripts + configs + rclone.conf + run-env 600
Cherry->>R2: download ponder_sync seed rehydrate Postgres
par alpha
Cherry->>Cherry: ensrainbow searchlight ponder alpha
Cherry-->>Cherry: "is_ready=1"
and mainnet
Cherry->>Cherry: ensrainbow subgraph ponder mainnet
Cherry-->>Cherry: "is_ready=1"
end
Cherry->>Cherry: pg_dump schemas
Cherry->>R2: upload sha-keyed dumps + sidecars
Cherry->>R2: upload enriched ponder_sync seed
GH->>Cherry: cherry-down terminate box
end
GH->>GH: Phase 2b load on plain runner
GH->>R2: download checkpoints
GH->>Target: ensdb-cli load alphaSchema + mainnetSchema
GH->>Railway: Phase 3 set ALPHA MAINNET schema vars
GH->>Railway: deploy heavy indexers
Railway->>Target: resume from warm schemas
Reviews (18): Last reviewed commit: "ci: split next-gen blue/green deploy int..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 22
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
.github/workflows/deploy_ensnode_blue_green.yml (1)
28-41: 🛠️ Refactor suggestion | 🟠 Major | ⚡ Quick winAdd a concurrency group to prevent simultaneous deploys to the same target.
This deploy workflow modifies Railway services but has no concurrency protection. Concurrent runs targeting the same environment could cause race conditions in service updates.
♻️ Suggested fix
Add after line 26 (after
permissions:):concurrency: group: deploy-ensnode-${{ inputs.target }} cancel-in-progress: false🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/deploy_ensnode_blue_green.yml around lines 28 - 41, The deploy-environment job lacks a concurrency group configuration, which could allow simultaneous deployments to the same target environment, causing race conditions. Add a concurrency block to the deploy-environment job that groups deployments by the target environment name using inputs.target and sets cancel-in-progress to false to prevent cancelling already-in-progress deployments. Insert this concurrency section after the permissions field in the deploy-environment job definition.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/checkpoint_reaper.yml:
- Around line 8-14: The workflow checkpoint_reaper.yml lacks a concurrency group
configuration which can result in overlapping hourly runs if the reaper takes
longer than an hour to complete. Add a concurrency section to the workflow after
the permissions block that specifies a unique concurrency group name (such as
one derived from the workflow name) with cancel-in-progress set to true. This
will ensure that when a new scheduled run is triggered, any previously running
instance is cancelled, preventing resource waste and rate limit issues.
In @.github/workflows/checkpoint.yml:
- Around line 57-69: The free-form inputs chain_ids, commit, and timestamp are
being directly interpolated into the shell script using ${{ }} syntax, which
creates a shell injection vulnerability if these inputs contain shell
metacharacters. To fix this, define these inputs as environment variables in the
GitHub Actions workflow before the run step executes the shell script. For
example, add an env section to the step that maps inputs.chain_ids,
inputs.commit, and inputs.timestamp to environment variables, then reference
these environment variables within the shell script using standard shell
variable syntax (e.g., $CHAIN_IDS instead of ${{ inputs.chain_ids }}) to ensure
they are treated as literal strings rather than being evaluated by the shell.
- Around line 82-96: In the "Report checkpoint R2 location" step, the OBJ
variable is directly interpolating user inputs (inputs.config, inputs.commit,
and inputs.timestamp) into the shell script, creating a shell injection
vulnerability. Instead of using direct interpolation with ${{ }}, define these
inputs as environment variables at the top of the step (similar to the pattern
used elsewhere), and then reference those environment variables in the shell
script to safely construct the OBJ variable and subsequent commands that use it.
In @.github/workflows/deploy_ensapi_hotfix.yml:
- Around line 8-26: Add a concurrency group to the deploy_ensapi_hotfix workflow
to prevent race conditions when simultaneous hotfix deploys target the same
environment. Use the workflow input variable (target) as the concurrency group
key to ensure only one deploy runs per target at a time. Additionally, remove
the unused id-token: write permission from the permissions section since the
workflow does not perform OIDC authentication, keeping only the contents: read
permission that is actually needed.
In @.github/workflows/index_checkpoint_and_deploy.yml:
- Around line 162-170: The gh workflow run command in the "Dispatch blue/green
deploy" step can fail silently and still exit with status 0 (success) if the
workflow is not found or permissions are insufficient, causing the deploy job to
incorrectly succeed even though no deployment was triggered. Add the --json flag
to the gh workflow run command to output structured JSON, then capture and
validate the response to ensure the workflow dispatch actually succeeded before
allowing the step to complete. If the dispatch fails, the step should exit with
a non-zero status to properly fail the job.
- Around line 23-29: Remove the unused id-token: write permission from the
workflow-level permissions block in the index_checkpoint_and_deploy.yml file.
The workflow only needs contents: read and actions: write permissions for the gh
CLI and downstream workflow dispatch operations, so the id-token: write line
should be deleted to reduce unnecessary privilege escalation and improve the
security posture of the workflow.
- Around line 45-72: In the "Resolve RC commit, version, and image tag" step,
the inputs.commit value is being directly interpolated into the shell script.
Pass this user input safely through an environment variable instead. Add an env
section to the step that captures inputs.commit as an environment variable, then
reference that environment variable (e.g., $COMMIT_INPUT) in the shell script
where the condition checks for inputs.commit and assigns it to the SHA variable,
rather than using direct template interpolation ${{ inputs.commit }}.
In `@apps/ensindexer/src/config/named-configs.test.ts`:
- Around line 18-20: The code in the test parser does not validate that
trimmed.indexOf("=") returns a valid index before using the eq variable for
string slicing. Add a check after calculating eq to ensure it is not -1; if
indexOf("=") returns -1 (meaning no equals sign was found), throw an error or
skip the line to fail fast instead of silently creating corrupted key/value
pairs with the slice operations. This will prevent malformed env file content
from being silently processed and potentially hiding test failures.
In `@packages/ensdb-cli/src/commands/load.ts`:
- Around line 75-77: Remove the unsafe `as MetadataRow[]` type cast in the
metadata file parsing block. Instead, parse the JSON from the file and validate
it against a Zod schema for MetadataRow[] before passing the rows to the
loadMetadata() function. Define or import a Zod schema that validates the
structure of MetadataRow objects, then use Zod's parse method to validate the
JSON result. This ensures malformed metadata files fail immediately with clear
validation errors rather than silently accepting invalid data.
In `@packages/ensdb-cli/src/lib/ensdb.ts`:
- Around line 60-77: The loadMetadata function processes multiple metadata rows
in a loop without transactional protection, meaning if any row insertion fails,
the function will have already committed previous rows, leaving the database in
a partial/inconsistent state. Wrap the entire for loop that iterates over rows
and performs the insert operations in a single database transaction by using
writer.ensDb.transaction() to ensure all metadata rows are written atomically as
an all-or-nothing operation.
In `@packages/ensdb-cli/src/lib/exec.ts`:
- Around line 12-31: The run() function currently has no timeout protection,
which allows spawned subprocesses (like pg_dump or pg_restore) to block
indefinitely if they stall due to network issues, locks, or hung processes. Add
a timeout mechanism to the Promise returned by run() that will reject with an
appropriate error if the child process does not complete within a reasonable
time limit (consider a configurable timeout value). This should be implemented
alongside the existing event handlers so that if the timeout fires before the
child process closes, the promise is rejected and the child process is
terminated to prevent wedging checkpoint jobs.
In `@packages/ensindexer-checkpoint/scripts/checkpoint.sh`:
- Around line 23-25: The trap cleanup EXIT command is currently registered after
cherry-up.sh executes, but since set -e is inherited from lib.sh, if
cherry-up.sh fails, the script exits immediately without registering the trap,
leaving the provisioned Cherry box orphaned. Move the trap cleanup EXIT line to
execute before the bash "$LIB_DIR/cherry-up.sh" call to ensure that the cleanup
function is registered and will execute on exit regardless of whether
cherry-up.sh succeeds or fails.
- Around line 47-53: The on_box function call contains a shell injection
vulnerability where variables are enclosed in single quotes without proper
escaping. Single quotes prevent variable expansion but do not escape embedded
single quotes, backticks, or dollar signs in the variable values themselves,
allowing injection attacks. Fix this by using printf '%q' to properly escape
each variable (MODE, CONFIG, SHA, SCHEMA, ALCHEMY_API_KEY, DO_LOAD, TARGET_URL,
TARGET_SCHEMA, DO_SEED, TIMESTAMP, CHAIN_IDS, and CKPT_SUFFIX) before they are
passed to the on_box command, which will safely handle any special characters or
quotes within the values.
In `@packages/ensindexer-checkpoint/scripts/cherry-down.sh`:
- Around line 15-17: The rm -f command that removes "$LIB_DIR/.box-id" and
"$LIB_DIR/.box-host" executes regardless of whether the curl DELETE call
succeeds or fails. If the DELETE operation fails, the local state files should
not be removed, as this destroys the reference to the running server.
Restructure the script so that the file removal is conditional and only executes
after a successful DELETE call, allowing retries to work properly if the initial
termination attempt fails.
In `@packages/ensindexer-checkpoint/scripts/cherry-up.sh`:
- Around line 24-40: The server ID is persisted to .box-id immediately after
creation, but if the provisioning loop times out waiting for the active state
and IP address, there is no cleanup mechanism to delete the orphaned server. Add
a rollback function that deletes the Cherry server by making a DELETE API call
to $API/servers/$id before the die command that handles the timeout failure
condition when the 120 iteration loop completes without achieving both active
state and a valid IP.
In `@packages/ensindexer-checkpoint/scripts/detect-done.sh`:
- Around line 9-11: Remove the error suppression (2>/dev/null) from both psql
queries that set the ready and c1 variables so that database errors and query
failures are visible instead of being hidden. Additionally, remove the default
parameter expansion patterns (:-?) from the echo statement that fall back to ?
when variables are empty, or replace them with explicit error checking that
fails the script if either query returns no result. This will allow the script
to fail fast when the database is unavailable or the schema is missing, rather
than silently returning ? values that cause the polling loop in remote-run.sh to
run indefinitely.
In `@packages/ensindexer-checkpoint/scripts/lib.sh`:
- Line 25: The SSH_OPTS configuration is effectively disabling host key
verification by using UserKnownHostsFile=/dev/null, which prevents SSH from
persisting and checking known hosts on subsequent connections. Remove the
UserKnownHostsFile=/dev/null parameter from the SSH_OPTS array to allow SSH to
properly store and verify host keys in the default known_hosts file, which will
enable proper MITM protection for all SSH operations using these options.
- Around line 70-82: The acquire_lock() function has a race condition where it
checks lock existence with r2_exists and then writes with rclone rcat in
separate operations, allowing concurrent processes to both pass the check. Fix
this by implementing token ownership verification: write the lock token (the
timestamp stored in variable now) to the lock file path stored in variable p,
then immediately read it back and verify that the value matches what was
written; if the read-back value differs, another process has overwritten the
lock and you should fail or retry appropriately.
In `@packages/ensindexer-checkpoint/scripts/remote-checkpoint.sh`:
- Around line 20-24: Add validation checks for MODE, DO_SEED, and CKPT_SUFFIX
after the variable initialization section to prevent unsafe operations. First,
add a check that verifies when MODE is set to "end-block", the CKPT_SUFFIX
variable must not be empty to avoid checkpoint key collisions with full-backfill
runs. Second, add a check that restricts DO_SEED to only be set to 1 when CONFIG
is "alpha", preventing undefined behavior on non-alpha canonical R2 seeds. Use
die() to terminate with clear error messages for each validation failure,
inserting these checks after the CKPT_SUFFIX assignment.
In `@packages/ensindexer-checkpoint/scripts/remote-resolve-end-blocks.sh`:
- Around line 14-25: The TIMESTAMP and chain_id values (variable c) are being
used unquoted directly in the SQL query on the line with the pg command without
numeric validation, which creates a security vulnerability. Before the pg
command executes the SQL query, add validation to ensure that TIMESTAMP is a
valid positive integer and that the chain_id value stored in c is also a valid
positive integer. Use a pattern matching approach or numeric validation function
to verify these values contain only digits, and exit with an error message if
validation fails for either TIMESTAMP or any individual chain_id.
In `@packages/ensindexer-checkpoint/scripts/remote-run.sh`:
- Around line 81-86: The readiness check for EnsRainbow has a logic flaw where
the final guard statement at line 85 only verifies the marker file exists, but
does not confirm the health endpoint was actually successful. If the loop times
out after 720 iterations while the marker file exists but the health check never
succeeds, the script incorrectly proceeds. Modify the guard statement to verify
that both the marker file exists AND the health check endpoint returned
successfully, not just the presence of the marker file, to ensure the service is
truly ready before continuing.
- Around line 127-138: The while loop that waits for backfill completion
(checking for is_ready=1) has no timeout mechanism and can run indefinitely if
the indexer gets stuck. Add a timeout mechanism to this loop by tracking the
elapsed time using a start timestamp variable initialized before the loop
begins, and within the loop check if the elapsed time exceeds a reasonable
maximum wait time (such as 30 minutes or 1800 seconds). If the timeout is
exceeded before is_ready=1 is detected, exit the loop and call die with an
appropriate error message indicating that the wait for backfill completion timed
out.
---
Outside diff comments:
In @.github/workflows/deploy_ensnode_blue_green.yml:
- Around line 28-41: The deploy-environment job lacks a concurrency group
configuration, which could allow simultaneous deployments to the same target
environment, causing race conditions. Add a concurrency block to the
deploy-environment job that groups deployments by the target environment name
using inputs.target and sets cancel-in-progress to false to prevent cancelling
already-in-progress deployments. Insert this concurrency section after the
permissions field in the deploy-environment job definition.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 91004aac-1854-4ae8-90bb-fce9cf067fcf
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (40)
.changeset/alpha-defaults-unigraph-searchlight.md.github/workflows/checkpoint.yml.github/workflows/checkpoint_reaper.yml.github/workflows/deploy_ensapi_hotfix.yml.github/workflows/deploy_ensnode_blue_green.yml.github/workflows/index_checkpoint_and_deploy.ymlapps/ensindexer/configs/README.mdapps/ensindexer/configs/alpha.envapps/ensindexer/configs/mainnet.envapps/ensindexer/src/config/config.test.tsapps/ensindexer/src/config/environment-defaults.tsapps/ensindexer/src/config/named-configs.test.tspackages/ensdb-cli/package.jsonpackages/ensdb-cli/src/cli.tspackages/ensdb-cli/src/commands/dump.tspackages/ensdb-cli/src/commands/load.tspackages/ensdb-cli/src/lib/ensdb.tspackages/ensdb-cli/src/lib/exec.tspackages/ensdb-cli/src/lib/pgtools.test.tspackages/ensdb-cli/src/lib/pgtools.tspackages/ensdb-cli/src/main.tspackages/ensdb-cli/tsconfig.jsonpackages/ensdb-cli/tsup.config.tspackages/ensdb-cli/vitest.config.tspackages/ensindexer-checkpoint/.gitignorepackages/ensindexer-checkpoint/README.mdpackages/ensindexer-checkpoint/package.jsonpackages/ensindexer-checkpoint/scripts/.shellcheckrcpackages/ensindexer-checkpoint/scripts/checkpoint.shpackages/ensindexer-checkpoint/scripts/cherry-down.shpackages/ensindexer-checkpoint/scripts/cherry-up.shpackages/ensindexer-checkpoint/scripts/config.example.shpackages/ensindexer-checkpoint/scripts/detect-done.shpackages/ensindexer-checkpoint/scripts/lib.shpackages/ensindexer-checkpoint/scripts/remote-checkpoint.shpackages/ensindexer-checkpoint/scripts/remote-provision.shpackages/ensindexer-checkpoint/scripts/remote-rehydrate.shpackages/ensindexer-checkpoint/scripts/remote-resolve-end-blocks.shpackages/ensindexer-checkpoint/scripts/remote-run.shpackages/ensindexer-checkpoint/scripts/remote-seed-export.sh
Index both mainnet-namespace configs in parallel on a single disposable box (shared Postgres + ponder_sync) instead of one box per config. Ponder indexes single-threaded, so two configs fit a high-core box; co-location restores the ~200GB ponder_sync seed once (not twice) and fetches the shared chain-1 RPC tail once. - bump default Cherry plan amd-ryzen-9900x -> amd-ryzen-9950x - remote-checkout.sh: shared cheap setup (stop stale stack, checkout @ SHA, install, build ensdb-cli); frees all four ports - remote-index-one.sh: index ONE config in an isolated process group on caller-assigned ports, graceful-stop only that config (fork-safe pgid capture; identity passed inline as process env, no shared .env.local) - remote-checkpoint-prod.sh: lock -> checkout -> rehydrate once -> index alpha + mainnet in parallel -> dump/upload/load each -> refresh seed once - index-both.sh: manual production runner - remote-run.sh: now a thin single-config wrapper (dev checkpoint flow unchanged) - index_checkpoint_and_deploy.yml: drop the per-config matrix; one job runs both configs on one box Per-config isolation: ponder ports 42069/42169, ensrainbow ports 3223/3224 + separate data dirs (labelsets differ: searchlight/1 vs subgraph/0; LevelDB takes an exclusive dir lock), separate schemas and process groups.
shrugs
commented
Jun 23, 2026
shrugs
left a comment
shrugs
left a comment
Member
Author
There was a problem hiding this comment.
I feel like there's some duplication going on between these two approaches (index and load, checkpoint). but actually they're the same thing, just with END_BLOCK configuration in the checkpoint case and no load, just dump/upload?
Collapse the production (index+load) and dev (point-in-time checkpoint) flows into one parameterized pipeline — they differ only by MODE (full-backfill vs end-block), CONFIGS, and the load/seed flags. Delete the duplicated remote-run.sh / remote-checkpoint-prod.sh / index-both.sh in favor of a single on-box orchestrator (remote-checkpoint.sh) and a single manual runner (checkpoint.sh). ensdb-cli + ensdb-sdk: - terminology → ensIndexerSchemaName throughout - dump always writes a single-row metadata sidecar (no opt-out) - load gains --force; default fail-fast if target schema/metadata exists; migrates the ensnode schema before restore; fails fast on unresolvable dump schema name - move schema exists/drop/rename + metadata read/write helpers into ensdb-sdk (public); CLI consumes them - bundle migrations into the CLI dist (tsup) so it works standalone - subprocess timeout in exec(); commented pg_dump/pg_restore args; v1.16.0 workflows: - new reusable .github/actions/deploy_railway_services composite action (update+redeploy a set of service ids to one image, ENSRainbow first) - index_checkpoint_and_deploy: 3 phases — deploy light (sepolia/*) → index+load alpha/mainnet on Cherry → deploy heavy (alpha/mainnet); timeout 960m; null-SHA guard; inputs via env (no shell injection) - deploy_ensnode_blue_green: rename inputs (version=schemas, tag=images), use the reusable action - deploy_ensapi_hotfix: use the reusable action; drop unused id-token - checkpoint.yml: drop chain_ids input (chains derived from config) - rename checkpoint_reaper.yml → checkpoint_gc.yml (garbage collector) box scripts hardening: - ship secrets (TARGET_URL, Cherry token) via 600 files, not SSH argv - cherry-down keeps the box handle when termination fails (no orphan) - per-run TOFU known_hosts; R2 lock owner-token race check - ensrainbow health gate requires a live health response; crash detection is process-liveness-first; numeric validation before end-block SQL - end-block mode derives indexed chain ids from the config (scripts/print-indexed-chain-ids.ts) docs/config: configs README → "Hosted Instance Configurations"; bump SELF_DESTRUCT_HOURS to 20; delete the named-configs drift-guard test.
Member
Author
|
@greptile review |
- write_rclone_conf: create the R2 credentials file mode 600 (subshell-scoped umask 077) so it isn't world-readable. - remote-rehydrate.sh: add 'require pg_restore' for a clear early failure.
Member
Author
|
@greptile review |
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
packages/ensindexer-checkpoint/scripts/remote-index-one.sh (1)
79-82: 🩺 Stability & Availability | 🟡 Minor
CHAIN_IDSderivation can silently pass with partial output from a failedtsxscript.When the
tsxscript fails after emitting some chain IDs, the assignmentCHAIN_IDS="$(... | paste -sd, -)"still succeeds because bash doesn't honorset -efor assignment failures — the captured partial output becomes the assignment value. The-nguard only checks for non-empty content, not successful exit, so the run proceeds with an incomplete chain set. This contradicts the deliberate pattern at lines 84–90, which explicitly captures to a file and checks the exit code to prevent silently indexing incomplete chains to realtime.Capture to a temporary file and check the exit code explicitly to match the end-block resolver pattern:
Suggested fix
- CHAIN_IDS="$(cd "$REPO_DIR/apps/ensindexer" && - ENSRAINBOW_URL="http://localhost:$RAINBOW_PORT" ENSINDEXER_SCHEMA_NAME="$SCHEMA" \ - pnpm exec tsx scripts/print-indexed-chain-ids.ts | paste -sd, -)" - [ -n "$CHAIN_IDS" ] || die "[$CONFIG] could not derive indexed chain ids from config" + CHAIN_IDS_OUT="/tmp/${SCHEMA}.chainids" + (cd "$REPO_DIR/apps/ensindexer" && + ENSRAINBOW_URL="http://localhost:$RAINBOW_PORT" ENSINDEXER_SCHEMA_NAME="$SCHEMA" \ + pnpm exec tsx scripts/print-indexed-chain-ids.ts) >"$CHAIN_IDS_OUT" || + die "[$CONFIG] could not derive indexed chain ids from config" + CHAIN_IDS="$(paste -sd, - <"$CHAIN_IDS_OUT")" + [ -n "$CHAIN_IDS" ] || die "[$CONFIG] derived empty chain id list"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/ensindexer-checkpoint/scripts/remote-index-one.sh` around lines 79 - 82, The CHAIN_IDS assignment using command substitution does not properly validate the exit code of the tsx script—bash allows partial output to be captured even when the script fails, and the subsequent non-empty check only verifies content exists, not that the command succeeded. Refactor this block to capture the output to a temporary file instead of command substitution, explicitly check the exit code of the pnpm exec tsx scripts/print-indexed-chain-ids.ts command, and only assign CHAIN_IDS if the exit code indicates success. This approach should match the pattern already used in the end-block resolver section around lines 84-90 to ensure failed chain ID derivation is caught before proceeding.packages/ensindexer-checkpoint/scripts/remote-rehydrate.sh (1)
42-48: 🗄️ Data Integrity & Integration | 🟠 MajorPrevent partial
ponder_syncrestores from being treated as complete.Line 42 skips restore whenever
ponder_syncschema exists. If Line 47pg_restorefails mid-restore, the schema will exist in a partial/corrupted state. On retry, the schema existence check treats this as a complete restore and skips the restore step, leaving the system with a corrupted cache.Suggested fix
elif r2_exists "$(r2_seed "$R2_SEED_OBJECT")"; then log "downloading + restoring ponder_sync from R2 (large; be patient)" - rclone copy --progress "$(r2_seed "$R2_SEED_OBJECT")" "$DATA_MOUNT/" - pg_restore --no-owner --no-privileges -j 4 -d "$PG_CONN" "$DATA_MOUNT/$R2_SEED_OBJECT" - rm -f "$DATA_MOUNT/$R2_SEED_OBJECT" + seed_path="$DATA_MOUNT/$R2_SEED_OBJECT" + rclone copyto --progress "$(r2_seed "$R2_SEED_OBJECT")" "$seed_path" + if ! pg_restore --exit-on-error --no-owner --no-privileges -j 4 -d "$PG_CONN" "$seed_path"; then + warn "ponder_sync restore failed; dropping partial schema before retry" + pg -c 'DROP SCHEMA IF EXISTS ponder_sync CASCADE;' || true + rm -f "$seed_path" + die "failed restoring ponder_sync from seed" + fi + rm -f "$seed_path" else🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/ensindexer-checkpoint/scripts/remote-rehydrate.sh` around lines 42 - 48, The schema existence check on line 42 (checking if ponder_sync exists) does not account for partial/corrupted restores. If the pg_restore command on line 47 fails midway, the ponder_sync schema will exist in a corrupted state, and on retry the existence check will incorrectly skip the restore step. Add error handling after the pg_restore command that removes the ponder_sync schema if the restore fails, ensuring the next retry will detect the schema as missing and re-attempt the full restore from scratch. Alternatively, use a completion marker file or flag that only gets set after a successful restore to track restore completion rather than relying solely on schema existence.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/deploy_ensnode_blue_green.yml:
- Around line 49-54: The "Require version or tag" step directly interpolates
workflow_dispatch inputs into the shell script at lines 51-52, creating a shell
injection vulnerability where crafted inputs with metacharacters could break the
condition and inject arbitrary commands. Instead, pass the inputs through
environment variables using the env: keyword at the step level, then reference
those environment variables in the shell script using $VARIABLE_NAME syntax,
following the same safer pattern used for environment variables elsewhere in the
workflow.
---
Outside diff comments:
In `@packages/ensindexer-checkpoint/scripts/remote-index-one.sh`:
- Around line 79-82: The CHAIN_IDS assignment using command substitution does
not properly validate the exit code of the tsx script—bash allows partial output
to be captured even when the script fails, and the subsequent non-empty check
only verifies content exists, not that the command succeeded. Refactor this
block to capture the output to a temporary file instead of command substitution,
explicitly check the exit code of the pnpm exec tsx
scripts/print-indexed-chain-ids.ts command, and only assign CHAIN_IDS if the
exit code indicates success. This approach should match the pattern already used
in the end-block resolver section around lines 84-90 to ensure failed chain ID
derivation is caught before proceeding.
In `@packages/ensindexer-checkpoint/scripts/remote-rehydrate.sh`:
- Around line 42-48: The schema existence check on line 42 (checking if
ponder_sync exists) does not account for partial/corrupted restores. If the
pg_restore command on line 47 fails midway, the ponder_sync schema will exist in
a corrupted state, and on retry the existence check will incorrectly skip the
restore step. Add error handling after the pg_restore command that removes the
ponder_sync schema if the restore fails, ensuring the next retry will detect the
schema as missing and re-attempt the full restore from scratch. Alternatively,
use a completion marker file or flag that only gets set after a successful
restore to track restore completion rather than relying solely on schema
existence.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: feef34e0-ab91-4522-a9e1-4c12e1105072
📒 Files selected for processing (12)
.github/actions/deploy_railway_services/action.yml.github/workflows/deploy_ensnode_blue_green.yml.github/workflows/index_checkpoint_and_deploy.ymlpackages/ensdb-cli/src/lib/migrations.tspackages/ensindexer-checkpoint/.gitignorepackages/ensindexer-checkpoint/README.mdpackages/ensindexer-checkpoint/scripts/checkpoint.shpackages/ensindexer-checkpoint/scripts/lib.shpackages/ensindexer-checkpoint/scripts/load-checkpoints.shpackages/ensindexer-checkpoint/scripts/remote-checkpoint.shpackages/ensindexer-checkpoint/scripts/remote-index-one.shpackages/ensindexer-checkpoint/scripts/remote-rehydrate.sh
…2.yml Leave the live deploy_ensnode_blue_green.yml untouched (restored to main); the version/tag-split rewrite lives in a parallel v2 workflow to validate before swapping.
Comment on lines
+56
to
+60
| (cd "$REPO_DIR/apps/ensrainbow" && | ||
| DATA_DIR="$RAINBOW_DATA_DIR" LABEL_SET_ID="$LABEL_SET_ID" LABEL_SET_VERSION="$LABEL_SET_VERSION" \ | ||
| PORT="$RAINBOW_PORT" pnpm run entrypoint >"/tmp/ensrainbow-$CONFIG.log" 2>&1) & | ||
| RAINBOW_MARKER="$RAINBOW_DATA_DIR/ensrainbow_db_ready" | ||
| rainbow_ready=0 |
Comment on lines
+105
to
+108
| // Upsert the metadata under the target schema (re-keyed by the writer). | ||
| if (metadata) { | ||
| await writer.writeEnsNodeMetadata(metadata); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Remote-checkpoint indexing → load → deploy
Summary
@ensnode/ensdb-cli(dump/load ENSIndexer schemas + their ENSNode metadata) and@ensnode/ensindexer-checkpoint(Cherry/R2 box orchestration) — plus four[next]-prefixed workflows.alphaenv defaults (searchlight/1 +unigraph) and introduces committedapps/ensindexer/configs/*.envas the single-source-of-truth build_id parity contract.Why
Indexing ENS from scratch on Railway/Render takes ~1 week, so it's never a viable release step. This builds the fast path proven by an earlier prototype into a one-trigger pipeline:
alpha,mainnet) on two Cherry boxes from the R2ponder_syncseed.<ns>Schema<VERSION>.Resume mechanism: the box runs the exact production identity (sourced from
configs/<config>.env, noEND_BLOCK_*), so Ponder'sbuild_idmatches the deployed service and it resumes rather than crashing. No build_id patching.Identifiers: docker image =
sha-<short>; schema name =<ns>Schema<VERSION>(so the RC pre-seed and the eventual release deploy share the schema); checkpoint dump = sha-keyed in R2.What's here
packages/ensdb-cli—dump/load(schema viapg_dump/pg_restore, ENSNode metadata via ensdb-sdk, auto schema-rename,--skip-if-exists).packages/ensindexer-checkpoint— bash: Cherry lifecycle (+ self-destruct watchdog), write-tuned Postgres, full-backfill + end-block modes, sha-keyed R2 lock/idempotency, alpha seed-refresh.apps/ensindexer/configs/{alpha,mainnet}.env+ drift-guard test — the build_id parity contract.environment-defaults.ts— alpha → searchlight/1 + unigraph (EFP excluded from the shared default: its datasources are mainnet-only and would break alpha-sepolia; enabled explicitly inalpha.env).[next]):index_checkpoint_and_deploy.yml,checkpoint.yml(dev point-in-time),deploy_ensapi_hotfix.yml,checkpoint_reaper.yml.deploy_ensnode_blue_green.yml— backward-compatible optionalimage_taginput (decouples image tag from schema version).Testing
pnpm test(ensindexer + ensdb-cli): 256/256 pass; typecheck clean; biome clean.ensindexer-checkpointsbucket (server-side copy, byte-verified).workflow_dispatch-only and need to be onmainto run. The first run (planned:checkpoint.yml,config=mainnet, Jan 2019) is the real integration test. The one thing only a live run confirms is config parity (box identity == deployed service env).Notes for Reviewer
workflow_dispatch); the only scheduled job (reaper) only ever terminatesensindexer-checkpoint-*Cherry boxes — never production.deploy_ensnode_blue_green.yml) is additive and backward-compatible.CHERRY_*,CF_R2_ACCESS_KEY_ID/CF_R2_SECRET_ACCESS_KEY,<TARGET>_ENSDB_ADMIN_URL(existingALCHEMY_API_KEYreused).Changeset: included for
environment-defaults(behavior change for alpha instances); the two new packages are private (no changeset required).