fix(lantern): one malformed record could stop the estate keeping any logs - #2
Merged
Conversation
…logs CF-18 — status_code is INTEGER and TEXT cannot hold a NUL byte, so a single crafted value made every batch insert throw, and the writer requeued the same poison for ever. Unauthenticated, from one endpoint served before the auth check. Both ingest paths are now sanitised, and a batch that fails on a data error is dropped rather than requeued, so one bad line from a trusted service cannot reproduce it either. CF-41 — the ingest quota keys on the forwarded client IP, not the tunnel socket address, which had put every remote caller in one bucket. CF-39 — Beacon watches the 8545 JSON-RPC surface: chain id asserted as 7412 on all three nodes, state roots compared three blocks below the lowest tip, and liveness requiring the RPC head to track the REST head. CF-48 — incidents render camelCase, so timestamps no longer vanish at exactly the moment Postgres is down. CF-40 — the operator documentation says what keyvault now does. It signs sweeps, so the old advice to fund the treasury by hand or disable withdrawals is gone from both files, replaced by the three steps that actually work. Compose halves of the cross-repo tickets: the seed node runs the EVM chain and publishes 8545 (CF-13), HEARTH_CHAIN_ID is one estate-wide setting with BEACON_HEARTH_CHAIN_ID defaulting to it (CF-30), and Beacon joins the origin allowlists (CF-49). VERIFICATION.md records what was executed against the remediated tree and what it returned — including an end-to-end run of the assembled stack, and a section naming what was not tested. Co-Authored-By: Claude <noreply@anthropic.com>
…hout The registry job builds its .env from .env.example and then mints the two keyvault secrets by hand, because keyvault rejects placeholders and short values. Nimbus joined that category in this same branch: NIMBUS_KEY_SECRET encrypts the RSA signing key at rest and env.ts exits if it is unset, a known placeholder, or under 24 characters. So the job brought the whole stack up from ghcr and nimbus exited on boot, taking every service that depends on it with it. The empty value in .env.example is correct and stays -- a secret whose whole purpose is to live outside the database should not ship in a file people copy -- so CI mints one the way an operator would. Co-Authored-By: Claude <noreply@anthropic.com>
…id not docker compose up --wait fails with "container X is unhealthy" and nothing else: no logs, no healthcheck output, no exit code. Diagnosing a red run then means pushing commits to add print statements, which is the same silent- failure shape this estate has already been bitten by twice -- a build that cancelled every target and exited 0, and a preflight that 204s while the handler never runs. On failure only, dump ps, the last healthcheck probe outputs and the tail of the logs for every container that is not healthy, grouped per container. Co-Authored-By: Claude <noreply@anthropic.com>
crucible exits with "PAY_SERVICE_TOKEN is required" on a stack built from .env.example, because compose loads .env wholesale and the example ships that value empty on purpose. The job minted the two keyvault secrets and stopped there, so crucible could never come up and beacon then opened incidents against it. Mint PAY_SERVICE_TOKEN as well, and record how to find the whole set rather than discovering them one red run at a time. Co-Authored-By: Claude <noreply@anthropic.com>
…l be built Eighteen documents in docs/ecosystem/, written against source rather than against the existing documentation — several repository MAP.md files turned out to be materially wrong, and 00-current-state.md §7 records which. The shape of it: 21 architecture decisions with the alternatives that were rejected, a 46-repository target, a domain model, fourteen executable phases, and a 203-item backlog with acceptance criteria. The decomposition into microservices comes first, because building Forge Hub, a marketplace and a developer platform on a system that cannot correctly run two replicas of anything means building all of it twice. Two things the plan insists on that were not asked for. The estate has no metrics, no traces, no dashboards and one webhook as its entire alerting story, so AD-20 puts OpenTelemetry in before the first repository is split — a decomposition you cannot measure is a decomposition you cannot prove safe. And the product accents were five shades of orange, worst pair ΔE 4.1, which is indistinguishable to everyone and identical under protanopia; assets/ carries a corrected set at 17.0 and a chart palette that was validated rather than chosen by eye. Work happens in a new parallel set of micro-* repositories under micro/. Nothing in repos/ is modified, deleted or archived — it keeps running, and it is the rollback target for its own replacement. PLAN.md, ECOSYSTEM.md and MICROSERVICES.md are superseded and removed; their still-true content is folded in. audit.md, upgrade.md and VERIFICATION.md stay — they are a live remediation track with ticket ids that other documents cite. Co-Authored-By: Claude <noreply@anthropic.com>
The chain step timed out against miner1 and reported it as a JSONDecodeError, because curl -sf prints nothing on a timeout and the python meant to read a chain id got an empty string. Every hearthd port here is published as 127.0.0.1:<host>:<container>, so no IPv6 listener exists, and localhost resolves to ::1 first -- every call pays a failed IPv6 connect before falling back. Measured locally: 0.003s to 127.0.0.1:8545 against 3.3s to localhost:8547. A slower runner crosses the -m 10 budget and the step fails on a healthy chain. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part of an estate-wide remediation of the 49-ticket defect register in
audit.md, sequenced by the phase plan inupgrade.md. Each ticket was resolved against its own Done when boxes, then handed to an independent reviewer told to assume it was not fixed and to walk the ticket's How it fails scenario through the new code.Verification
pnpm typecheckpasses across every workspace in this repo.What has NOT been verified
Stated plainly so review effort goes to the right places:
docker compose buildreturns exit 0 while cancelling every target, so a green build here would prove nothing anyway.partialorINCOMPLETE; where any of those touch this repo they are listed in the commit body rather than hidden.🤖 Generated with Claude Code