Skip to content

fix(scenarios): use -internal aggregate Service for TM RPC/REST endpoints#294

Merged
bdchatham merged 1 commit into
mainfrom
fix/scenario-tm-rpc-endpoint
May 19, 2026
Merged

fix(scenarios): use -internal aggregate Service for TM RPC/REST endpoints#294
bdchatham merged 1 commit into
mainfrom
fix/scenario-tm-rpc-endpoint

Conversation

@bdchatham
Copy link
Copy Markdown
Collaborator

The major-upgrade scenario's bash steps (compute-target-height, resolve-proposal-id, wait-for-proposal-to-pass) curl Tendermint endpoints at per-pod Service DNS:

```
http://${SEI_DEPLOYMENT}-0.${SEI_NAMESPACE}.svc.cluster.local:26657/status
http://${SEI_DEPLOYMENT}-0.${SEI_NAMESPACE}.svc.cluster.local:1317/cosmos/...
```

On the live harbor manual run this dies with:

```
curl: (7) Failed to connect to bench-major-upgrade-X-0.nightly.svc:26657:
Could not connect to server
```

Root cause

Per-pod Services on a SeiNodeDeployment expose only the EVM ports (8545/8546). Tendermint RPC (26657) and REST (1317) are surfaced only on the deployment-scoped `-internal` aggregate Service. From a recent SND's `.status`:

```yaml
internalService:
name: bench-major-upgrade-X-internal
ports:
evmHttp: 8545
rest: 1317
rpc: 26657
perPodServices:

Fix

Three sed substitutions in `scenarios/major-upgrade.yaml`:

  • L143: `${SEI_DEPLOYMENT}-0...:26657` → `${SEI_DEPLOYMENT}-internal...:26657`
  • L236: `${SEI_DEPLOYMENT}-0...:1317` → `${SEI_DEPLOYMENT}-internal...:1317`
  • L385: same as L236 in the wait-for-proposal-to-pass step

TM RPC/REST are stateless query endpoints so the aggregate (any healthy validator backing the Service) is the right target.

Symptom chain on the live run

  • `compute-target-height` failed with curl rc=7 → exited 1
  • `workflow-vars-` ConfigMap never got created
  • Downstream `submit-upgrade-proposal` step's seitask-runner pod went into `CreateContainerConfigError` because its `envFrom: configMapRef` pointed at a ConfigMap that didn't exist

Single fix at the source resolves the whole cascade.

Bug count

Fix #8 in the major-upgrade-runs-end-to-end debugging chain. Same shape as several earlier ones: an assumption about the wider system that doesn't survive contact with the actual cluster. The audit's "trust but verify" principle continues to earn its keep.

🤖 Generated with Claude Code

…ints

The major-upgrade scenario's bash steps (compute-target-height,
resolve-proposal-id, wait-for-proposal-to-pass) curl Tendermint
endpoints at \${SEI_DEPLOYMENT}-0.\${SEI_NAMESPACE}.svc.cluster.local
on ports 26657 (RPC) and 1317 (REST). This dies with:

  curl: (7) Failed to connect to bench-major-upgrade-X-0.nightly.svc:26657:
  Could not connect to server

Root cause: per-pod Services on a SeiNodeDeployment expose ONLY the EVM
ports (8545/8546). Tendermint RPC (26657) and REST (1317) are surfaced
ONLY via the deployment-scoped \`<snd-name>-internal\` aggregate Service
(see SeiNodeDeployment.status.internalService + .endpoints.{tendermintRpc,
tendermintRest}).

Concretely from a recent SND status:
  tendermintRpc:  http://<snd-name>-internal.nightly.svc:26657
  tendermintRest: http://<snd-name>-internal.nightly.svc:1317

Change all three bash steps to use the -internal aggregate Service.
TM RPC/REST are stateless query endpoints so the aggregate (any
healthy validator) is the right target.

Symptom chain on the live harbor run that caught this:
- compute-target-height failed with curl rc=7 (connection refused)
- workflow-vars ConfigMap never got created
- downstream submit-upgrade-proposal step waited as a
  CreateContainerConfigError because its \`envFrom: configMapRef\`
  pointed at a ConfigMap that didn't exist

Single fix at the source resolves the whole cascade.
@cursor
Copy link
Copy Markdown

cursor Bot commented May 19, 2026

You have used all Bugbot PR reviews included in your free trial for your GitHub account on this workspace.

To continue using Bugbot reviews, enable Bugbot for your team in the Cursor dashboard.

@bdchatham bdchatham merged commit e4da5ca into main May 19, 2026
2 checks passed
@bdchatham bdchatham deleted the fix/scenario-tm-rpc-endpoint branch May 19, 2026 21:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant