Skip to content

test(blob): fix "sensor schema missing" flake in per-device-type sharding suite - #1750

Merged
kriszyp merged 1 commit into
mainfrom
kris/blob-sharding-schema-race
Jul 10, 2026
Merged

test(blob): fix "sensor schema missing" flake in per-device-type sharding suite#1750
kriszyp merged 1 commit into
mainfrom
kris/blob-sharding-schema-race

Conversation

@kriszyp

@kriszyp kriszyp commented Jul 10, 2026

Copy link
Copy Markdown
Member

Problem

The Per-device-type LMDB database sharding suite in integrationTests/apiTests/blob.test.mjs intermittently fails in CI with:

AssertionError [ERR_ASSERTION]: sensor schema missing

followed by cascading Table 'sensor.SensorBlob' does not exist / expected 200 "OK", got 404 failures. Observed most recently on Integration Tests 3/6 (Node.js v22) — a single shard/runtime out of ~24, while every other shard and runtime passed, which is the classic flake signature.

Root cause

All three device schemas (thermostat, doorlock, sensor) are declared in one schema.graphql loaded by a single component, then before() does:

await restartHttpWorkers(client, '/openapi', 120000);

restartHttpWorkers only waits on the /openapi readiness probe, which returns 200 the moment a worker is back up — that does not guarantee the component's schema.graphql has been parsed and all three tables registered and propagated to the metadata that describe_all / REST read. The first test then asserts all three schemas in a single shot and races the tail of component load. sensor — the last of the three declared — is the one that loses the race, and the subsequent sensorblob REST calls 404 for the same reason.

Fix

After the restart, poll describe_all until all three device schemas are visible before running the assertions (30s budget, 250ms interval). Returns immediately in the happy path, so it adds no meaningful cost to normal runs — it only absorbs the propagation window that was previously assumed to be zero. Test-only change; no product code touched.

This is the same "readiness poll instead of assumed timing" approach as #1091.

Verification

  • prettier@3.9 clean, node --check clean.
  • Not run through the full local integration suite: reproducing the race is nondeterministic, so a single green run wouldn't prove the fix — the change is a poll-until-ready that cannot break the already-passing case.

🤖 Generated with Claude Code

The per-device-type LMDB sharding suite restarts http_workers and then
immediately asserts all three device schemas (thermostat/doorlock/sensor)
are visible in describe_all. restartHttpWorkers only waits on the /openapi
readiness probe, which returns 200 as soon as a worker is back up — before
the component's schema.graphql has been parsed and all three tables
registered and propagated to the metadata describe_all/REST read. That race
intermittently fails with "sensor schema missing" (the last of the three)
and cascading SensorBlob 404s.

Poll describe_all until every device schema is visible before running the
assertions. Returns immediately in the happy path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a helper function waitForDeviceSchemas in integrationTests/apiTests/blob.test.mjs to poll the describe_all operation until all required device schemas are visible, addressing an intermittent test flake. The review feedback suggests using optional chaining and error handling when parsing the HTTP response body within the polling loop to prevent potential TypeErrors from masking the actual server errors.

Comment on lines +467 to +473
let body = '';
while (Date.now() < deadline) {
const r = await client.req().send({ operation: 'describe_all' });
body = JSON.stringify(r.body);
if (required.every((frag) => body.includes(frag))) return;
await setTimeout(250);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

When asserting on HTTP response bodies in tests, especially within polling or retry blocks, use optional chaining (e.g., r?.body) to prevent TypeErrors from masking the actual server error or response text when the server returns a non-JSON or empty response.

	let body = '';
	while (Date.now() < deadline) {
		try {
			const r = await client.req().send({ operation: 'describe_all' });
			body = JSON.stringify(r?.body ?? {});
			if (required.every((frag) => body.includes(frag))) return;
		} catch (err) {
			body = err.message;
		}
		await setTimeout(250);
	}
References
  1. Use optional chaining (e.g., res?.body?.message) when asserting on HTTP response bodies in tests, especially within polling or retry blocks. This prevents TypeErrors from masking the actual server error or response text when the server returns a non-JSON or empty response.

@claude

claude Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Reviewed; no blockers found.

Comment thread integrationTests/apiTests/blob.test.mjs
Comment thread integrationTests/apiTests/blob.test.mjs
@kriszyp
kriszyp merged commit a6c5de9 into main Jul 10, 2026
55 of 56 checks passed
@kriszyp
kriszyp deleted the kris/blob-sharding-schema-race branch July 10, 2026 12:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants