Describe the bug
The store-gateway-client compression sub-tests in integration/grpc_compression_test.go:119 (added by PR #7459) keep the ingester running while issuing the test query. The only synchronisation gate before the query is cortex_storegateway_bucket_sync_total > 0, which only proves a sync cycle ran — it doesn't guarantee the block has been loaded into the store-gateway, nor that c.Query("series_1") is actually served by the store-gateway gRPC client (the thing -querier.store-gateway-client.grpc-compression applies to) rather than by the still-live ingester.
If the querier serves the request from the ingester, the compression flag is never exercised, and the test passes regardless of whether store-gateway gRPC compression works for that codec. This is a particular concern given the recent TestGRPCCompression/ingester_client/zstd flake observed on CI ARM64 (where the symptom was a different startup race, but the same lack of determinism around which path serves the query is the underlying gap).
To Reproduce
- Locally break store-gateway gRPC compression by, for example, registering a no-op
Compressor for zstd only on the store-gateway side.
- Run
go test -tags requires_docker,integration,integration_grpc -run TestGRPCCompression ./integration/....
- Observe the
store_gateway_client/zstd sub-test still passes.
Expected behavior
Make the test deterministic about which path is exercised:
- Stop the ingester before issuing the query, so the querier has no choice but to fan out to the store-gateway.
- Or assert that a store-gateway-side query metric (e.g.
cortex_storegateway_request_duration_seconds_count{operation="series"}) increased.
That way a regression in the store-gateway gRPC compression path is actually caught.
Environment:
Additional Context
Describe the bug
The store-gateway-client compression sub-tests in
integration/grpc_compression_test.go:119(added by PR #7459) keep the ingester running while issuing the test query. The only synchronisation gate before the query iscortex_storegateway_bucket_sync_total > 0, which only proves a sync cycle ran — it doesn't guarantee the block has been loaded into the store-gateway, nor thatc.Query("series_1")is actually served by the store-gateway gRPC client (the thing-querier.store-gateway-client.grpc-compressionapplies to) rather than by the still-live ingester.If the querier serves the request from the ingester, the compression flag is never exercised, and the test passes regardless of whether store-gateway gRPC compression works for that codec. This is a particular concern given the recent
TestGRPCCompression/ingester_client/zstdflake observed on CI ARM64 (where the symptom was a different startup race, but the same lack of determinism around which path serves the query is the underlying gap).To Reproduce
Compressorforzstdonly on the store-gateway side.go test -tags requires_docker,integration,integration_grpc -run TestGRPCCompression ./integration/....store_gateway_client/zstdsub-test still passes.Expected behavior
Make the test deterministic about which path is exercised:
cortex_storegateway_request_duration_seconds_count{operation="series"}) increased.That way a regression in the store-gateway gRPC compression path is actually caught.
Environment:
release-1.21(PR fixes panic when using snappy with ingester client #7459 backport in PR Backport security audit fixes (V01-V07), snappy gRPC fix, and stream push panic fix to release-1.21 #7574).Additional Context
TestGRPCCompression/ingester_client/zstd(ARM64 startup race); fixing the test determinism here is independent but lowers overall flake surface.