Skip to content

fix: let resource-doc and swagger caches survive an unreachable Redis - #59

Merged
hongwei1 merged 1 commit into
develop-obpfrom
fix/resource-doc-cache-failsafe
Aug 4, 2026
Merged

fix: let resource-doc and swagger caches survive an unreachable Redis#59
hongwei1 merged 1 commit into
develop-obpfrom
fix/resource-doc-cache-failsafe

Conversation

@hongwei1

@hongwei1 hongwei1 commented Aug 4, 2026

Copy link
Copy Markdown
Owner

What and why

Caching has two groups of cache accessors sitting directly next to each other. The product caches route through tryGet/trySet, whose comment states the intent plainly:

Fail-safe wrappers around Redis.use … If Redis is unreachable (dev without a running Redis, transient failure, etc.) we treat it as a miss and recompute instead of failing the whole request.

The resource-doc and swagger caches immediately above them did not. They called Redis.use bare — and Redis.use throws rather than returning None. So a Redis blip turned every /resource-docs, /swagger, OpenAPI and message-docs request into a 500. Those are the documents API Explorer and Portal load on startup, so the blast radius is the whole developer-facing surface.

Two adjacent groups, one protected and one not, with the explaining comment attached to the protected one: this reads as a missing wrapper rather than a deliberate difference.

Changes

obp-api/src/main/scala/code/api/cache/Caching.scala (only file touched)

  • All four get/set pairs — dynamic resource docs, static resource docs, all resource docs, static swagger docs — now go through the existing tryGet/trySet wrappers.
  • Explicit return types added to match the product caches: Option[String] for gets, Unit for sets. Every call site already discarded the set return value, so narrowing it is source-compatible (verified by compiling).
  • The wrapper comment lost its "for product caches" qualifier, since it now covers both groups.

Test evidence

Measured by running the full suite against a dead Redis port (OBP_CACHE_REDIS_PORT=6399), which makes every Redis call fail. Both runs on the same base (4d21dc6cf), same machine:

tests failures
before 3273 135
after 3273 39

The 96 that disappear are exactly the suites served from these caches:

suite before after
ResourceDocsTest 55 0
V7ResourceDocsAggregationTest 12 0
SwaggerDocsTest 11 0
MessageDocsJsonSchemaTest 8 0
DynamicEndpointsTest 6 0
ResourceDocsTechnologyTest 2 0
GetMessageDocsSwaggerTest 1 0
Http4sServerIntegrationTest 1 0

With a healthy Redis the suite is unchanged: 3273 tests, 0 failures, identical to the pre-change baseline.

CI on this branch: all jobs green.

The remaining 39, classified

These are other Redis dependencies, untouched by this change. Verified by aggregating the JedisConnectionException stack frames across all four shard logs:

category failures suites unguarded call sites
rate limiting 23 RateLimitTest 7, RateLimitingTest (v4) 6, RateLimitsTest 4, AuthRateLimiterTest 3, ConsumerTest 2, RateLimitingTest (v5.1) 1 RateLimitingUtil.getCounterState, consumerRateLimitState, getCallCounterForPeriod, MappedRateLimitingProvider.createConsumerCallLimits, Caching.invalidateRateLimitCache
endpoints that exist to inspect Redis 12 LogCacheEndpointTest 7, CacheEndpointsTest 5 inherent — these endpoints read Redis by definition
consent / dauth error wording 4 ConsentsTest 1, ConsentRequestTest 2, dauthTest 1 see below

No resource-doc or swagger call site remains in those stacks.

The third group is worth being precise about, because it is not a 500: the status-code assertion passes and the next line fails. In ConsentRequestTest.scala:169-170 the endpoint still returns 401, but the message is not ConsentHeaderValueInvalid. With Redis down these paths still reject the request correctly, just with different wording. Their counts are identical before and after this change.

Trade-offs and known limitations

  • Rate limiting is not made fail-safe here, and one dead branch stays dead. RateLimitingUtil calls Redis.use directly rather than going through Caching, so this change cannot reach it. In particular getCounterState's case None => "REDIS_UNAVAILABLE" fail-open branch remains unreachable, because Redis.use throws instead of returning None. Making it live means changing Redis.use itself, which every caller shares — a larger change with its own blast radius, deliberately not bundled here.
  • Silent degradation is the point, but it is still degradation. When Redis is unreachable these endpoints now recompute on every request instead of serving from cache. That is the intended trade (a slow correct answer beats a fast 500) and matches what the product caches already do, but under sustained Redis loss the resource-docs endpoints lose their caching entirely. The failures are logged at debug level by the existing wrappers.
  • memoize-based caches were already safe and are untouched. Caching.memoizeSyncWithProvider/memoizeWithProvider go through scalacache, which logs Failed to read from cache at WARN and continues. They produce a lot of log noise when Redis is down but do not fail requests.
  • No new tests. The behaviour is verified by the dead-port run above rather than by a unit test, since reproducing "Redis unreachable" inside the suite would mean either a second Redis or mocking Redis.use — both heavier than the property being checked. The before/after numbers are reproducible with OBP_CACHE_REDIS_PORT=<unused port> ./run_tests_parallel.sh.

The product caches in Caching already route through tryGet/trySet, whose
comment states the intent plainly: if Redis is unreachable, treat it as a
miss and recompute instead of failing the whole request. The resource-doc
and swagger caches sit directly above them in the same file and did not --
they called Redis.use bare, and Redis.use throws rather than returning None.
A Redis blip therefore turned every /resource-docs, /swagger, OpenAPI and
message-docs request into a 500. These are exactly the documents API
Explorer and Portal load on startup, so the blast radius is the whole
developer-facing surface.

Route all four get/set pairs through the same wrappers and give them
explicit return types matching the product caches (Option[String] / Unit).
Every call site already discarded the set return value, so narrowing it to
Unit is source-compatible.

Measured by running the suite against a dead Redis port
(OBP_CACHE_REDIS_PORT=6399), which makes every Redis call fail:

  before  3273 tests, 135 failures
  after   3273 tests,  39 failures

The 96 that disappear are the ones served from these caches: ResourceDocsTest
(55), V7ResourceDocsAggregationTest (12), SwaggerDocsTest (11),
MessageDocsJsonSchemaTest (8), DynamicEndpointsTest (6),
ResourceDocsTechnologyTest (2), GetMessageDocsSwaggerTest (1) and
Http4sServerIntegrationTest (1). With a healthy Redis the suite is unchanged:
3273 tests, 0 failures.

The remaining 39 are other Redis dependencies, untouched here: rate limiting
(23), the endpoints that exist to inspect Redis itself (12), and four
consent/dauth scenarios that assert on an error message rather than a status
code -- those still reject the request correctly, just with different wording.

Rate limiting is unaffected in another way worth stating: RateLimitingUtil
calls Redis.use directly rather than going through Caching, so the
"REDIS_UNAVAILABLE" branch in getCounterState stays dead code. Making it live
would mean changing Redis.use itself, which every caller shares.
@sonarqubecloud

sonarqubecloud Bot commented Aug 4, 2026

Copy link
Copy Markdown

@hongwei1
hongwei1 merged commit f697dd1 into develop-obp Aug 4, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant