Skip to content

Wire Valkey cache scaffolding#35

Merged
kossoy merged 2 commits into
mainfrom
feat/folio-10-20-wire-valkey-caching-scaffolding-config--
Jun 26, 2026
Merged

Wire Valkey cache scaffolding#35
kossoy merged 2 commits into
mainfrom
feat/folio-10-20-wire-valkey-caching-scaffolding-config--

Conversation

@kossoy

@kossoy kossoy commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Refs #20

What changed

  • Added optional Valkey cache config with private service defaults (valkey:6379) and CACHE_* env overrides for deployment secret-store/Infisical values.
  • Added internal/cache with go-redis RESP2 pool setup, startup ping passthrough mode, cache-aside GetOrCompute, per-key SET NX PX locks, TTL jitter, stale envelopes, epoch bumping, targeted delete, OK Folio key helpers, and stable filter hashing.
  • Wired gallery catalog reads through epoch-keyed cache-aside metadata caching.
  • Bumped okf:epoch after API extractions, scheduled extractions, favorite edits, and retry status updates; favorite/retry edits also delete the per-photo cache key.
  • Guarded cache passthrough mode with atomic.Bool so concurrent handlers do not race when Valkey errors flip the client into passthrough.
  • Added miniredis unit coverage for cache behavior, key scheme/filter hashing, invalidation, and API epoch refresh.

Review fixes

  • Fixed scheduled extraction cache coherence by sharing the API cache client with the scheduler and bumping the epoch after scheduled downloads.
  • Fixed the cache passthrough data race by replacing the unsynchronized bool with an atomic flag.
  • Also covered retry status updates, which change catalog-visible rows, with targeted delete plus epoch bump.

Verification

git fetch origin && git rebase origin/main
Current branch feat/folio-10-20-wire-valkey-caching-scaffolding-config-- is up to date.

go test -race ./internal/cache
ok   ok-folio/internal/cache

go test ./internal/api ./internal/cache
ok   ok-folio/internal/api
ok   ok-folio/internal/cache

./scripts/product-verifier.sh
npm ci completed successfully.
npm run build completed successfully.
go test ./... completed successfully, including:
?    ok-folio/cmd/extractor [no test files]
ok   ok-folio/internal/api
ok   ok-folio/internal/cache
ok   ok-folio/internal/config
ok   ok-folio/internal/database
ok   ok-folio/internal/gallery
ok   ok-folio/internal/provider/telegram
ok   ok-folio/internal/provider/webgallery
?    ok-folio/internal/scheduler [no test files]
ok   ok-folio/internal/scraper
ok   ok-folio/internal/testguard
ok   ok-folio/pkg/retry

Deferred: deployed Valkey integration verification is intentionally left for the later stack verification issue; this PR is unit-tested with miniredis only.

Maestro-Backend: codex (0-13m); codex (13m-end, in_place_respawn)

Greptile Summary

This PR adds optional Valkey caching for gallery catalog reads. The main changes are:

  • New cache configuration with private Valkey defaults and environment overrides.
  • A new cache package with RESP2 Redis setup, cache-aside reads, locks, TTL jitter, epoch keys, and key helpers.
  • Gallery catalog reads now use epoch-keyed cache entries.
  • Extraction, retry, favorite, and scheduler writes now bump or clear cache state.
  • Unit coverage was added for cache behavior, config, keys, and catalog invalidation.

Confidence Score: 5/5

The cache scaffolding is optional and keeps existing behavior available when caching is disabled.

The changed cache, configuration, API invalidation, and key-helper paths are covered by focused unit tests and the repository verification reported successful Go and frontend builds.

T-Rex T-Rex Logs

What T-Rex did

  • Ran YAML-based cache configuration tests to compare pre-change behavior with post-change defaults, environment overrides, and invalid CACHE_PORT handling.
  • Executed focused internal/cache package tests and confirmed TestCacheCoreContractRuntime passed and all internal/cache tests passed.
  • Walked through HTTP API flows for the gallery catalog and related Redis-backed keys, verifying before-state versus after-state changes for okf keys and epoch counters across GET, POST, and retry sequences.

View all artifacts

T-Rex Ran code and verified through T-Rex

Reviews (2): Last reviewed commit: "Fix cache invalidation review findings" | Re-trigger Greptile

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9d85414204

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread internal/api/gallery.go
Comment on lines +98 to +100
response, err := okfcache.GetOrCompute(r.Context(), s.cache, key, 2*time.Minute, func(ctx context.Context) (galleryCatalogResponse, error) {
return s.galleryCatalogResponse(ctx, limit, offset, filters)
})

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Invalidate catalog cache after scheduled extraction

In deployments with Scheduler.Enabled, the scheduler writes new DownloadedPhoto rows in internal/scheduler/scheduler.go:62-99 but this new catalog cache is only epoch-bumped from the API extraction and favorite handlers. If a catalog page has been cached here before a scheduled run downloads photos, /api/v1/gallery/catalog continues serving the old photo list/facets for the cache TTL instead of reflecting the completed import; wire the same BumpEpoch invalidation into scheduled extraction or the shared write path.

Useful? React with 👍 / 👎.

Comment thread internal/cache/cache.go
Comment on lines +41 to +45
type Client struct {
r redis.Cmdable
logger zerolog.Logger
passthrough bool
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Guard the passthrough flag across requests

During a Valkey error while the server is handling concurrent requests, this bool is written in error paths such as GetOrCompute, BumpEpoch, Delete, and Epoch while other goroutines read it via Passthrough(). Because Server.cache is shared by all HTTP handlers, those unsynchronized reads/writes create a Go data race and can leave cache behavior undefined; use an atomic or mutex-protected state instead.

Useful? React with 👍 / 👎.

Comment thread internal/api/gallery.go
@kossoy kossoy force-pushed the feat/folio-10-20-wire-valkey-caching-scaffolding-config-- branch from 6d8ef4b to 22f53b1 Compare June 26, 2026 01:01
@kossoy kossoy merged commit 8facaaf into main Jun 26, 2026
4 checks passed
@kossoy kossoy deleted the feat/folio-10-20-wire-valkey-caching-scaffolding-config-- branch June 26, 2026 01:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant