Skip to content

Add an action to sync the remote Valkey cache#3032

Merged
gbrodman merged 1 commit intogoogle:masterfrom
gbrodman:valkeyCursor
May 5, 2026
Merged

Add an action to sync the remote Valkey cache#3032
gbrodman merged 1 commit intogoogle:masterfrom
gbrodman:valkeyCursor

Conversation

@gbrodman
Copy link
Copy Markdown
Collaborator

@gbrodman gbrodman commented May 1, 2026

This uses two cursors (one for hosts and one for domains) to track our progress in "catching up", or syncing recent changes to the database, using the update-timestamp field. When the cache is in use and fully caught up, these
cursors should be kept relatively up-to-date to the actual time, i.e. less than one hour behind

note: while the generated schema changes due to the cursor enum additions, the golden file doesn't change because our SQL scripts server-side don't enforce the enum restriction


This change is Reviewable

@gbrodman gbrodman added the kokoro:force-run Force a Kokoro build. label May 4, 2026
@domain-registry-eng domain-registry-eng removed the kokoro:force-run Force a Kokoro build. label May 4, 2026
@gbrodman gbrodman added the kokoro:force-run Force a Kokoro build. label May 4, 2026
@domain-registry-eng domain-registry-eng removed the kokoro:force-run Force a Kokoro build. label May 4, 2026
@gbrodman gbrodman added the kokoro:force-run Force a Kokoro build. label May 4, 2026
@domain-registry-eng domain-registry-eng removed the kokoro:force-run Force a Kokoro build. label May 4, 2026
Copy link
Copy Markdown
Collaborator

@ptkach ptkach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How frequently do you intent to run it and how many domains do you expect it to sync (based on historic data)?

@ptkach reviewed 8 files and made 2 comments.
Reviewable status: 8 of 9 files reviewed, 1 unresolved discussion (waiting on gbrodman).


core/src/main/java/google/registry/batch/SyncRemoteCacheAction.java line 153 at r2 (raw file):

            .getResultList();
    if (hosts.isEmpty()) {
      logger.atInfo().log("No domains to process");

hosts?

Copy link
Copy Markdown
Collaborator Author

@gbrodman gbrodman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm thinking we run this every 15-20 minutes (we want data to be at most one hour out of date).

The batch size (the amount that we read from Postgres) is at most 10,000. Looking back at a few recent random days, the number of domains changed (select count(*) from "Domain" where update_timestamp < X and update_timestamp > Y ranges between like 1500 and 15,000. There are significantly fewer host updates.

@gbrodman made 2 comments.
Reviewable status: 8 of 9 files reviewed, 1 unresolved discussion (waiting on ptkach).


core/src/main/java/google/registry/batch/SyncRemoteCacheAction.java line 153 at r2 (raw file):

Previously, ptkach (Pavlo Tkach) wrote…

hosts?

whoops

Copy link
Copy Markdown
Collaborator

@ptkach ptkach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ptkach reviewed 1 file and all commit messages, and resolved 1 discussion.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on gbrodman).

This uses two cursors (one for hosts and one for domains) to track our
progress in "catching up", or syncing recent changes to the database,
using the update-timestamp field. When the cache is in use and fully
caught up, these
cursors should be kept relatively up-to-date to the actual time, i.e.
less than one hour behind
Copy link
Copy Markdown
Collaborator

@ptkach ptkach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ptkach reviewed 1 file and all commit messages.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on gbrodman).

@gbrodman gbrodman added the kokoro:force-run Force a Kokoro build. label May 4, 2026
@domain-registry-eng domain-registry-eng removed the kokoro:force-run Force a Kokoro build. label May 4, 2026
@gbrodman gbrodman added the kokoro:force-run Force a Kokoro build. label May 5, 2026
@domain-registry-eng domain-registry-eng removed the kokoro:force-run Force a Kokoro build. label May 5, 2026
@gbrodman gbrodman added this pull request to the merge queue May 5, 2026
Merged via the queue into google:master with commit 81b3a2f May 5, 2026
14 checks passed
@gbrodman gbrodman deleted the valkeyCursor branch May 5, 2026 16:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants