Add an action to sync the remote Valkey cache#3032
Conversation
ptkach
left a comment
There was a problem hiding this comment.
How frequently do you intent to run it and how many domains do you expect it to sync (based on historic data)?
@ptkach reviewed 8 files and made 2 comments.
Reviewable status: 8 of 9 files reviewed, 1 unresolved discussion (waiting on gbrodman).
core/src/main/java/google/registry/batch/SyncRemoteCacheAction.java line 153 at r2 (raw file):
.getResultList(); if (hosts.isEmpty()) { logger.atInfo().log("No domains to process");
hosts?
gbrodman
left a comment
There was a problem hiding this comment.
I'm thinking we run this every 15-20 minutes (we want data to be at most one hour out of date).
The batch size (the amount that we read from Postgres) is at most 10,000. Looking back at a few recent random days, the number of domains changed (select count(*) from "Domain" where update_timestamp < X and update_timestamp > Y ranges between like 1500 and 15,000. There are significantly fewer host updates.
@gbrodman made 2 comments.
Reviewable status: 8 of 9 files reviewed, 1 unresolved discussion (waiting on ptkach).
core/src/main/java/google/registry/batch/SyncRemoteCacheAction.java line 153 at r2 (raw file):
Previously, ptkach (Pavlo Tkach) wrote…
hosts?
whoops
ptkach
left a comment
There was a problem hiding this comment.
@ptkach reviewed 1 file and all commit messages, and resolved 1 discussion.
Reviewable status:complete! all files reviewed, all discussions resolved (waiting on gbrodman).
This uses two cursors (one for hosts and one for domains) to track our progress in "catching up", or syncing recent changes to the database, using the update-timestamp field. When the cache is in use and fully caught up, these cursors should be kept relatively up-to-date to the actual time, i.e. less than one hour behind
ptkach
left a comment
There was a problem hiding this comment.
@ptkach reviewed 1 file and all commit messages.
Reviewable status:complete! all files reviewed, all discussions resolved (waiting on gbrodman).
This uses two cursors (one for hosts and one for domains) to track our progress in "catching up", or syncing recent changes to the database, using the update-timestamp field. When the cache is in use and fully caught up, these
cursors should be kept relatively up-to-date to the actual time, i.e. less than one hour behind
note: while the generated schema changes due to the cursor enum additions, the golden file doesn't change because our SQL scripts server-side don't enforce the enum restriction
This change is