Re-instantiate replay progress reporting #1096
Conversation
| last_logged_percentage = percentage; | ||
| } | ||
| // Print _something_ even if we don't know what's still ahead. | ||
| } else if tx_offset % 10_000 == 0 { |
There was a problem hiding this comment.
Q: In what case would this happen? Would this flood the log?
There was a problem hiding this comment.
This would happen for future implementations of the History trait which, for some reason, can’t determine how many txs are yet to come. For example, apply could be continuously fed from a replication stream.
In this case, we’d print a log every 10k txs, which seems reasonable.
| let progress = |tx_offset: u64| { | ||
| if let Some(max_tx_offset) = max_tx_offset { | ||
| let percentage = f64::floor((tx_offset as f64 / max_tx_offset as f64) * 100.0) as i32; | ||
| if percentage > last_logged_percentage && percentage % 10 == 0 { |
There was a problem hiding this comment.
Our general UX issue with this sort of logging would be that if it takes a really long time to get to the first 10% we really don't know how long it will take to complete the reload until we see that first 10% log. I would suggest changing this so that it outputs a log every 15 seconds instead of doing a percentage based log. If its not an easy change then don't worry about it this is fine for now.
There was a problem hiding this comment.
Hm, ya, not sure if we'd want to spawn a thread just for this, or ask for the system time a couple of million times. I'll take a note that we might want to stick a tokio::runtime::Handle into RelationalDB for these kinds of things.
Useful for reporting replay progress. Include note that it is somewhat similar to `std::iter::Iterator::size_hint`.
The percentage is calculated as starting from the zero offset, although that may change in the future.
kim
force-pushed
the
kim/commitlog2/replay-progress
branch
from
548bc2e
to
c675c8e
Compare
Description of Changes
Output was wrongly logged at
debug, and percentage reporting was missing a way to obtain the max offset to expect.Note that this may need some reconsideration once we apply history suffixes. Do not rely on the output as is.
API and ABI breaking changes
Nada
Expected complexity level and risk
1
Testing