feat(transfer): add wire_to_flat_ndx partition_point counters (INC_RECURSE I4)#4120
Merged
Conversation
…CURSE I4) Adds INC_RECURSE diagnostic counter I4 (#2199): two process-global AtomicU64 counters per side (generator and receiver) that track every wire_to_flat_ndx / flat_to_wire_ndx invocation and the cumulative partition_point comparison depth across the segment table. The counters are sampled at end-of-transfer in the existing finalize paths used by PR #4103 for the first-byte latency probe: - generator: GeneratorContext::run, just before returning GeneratorStats - receiver: finalize_transfer in receiver/transfer/phases.rs Both sides emit a debug_log!(Genr, 1, ...) line with the totals and, when the optional tracing feature is enabled, a tracing::debug! event at rsync::generator::ndx_convert / rsync::receiver::ndx_convert. A small partition_point_depth(len) helper approximates the worst-case comparison count as floor(log2(len)) + 1. Public API: a new ndx_convert_totals() free function in both modules returns (call_count, cumulative_partition_point_depth). It is the only addition to the public surface; no CLI flag, no wire-format change, no change to the existing wire_to_flat_ndx / flat_to_wire_ndx signatures. Unit tests in both crates exercise the converters and assert: - the call counter monotonically grows by at least the number of invocations made in the test (>= guards against concurrent test runs incrementing the shared counter) - the cumulative partition_point depth grows by at least N * depth(segment_table_len) after N calls against a known-size segment table Closes #2199
3 tasks
oferchen
added a commit
that referenced
this pull request
May 16, 2026
…4142) * feat(transfer): instrument prepare_pending_acl cost (INC_RECURSE I5) Adds INC_RECURSE diagnostic counter I5 (#2200): two process-global AtomicU64 counters that track every prepare_pending_acl invocation on the generator file-list path and the cumulative wall time spent inside the helper across all generator transfers. The counters fire from a single record_prepare_acl helper bumped from prepare_pending_acl itself, which wraps the existing body in an Instant::now() pair. The early-return path (entry.is_symlink() or !flags.acls) is still counted so per-segment call rate stays comparable to entry count, but contributes a near-zero ns delta. Totals are sampled in GeneratorContext::run via prepare_acl_totals() in the same finalize block used by PR #4103 (I1 first-byte latency), PR #4120 (I4 NDX partition_point) and PR #4121 (I3 flush rate), and emitted via: - debug_log!(Genr, 1, "generator prepare_pending_acl totals: \ calls=N elapsed_ns=M") - tracing::debug! event on rsync::generator::prepare_acl when the optional tracing feature is enabled Two unit tests in crates/transfer/src/generator/tests.rs cover the new counters: - prepare_acl_call_counter_increments asserts the call counter and the cumulative elapsed_ns counter grow by at least N and sum-of-durations after N synthetic record_prepare_acl invocations. The assertion uses >= because the counter is shared across the process and other tests may run concurrently. - prepare_acl_totals_observable_without_prep asserts the totals snapshot is a pure read - constructing a generator does not bump the counter on its own, so two adjacent reads with no intervening prep call must return identical values. No new CLI flag, no wire-format change, no change to the public prepare_pending_acl signature. Closes #2200 * fix(transfer): rewrap rustdoc list items for prepare_pending_acl counters
4 tasks
oferchen
added a commit
that referenced
this pull request
May 16, 2026
…RSE I2) (#4143) Adds INC_RECURSE diagnostic counter I2 (#2197): two process-global AtomicU64 counters that track every encode_and_send_segment invocation on the generator file-list path and the cumulative wall time spent inside the helper across all generator transfers. The counters fire from a single record_segment_dispatch helper bumped from encode_and_send_segment itself, which wraps the existing body in an Instant::now() pair. The zero-entry early-return path is still counted so per-transfer dispatch rate stays comparable to segment scheduler activity, but contributes a near-zero ns delta. Totals are sampled in GeneratorContext::run via segment_dispatch_totals() in the same finalize block used by PR #4103 (I1 first-byte latency), PR #4120 (I4 NDX partition_point), PR #4121 (I3 flush rate) and PR #4142 (I5 prepare_pending_acl), and emitted via: - debug_log!(Genr, 1, "generator encode_and_send_segment totals: \ calls=N elapsed_ns=M") - tracing::debug! event on rsync::generator::segment_dispatch when the optional tracing feature is enabled Two unit tests in crates/transfer/src/generator/tests.rs cover the new counters: - segment_dispatch_call_counter_increments asserts the call counter and the cumulative elapsed_ns counter grow by at least N and sum-of-durations after N synthetic record_segment_dispatch invocations. The assertion uses >= because the counter is shared across the process and other tests may run concurrently. - segment_dispatch_totals_observable_without_dispatch asserts the totals snapshot is a pure read - constructing a generator does not bump the counter on its own, so two adjacent reads with no intervening dispatch call must return identical values. No new CLI flag, no wire-format change, no change to the public encode_and_send_segment signature. Closes #2197
Merged
oferchen
added a commit
that referenced
this pull request
May 18, 2026
…CURSE I4) (#4120) * feat(transfer): add wire_to_flat_ndx partition_point counters (INC_RECURSE I4) Adds INC_RECURSE diagnostic counter I4 (#2199): two process-global AtomicU64 counters per side (generator and receiver) that track every wire_to_flat_ndx / flat_to_wire_ndx invocation and the cumulative partition_point comparison depth across the segment table. The counters are sampled at end-of-transfer in the existing finalize paths used by PR #4103 for the first-byte latency probe: - generator: GeneratorContext::run, just before returning GeneratorStats - receiver: finalize_transfer in receiver/transfer/phases.rs Both sides emit a debug_log!(Genr, 1, ...) line with the totals and, when the optional tracing feature is enabled, a tracing::debug! event at rsync::generator::ndx_convert / rsync::receiver::ndx_convert. A small partition_point_depth(len) helper approximates the worst-case comparison count as floor(log2(len)) + 1. Public API: a new ndx_convert_totals() free function in both modules returns (call_count, cumulative_partition_point_depth). It is the only addition to the public surface; no CLI flag, no wire-format change, no change to the existing wire_to_flat_ndx / flat_to_wire_ndx signatures. Unit tests in both crates exercise the converters and assert: - the call counter monotonically grows by at least the number of invocations made in the test (>= guards against concurrent test runs incrementing the shared counter) - the cumulative partition_point depth grows by at least N * depth(segment_table_len) after N calls against a known-size segment table Closes #2199 * chore: regenerate Cargo.lock * style: cargo fmt
oferchen
added a commit
that referenced
this pull request
May 18, 2026
…4142) * feat(transfer): instrument prepare_pending_acl cost (INC_RECURSE I5) Adds INC_RECURSE diagnostic counter I5 (#2200): two process-global AtomicU64 counters that track every prepare_pending_acl invocation on the generator file-list path and the cumulative wall time spent inside the helper across all generator transfers. The counters fire from a single record_prepare_acl helper bumped from prepare_pending_acl itself, which wraps the existing body in an Instant::now() pair. The early-return path (entry.is_symlink() or !flags.acls) is still counted so per-segment call rate stays comparable to entry count, but contributes a near-zero ns delta. Totals are sampled in GeneratorContext::run via prepare_acl_totals() in the same finalize block used by PR #4103 (I1 first-byte latency), PR #4120 (I4 NDX partition_point) and PR #4121 (I3 flush rate), and emitted via: - debug_log!(Genr, 1, "generator prepare_pending_acl totals: \ calls=N elapsed_ns=M") - tracing::debug! event on rsync::generator::prepare_acl when the optional tracing feature is enabled Two unit tests in crates/transfer/src/generator/tests.rs cover the new counters: - prepare_acl_call_counter_increments asserts the call counter and the cumulative elapsed_ns counter grow by at least N and sum-of-durations after N synthetic record_prepare_acl invocations. The assertion uses >= because the counter is shared across the process and other tests may run concurrently. - prepare_acl_totals_observable_without_prep asserts the totals snapshot is a pure read - constructing a generator does not bump the counter on its own, so two adjacent reads with no intervening prep call must return identical values. No new CLI flag, no wire-format change, no change to the public prepare_pending_acl signature. Closes #2200 * fix(transfer): rewrap rustdoc list items for prepare_pending_acl counters
oferchen
added a commit
that referenced
this pull request
May 18, 2026
…RSE I2) (#4143) Adds INC_RECURSE diagnostic counter I2 (#2197): two process-global AtomicU64 counters that track every encode_and_send_segment invocation on the generator file-list path and the cumulative wall time spent inside the helper across all generator transfers. The counters fire from a single record_segment_dispatch helper bumped from encode_and_send_segment itself, which wraps the existing body in an Instant::now() pair. The zero-entry early-return path is still counted so per-transfer dispatch rate stays comparable to segment scheduler activity, but contributes a near-zero ns delta. Totals are sampled in GeneratorContext::run via segment_dispatch_totals() in the same finalize block used by PR #4103 (I1 first-byte latency), PR #4120 (I4 NDX partition_point), PR #4121 (I3 flush rate) and PR #4142 (I5 prepare_pending_acl), and emitted via: - debug_log!(Genr, 1, "generator encode_and_send_segment totals: \ calls=N elapsed_ns=M") - tracing::debug! event on rsync::generator::segment_dispatch when the optional tracing feature is enabled Two unit tests in crates/transfer/src/generator/tests.rs cover the new counters: - segment_dispatch_call_counter_increments asserts the call counter and the cumulative elapsed_ns counter grow by at least N and sum-of-durations after N synthetic record_segment_dispatch invocations. The assertion uses >= because the counter is shared across the process and other tests may run concurrently. - segment_dispatch_totals_observable_without_dispatch asserts the totals snapshot is a pure read - constructing a generator does not bump the counter on its own, so two adjacent reads with no intervening dispatch call must return identical values. No new CLI flag, no wire-format change, no change to the public encode_and_send_segment signature. Closes #2197
oferchen
added a commit
that referenced
this pull request
May 18, 2026
…CURSE I4) (#4120) * feat(transfer): add wire_to_flat_ndx partition_point counters (INC_RECURSE I4) Adds INC_RECURSE diagnostic counter I4 (#2199): two process-global AtomicU64 counters per side (generator and receiver) that track every wire_to_flat_ndx / flat_to_wire_ndx invocation and the cumulative partition_point comparison depth across the segment table. The counters are sampled at end-of-transfer in the existing finalize paths used by PR #4103 for the first-byte latency probe: - generator: GeneratorContext::run, just before returning GeneratorStats - receiver: finalize_transfer in receiver/transfer/phases.rs Both sides emit a debug_log!(Genr, 1, ...) line with the totals and, when the optional tracing feature is enabled, a tracing::debug! event at rsync::generator::ndx_convert / rsync::receiver::ndx_convert. A small partition_point_depth(len) helper approximates the worst-case comparison count as floor(log2(len)) + 1. Public API: a new ndx_convert_totals() free function in both modules returns (call_count, cumulative_partition_point_depth). It is the only addition to the public surface; no CLI flag, no wire-format change, no change to the existing wire_to_flat_ndx / flat_to_wire_ndx signatures. Unit tests in both crates exercise the converters and assert: - the call counter monotonically grows by at least the number of invocations made in the test (>= guards against concurrent test runs incrementing the shared counter) - the cumulative partition_point depth grows by at least N * depth(segment_table_len) after N calls against a known-size segment table Closes #2199 * chore: regenerate Cargo.lock * style: cargo fmt
oferchen
added a commit
that referenced
this pull request
May 18, 2026
…4142) * feat(transfer): instrument prepare_pending_acl cost (INC_RECURSE I5) Adds INC_RECURSE diagnostic counter I5 (#2200): two process-global AtomicU64 counters that track every prepare_pending_acl invocation on the generator file-list path and the cumulative wall time spent inside the helper across all generator transfers. The counters fire from a single record_prepare_acl helper bumped from prepare_pending_acl itself, which wraps the existing body in an Instant::now() pair. The early-return path (entry.is_symlink() or !flags.acls) is still counted so per-segment call rate stays comparable to entry count, but contributes a near-zero ns delta. Totals are sampled in GeneratorContext::run via prepare_acl_totals() in the same finalize block used by PR #4103 (I1 first-byte latency), PR #4120 (I4 NDX partition_point) and PR #4121 (I3 flush rate), and emitted via: - debug_log!(Genr, 1, "generator prepare_pending_acl totals: \ calls=N elapsed_ns=M") - tracing::debug! event on rsync::generator::prepare_acl when the optional tracing feature is enabled Two unit tests in crates/transfer/src/generator/tests.rs cover the new counters: - prepare_acl_call_counter_increments asserts the call counter and the cumulative elapsed_ns counter grow by at least N and sum-of-durations after N synthetic record_prepare_acl invocations. The assertion uses >= because the counter is shared across the process and other tests may run concurrently. - prepare_acl_totals_observable_without_prep asserts the totals snapshot is a pure read - constructing a generator does not bump the counter on its own, so two adjacent reads with no intervening prep call must return identical values. No new CLI flag, no wire-format change, no change to the public prepare_pending_acl signature. Closes #2200 * fix(transfer): rewrap rustdoc list items for prepare_pending_acl counters
oferchen
added a commit
that referenced
this pull request
May 18, 2026
…RSE I2) (#4143) Adds INC_RECURSE diagnostic counter I2 (#2197): two process-global AtomicU64 counters that track every encode_and_send_segment invocation on the generator file-list path and the cumulative wall time spent inside the helper across all generator transfers. The counters fire from a single record_segment_dispatch helper bumped from encode_and_send_segment itself, which wraps the existing body in an Instant::now() pair. The zero-entry early-return path is still counted so per-transfer dispatch rate stays comparable to segment scheduler activity, but contributes a near-zero ns delta. Totals are sampled in GeneratorContext::run via segment_dispatch_totals() in the same finalize block used by PR #4103 (I1 first-byte latency), PR #4120 (I4 NDX partition_point), PR #4121 (I3 flush rate) and PR #4142 (I5 prepare_pending_acl), and emitted via: - debug_log!(Genr, 1, "generator encode_and_send_segment totals: \ calls=N elapsed_ns=M") - tracing::debug! event on rsync::generator::segment_dispatch when the optional tracing feature is enabled Two unit tests in crates/transfer/src/generator/tests.rs cover the new counters: - segment_dispatch_call_counter_increments asserts the call counter and the cumulative elapsed_ns counter grow by at least N and sum-of-durations after N synthetic record_segment_dispatch invocations. The assertion uses >= because the counter is shared across the process and other tests may run concurrently. - segment_dispatch_totals_observable_without_dispatch asserts the totals snapshot is a pure read - constructing a generator does not bump the counter on its own, so two adjacent reads with no intervening dispatch call must return identical values. No new CLI flag, no wire-format change, no change to the public encode_and_send_segment signature. Closes #2197
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
AtomicU64s per side that track everywire_to_flat_ndx/flat_to_wire_ndxcall and the cumulativepartition_pointcomparison depth.finalize_transfer) viadebug_log!(Genr, 1, ...)and, under the optionaltracingfeature,tracing::debug!atrsync::generator::ndx_convert/rsync::receiver::ndx_convert.ndx_convert_totals()free function in bothtransfer::generatorandtransfer::receiverreturning(call_count, cumulative_partition_point_depth). No CLI flag, no wire-format change, no change to existing function signatures.Test plan
cargo nextest run -p transferexercises the four new unit tests (two per side) asserting the call counter increments and the cumulative partition_point depth grows under N invocations against a known-size segment table.Closes #2199