Skip to content

send_packets: fix miscalibrated overflow guards in calc_sleep_time (#974)#999

Merged
fklassen merged 2 commits into
v4.5.3-beta1from
fix-974-timing-overflow
Jul 16, 2026
Merged

send_packets: fix miscalibrated overflow guards in calc_sleep_time (#974)#999
fklassen merged 2 commits into
v4.5.3-beta1from
fix-974-timing-overflow

Conversation

@fklassen

Copy link
Copy Markdown
Member

Summary

Fixes #974 (constant-rate -M/-p replays stop sleeping and blast the remainder at full
speed near EOF on long/high-volume replays).

Root cause: in calc_sleep_time() (src/send_packets.c), both the speed_mbpsrate and
speed_packetrate branches guard a multiply-before-divide against COUNTER (64-bit)
overflow using COUNTER_OVERFLOW_RISK (COUNTER_MAX >> 23, ≈ 2.199e12). That threshold is
miscalibrated by orders of magnitude relative to where the guarded multiply actually
overflows, so the "safe" branch is essentially never taken in practice and the multiply
silently overflows on any realistically long replay.

Root-cause arithmetic

COUNTER_OVERFLOW_RISK = (COUNTER_MAX) >> 23 = 2^41 ≈ 2.199e12 (for the default 64-bit
COUNTER).

speed_mbpsrate guards bits_sent * 1000000000. That product overflows COUNTER once:

bits_sent > COUNTER_MAX / 1e9 ≈ 1.8447e10  (~2.3GB sent, at any bps)

That's ~119x below the guard's 2.199e12 threshold — so the guard essentially never
protects the multiply. This lines up almost exactly with the reporter's log in #974: their
divergence starts right around 2.3GB sent ("Actual: 11462618 packets (2302609947 bytes)..."
followed a few thousand packets later by nap=0s 0nsec for the rest of the run), which is
the computed overflow boundary (2,305,843,009 bytes) to within 0.14%.

speed_packetrate guards pkts_sent * 1000000000 * 3600. That overflows once:

pkts_sent > COUNTER_MAX / (1e9 * 3600) ≈ 5.12e6 packets

~4.3e5x below the same 2.199e12 guard threshold. This branch is untouched by #989 and has
the same class of bug — just as live, but with a much lower packet-count trigger point (~5.1M
packets, vs. millions/tens-of-millions being a completely normal packetrate-replay size), so
it's a latent bug that just hasn't been reported yet.

Verified both boundary computations and the reporter's log alignment with a throwaway C
program (compiled, run, then deleted — not part of this diff) comparing the old formula, a
long double reference, and the new formula across: the exact reporter boundary, one bit past
it, multi-terabyte/petabyte-scale bits_sent at low bps, and 40–100 Gbps-class bps at
short duration.

Is #989 sufficient?

#989 (thanks @AB-Coder96 for the reproduction and validation work — the before/after Mbps
numbers on their synthetic 22M-packet pcap are a good confirmation of the speed_mbpsrate
symptom) replaces the guarded multiply with quotient/remainder decomposition:

next_tx_ns = (bits_sent / bps) * 1000000000;
next_tx_ns += ((bits_sent % bps) * 1000000000) / bps;

This fixes the reported long-duration/low-rate case, but the remainder term
(bits_sent % bps) * 1000000000 can itself overflow: bits_sent % bps can be as large as
bps - 1, so this term overflows once bps > COUNTER_MAX / 1e9 ≈ 1.8447e10 bits/sec
(~18.4 Gbps). tcpreplay is a line-rate packet generator (per this project's own design
goals) and 40/100 Gbps-class -M values are realistic, not hypothetical. I confirmed this
concretely with a crafted worst-case remainder at bps = 40 Gbps:

bits_sent=159999999999 bps=40000000000
PR#989 formula result=3077662796 ns   (reference ≈ 4000000000 ns)   -- ~23% low, wrong

So #989 alone would fix the reported case but reintroduce a (smaller, but real) overflow
window at high -M rates. #982 is broader (touches both branches) but reorders the existing
guarded-multiply pattern using the same miscalibrated COUNTER_OVERFLOW_RISK threshold, so
it doesn't address the root cause either (the "safe" branch is still gated behind a threshold
~119x/~4.3e5x too high to ever trigger where it's actually needed).

What this PR does

For both speed_mbpsrate and speed_packetrate, replace the guarded-multiply / threshold
pattern with a single multiply-then-divide done in a wider-than-COUNTER intermediate
(unsigned __int128), which cannot overflow for any COUNTER-representable inputs on either
side — no threshold, no decomposition, no residual overflow window at any realistic (or even
unrealistic) bps/pph:

next_tx_ns = (COUNTER)(((unsigned __int128)bits_sent * 1000000000ULL) / bps);
...
next_tx_ns = (COUNTER)(((unsigned __int128)pkts_sent * 1000000000ULL * 3600ULL) / pph);

This is cheaper than both the old code and #989's version: one multiply + one divide per
call, vs. the old code's branch + potential extra multiply, or #989's extra % + multiply +
divide. No new syscalls, no heap allocation.

__int128/unsigned __int128 is a GCC/Clang extension, not standard C. It's available on
GCC and Clang for all of this project's stated 64-bit targets (Linux, macOS/Darwin, *BSD,
Solaris, Haiku, Cygwin) when built with those compilers, but it is non-standard, so I've
gated it behind #if defined(__SIZEOF_INT128__) (the compiler-defined macro that indicates
the type's availability) with a portable fallback for any compiler where it's absent:

next_tx_ns = (bits_sent * 1000ULL) / bps * 1000000ULL;
...
next_tx_ns = ((pkts_sent * 1000000ULL) / pph * 1000ULL) * 3600ULL;

This fallback is exactly the old code's own "safe" branch (the one previously gated behind
the miscalibrated threshold), just applied unconditionally instead of behind a guard that
almost never triggered it. It remains overflow-free for bits_sent up to ~1.8e16
(~2.3 petabytes sent) / pkts_sent up to ~1.8e13 packets — far beyond any real replay — so
it's a safe portability fallback, not a compromise on correctness for realistic inputs.

Flagging for review: the __int128 usage is a portability tradeoff worth a human
sign-off per project convention — happy to drop the __int128 path entirely and just ship the
unconditional-fallback formula everywhere if that's preferred (it's already correct for every
realistic input; __int128 only buys correctness at truly extreme, unrealistic values).

Out of scope / noted for a follow-up: while auditing this function, I noticed
*skip_length = ((tx_ns - next_tx_ns) * bps) / 8000000000; a few lines below the
speed_mbpsrate fix has a similar-shaped multiply that can overflow at very high bps
combined with a large scheduling-stall-induced tx_ns - next_tx_ns delta (back-of-envelope:
~184ms stall at 100Gbps is enough). I did not touch it since it's a distinct code path from
what #974 reports and wasn't touched by #989/#982 either — flagging in case a human wants a
follow-up.

Verification

  • Clean out-of-tree build: mkdir build && cd build && ../autogen.sh && ../configure --disable-local-libopts --enable-debug && make — succeeds with no warnings on
    send_packets.c (verified for both tcpreplay and tcpreplay-edit object builds, the two
    binaries that compile this file).
  • No live-traffic test run (this sandbox has no root/NIC), so build correctness + reasoned
    arithmetic verification is the primary gate here, per the numbers above and the boundary
    tests below.
  • Wrote (and deleted before committing — not part of this diff) a standalone C program
    exercising the old formula, the __int128 fix, the fallback formula, and Fix Mbps pacing overflow on long replays #989's own
    formula, cross-checked against a long double reference, across:
    • the exact reporter boundary (bits_sent ≈ 1.8447e10, bps = -M 30) — reproduces the
      old code's bug (result jumps from a correct ~6.15e11 ns to just 9 ns one bit later);
    • 5TB sent at -M 1, and bits_sent near 2^63 at -M 1 (multi-terabyte/petabyte-scale,
      low bps);
    • 100 Gbps-class bps at short duration (1GB/10GB sent);
    • the crafted worst-case remainder at 40 Gbps that breaks Fix Mbps pacing overflow on long replays #989's formula specifically
      (shown above);
    • speed_packetrate at the ~5.12M-packet boundary, at [Bug] Tcpreplay ceases to sleep between packets when executed with -M 10 #974's own 22M-packet scale, and at
      4 billion packets (spanning the 32-bit boundary).
    • In every case the __int128 fix matched the long double reference; the fallback
      formula matched to within sub-millisecond truncation (the same truncation
      characteristics as the old code's already-shipped "safe" branch — not a new regression).

References

galastar and others added 2 commits July 16, 2026 12:56
)

The speed_mbpsrate and speed_packetrate branches of calc_sleep_time()
each had a tiered guard meant to avoid overflowing COUNTER (64-bit)
before dividing, gated on COUNTER_OVERFLOW_RISK (COUNTER_MAX >> 23,
~2.2e12). In both cases the guard's threshold was many orders of
magnitude higher than the point where the guarded multiply actually
overflows:

- speed_mbpsrate: bits_sent * 1e9 overflows once bits_sent exceeds
  ~1.8e10 (~2.3GB sent) -- ~119x below the guard's threshold.
- speed_packetrate: pkts_sent * 1e9 * 3600 overflows once pkts_sent
  exceeds ~5.1M packets -- ~4.3e5x below the guard's threshold.

So for any real long-running -M/-p replay, the "safe" branch was
essentially never taken and the multiply silently overflowed,
wrapping next_tx_ns to a small/garbage value and causing tcpreplay to
stop sleeping between packets near EOF, exactly matching #974's
reported symptom (confirmed against the reporter's log: their
divergence starts at ~2.3GB sent, matching the computed boundary).

Replace both guarded multiplies with a widened-precision multiply
(unsigned __int128, available on GCC/Clang for all listed 64-bit
targets) divided by the rate, which cannot overflow for any
COUNTER-representable input, with a portable non-__int128 fallback
that unconditionally uses the previously-underused-but-safe
reordering (multiply/divide split around the division) instead of a
threshold guard.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Credit @blaa (reported #974) and @AB-Coder96 (diagnosed the bug,
validated repro numbers, prototype fix in #989).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@fklassen
fklassen marked this pull request as ready for review July 16, 2026 21:04
@fklassen
fklassen merged commit a21ea03 into v4.5.3-beta1 Jul 16, 2026
2 checks passed
@fklassen
fklassen deleted the fix-974-timing-overflow branch July 16, 2026 21:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants