Stable upload progress and honest ETA in miren deploy#804
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (4)
🚧 Files skipped from review as they are similar to previous changes (3)
📝 WalkthroughWalkthroughThis PR changes upload progress reporting in the deploy CLI explain mode from displaying estimated total compressed bytes to displaying estimated time remaining (ETA). The Comment |
The "Uploading artifacts: N% — uploaded / ~TOTAL at SPEED" line had a total estimate that wandered all over the place during slow uploads (observed: ~974 KB initial, climbing to ~80 MB, periodically dropping back to ~20 MB). The math was sampling uncompressed-bytes-in and compressed-bytes-out on different sides of a gzip buffer plus io.Pipe, so under network back-pressure their ratio was meaningless. Drop the synthetic total entirely. The percentage still works because both numerator (written) and denominator (totalUncompressed) only count the delta-filtered subset of files. Add an ETA extrapolated in the time domain: elapsed * (1 - frac) / frac. This avoids the unit-mixing that broke the old projection. A 5s warmup gates it so the first few wildly-fluctuating ticks do not show, and the display uses "(eta ~Xh Ym)" with a tilde to telegraph that it is approximate. Closes MIR-1136
77de783 to
cde6863
Compare

While babysitting a slow deploy I noticed the progress line was lying to me. The "total" in
Uploading artifacts: 0% — 24 KB / ~45.7 MB at 2.4 KB/swas swinging wildly: it started at ~974 KB, climbed steadily to ~80 MB over a few minutes, dropped back to ~20 MB, climbed again. Useless for figuring out how long the upload had left.Tracing the math, the underlying bug was that
EstimatedTotalBytes = BytesRead / Fractionwas mixing two counters sampled on different sides of the gzip pipeline.writtenis uncompressed source bytes that have entered tar/gzip, whileBytesReadis compressed bytes drained by the HTTP uploader on the other side of a gzip buffer plus io.Pipe. Under network back-pressure they advance in bursts at very different ratios, and their quotient is meaningless.The fix is to stop pretending we know the compressed total. We don't, and projecting it from the runtime ratio doesn't work. The percentage itself was always fine, because both numerator and denominator are uncompressed bytes from the delta-filtered subset of files, so it correctly reports "fraction of the upload work consumed."
Since estimating remaining time is genuinely useful on slow uploads (the calculator-free "this will take 8 hours" use case), I also wove in a time-domain ETA:
elapsed * (1 - frac) / frac. That formula sidesteps the unit-mixing trap entirely. There's a 5s warmup so early wild numbers don't show, and the display uses a tilde (eta ~8h 15m) to be upfront about being approximate.The new line reads
Uploading artifacts: 1% — 1.0 MB at 2.4 KB/s (eta ~8h 15m), in both explain mode and the TUI.Closes MIR-1136