Skip to content

v2.3.0 — Auto-tuned Parallel Streams + Gzip Writer Pooling

Choose a tag to compare

View all tags
@javimosch javimosch released this 09 Aug 17:54
· 9 commits to master since this release
v2.3.0
69234f4

Summary

  • Auto-tuned parallel streams based on file size (CERN/GridFTP research)
  • Pooled gzip.Writer objects to reduce GC pressure on compressible data
  • Larger relay write queue (128 frames) for better parallel stream pipelining

Auto-tuned parallel streams

File size → stream count mapping based on GridFTP/CERN research (2-10 streams optimal):

  • 5-20 MiB → 2 streams
  • 20-100 MiB → 3 streams
  • 100+ MiB → 4 streams

Override with RCMD_PARALLEL_STREAMS=N (0 = disabled).

Gzip writer pooling

gzip.Writer is ~40KB per allocation. Previously created a new one per chunk. Now pooled with sync.Pool + Reset() for reuse. Helps most with compressible data.

Benchmark (50MB random → rbm21, 77ms RTT)

Method Time vs scp
scp 75.6s 1.0x
rcmd auto-3 streams 69.1s 1.10x

10MB zeros (compressible): 465ms (22 MB/s)
All transfers verified with sha256sum.

Test plan

  • Unit tests pass
  • 10MB random — integrity verified
  • 10MB zeros — integrity verified
  • 50MB random — integrity verified
  • Auto-parallel selects correct stream count by file size

Generated with Devin

Assets 7
Loading