feat: stream multipart file uploads to reduce memory usage

[jnunemaker]

Summary

• Fixes excessive memory usage when uploading files via multipart form data
• Previously, uploading a 10 MB file consumed ~40 MB of memory (4x the file size)
• Now files are streamed in 64 KB chunks, drastically reducing memory footprint

Root Cause

The issue was in Request::Body#generate_multipart which:

1. Read entire file into memory with file.read
2. Concatenated it into a growing string with <<
3. Caused multiple string reallocations

Solution

Introduced StreamingMultipartBody class that implements an IO-like interface (read, size, rewind) which Net::HTTP can use via body_stream. Files are read in chunks on-demand instead of being loaded entirely into memory.

Changes

File	Description
lib/httparty/request/streaming_multipart_body.rb	New streaming IO wrapper
lib/httparty/request/body.rb	Added streaming?, to_stream, prepared_parts
lib/httparty/request.rb	Use body_stream for file uploads

Usage

Streaming is automatic for file uploads. To opt-out:

HTTParty.post(url, body: { file: File.open('large.bin') }, stream_body: false)

Test plan

• Added unit tests for StreamingMultipartBody (read, size, rewind, chunking)
• Added unit tests for Body#streaming? and Body#to_stream
• Added integration tests verifying body_stream is used for file uploads
• Added test verifying streaming produces identical content to non-streaming
• All 760 existing tests pass
• All 186 Cucumber scenarios pass

Fixes issue reported in: https://gist.github.com/janko/238bbcc78b369ce3438365e5507bc671

🤖 Generated with Claude Code

[jnunemaker]

Based on this gist, I ran the test again and the results show a massive improvement:

Memory Comparison Results (10 MB file upload)

Library	Memory	Notes
HTTParty (streaming - NEW)	0.00 MB	🎉 Essentially zero!
RestClient	4.55 MB
Net::HTTP	11.23 MB
HTTParty (non-streaming - OLD)	20.05 MB

Original gist results (for reference):

Library	Memory
http.rb	0.10 MB
Net::HTTP	0.02 MB
RestClient	9.03 MB
HTTParty	40.03 MB ← was the problem

Summary

HTTParty with streaming now uses essentially zero additional memory for file uploads,
making it competitive with (or better than) http.rb's streaming implementation. The fix
reduced memory usage from 40 MB to ~0 MB for a 10 MB file upload.

[jnunemaker]

@janko thanks for making that gist years ago. I found it and put it on my todo list. Hadn't went after it until today. HTTParty is hopefully better now.