Fix Net::Protocol::BufferedIO#write when sending large multi-byte string

[eitoball]

This commit should fix Net::Protocol::BufferedIO#write when sending large multi-byte string like following example.

$ ruby -rnet/http -rjson -v -e "Net::HTTP.post(URI('http://httpbin.org/post'), { text: 'あ'*100_000 }.to_json, 'Content-Type' => 'application/json')"
ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
Traceback (most recent call last):
        19: from -e:1:in `<main>'
        18: from lib/ruby/2.6.0/net/http.rb:500:in `post'
        17: from lib/ruby/2.6.0/net/http.rb:605:in `start'
        16: from lib/ruby/2.6.0/net/http.rb:920:in `start'
        15: from lib/ruby/2.6.0/net/http.rb:502:in `block in post'
        14: from lib/ruby/2.6.0/net/http.rb:1281:in `post'
        13: from lib/ruby/2.6.0/net/http.rb:1493:in `send_entity'
        12: from lib/ruby/2.6.0/net/http.rb:1479:in `request'
        11: from lib/ruby/2.6.0/net/http.rb:1506:in `transport_request'
        10: from lib/ruby/2.6.0/net/http.rb:1506:in `catch'
         9: from lib/ruby/2.6.0/net/http.rb:1507:in `block in transport_request'
         8: from lib/ruby/2.6.0/net/http/generic_request.rb:123:in `exec'
         7: from lib/ruby/2.6.0/net/http/generic_request.rb:189:in `send_request_with_body'
         6: from lib/ruby/2.6.0/net/protocol.rb:247:in `write'
         5: from lib/ruby/2.6.0/net/protocol.rb:265:in `writing'
         4: from lib/ruby/2.6.0/net/protocol.rb:248:in `block in write'
         3: from lib/ruby/2.6.0/net/protocol.rb:275:in `write0'
         2: from lib/ruby/2.6.0/net/protocol.rb:275:in `each_with_index'
         1: from lib/ruby/2.6.0/net/protocol.rb:275:in `each'
lib/ruby/2.6.0/net/protocol.rb:280:in `block in write0': undefined method `bytesize' for nil:NilClass (NoMethodError)

[ioquatix]

Ouch! Nice catch.

[bjeanes]

I'm not sure if the same issue, but I am having request body corruption when sending large multi-byte bodies, but no apparent exception.

This script reproduces it for me with appropriate body:

# frozen_string_literal: true

require "net/http"
require "net/https"
require "uri"
require "json"

body = File.read("./test-data.txt")
uri = URI.parse "https://httpbin.org/anything"

Net::HTTP.start(uri.host, /^https/.match?(uri.scheme) ? 443 : 80, use_ssl: /^https/.match?(uri.scheme)) do |http|
  req = Net::HTTP::Post.new(uri.request_uri)
  req["Content-Type"] = "text/plain"
  req.body = body

  res = http.request(req)

  p JSON.parse(res.body)["data"] == body
end

This returns false but if I change the above URL to http:// it will return true.

I can't share the request body I'm using as it has sensitive data but if your own test body from this issue doesn't reproduce, I can try to reduce my sample to something shareable.

Is this likely a different symptom of the same issue or a different issue altogether?

[bjeanes]

I just compiled 2.6 with this patch and it does fix the issue.

FWIW, this seems like a pretty major one. This brought down the entirety of all ElasticSearch bulk indexing for our site because the NDJSON bodies were corrupted to the point of being invalid JSON. Worse, some bodies were still valid JSON but the (large) fields had malformed content (i.e. corrupted semantics without corrupting syntax, though this was rarer).

EDIT: I'd like to leave a comment on https://bugs.ruby-lang.org/issues/15468 but I don't seem to see how to do that. I'll open a separate issue for this symptom if that is helpful.

[adamof]

We are seeing this when we use Net:HTTP via Faraday and sending big requests.
lostisland/faraday#842

[zloyrusskiy]

I think that it is necessary to add a test so that the regression does not repeat in the future

[amatsuda]

@zloyrusskiy See the actual commit by nobu 1680a13

[morgoth]

In our case, while requesting some third party API, there was no error raised, but a valid http response returned (however it was a "bad request"), so it silently bitten us. Having 2.6.1 released with this fix would be much appreciated.

[ricardovj]

This also silently caused a bunch of issues for us sending emails and it took us a few days to figure out it was related to the 2.6 release and we had to rollback. Releasing this will be much appreciated 🙏

[oshanz]

It's better to note this down this in the ruby 2.6.0 release page as a known issue

[ioquatix]

@oshanz can you make a PR against https://github.com/ruby/www.ruby-lang.org