Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Net::Protocol::BufferedIO#write when sending large multi-byte string #2058

Conversation

@eitoball
Copy link

@eitoball eitoball commented Dec 27, 2018

This commit should fix Net::Protocol::BufferedIO#write when sending large multi-byte string like following example.

$ ruby -rnet/http -rjson -v -e "Net::HTTP.post(URI('http://httpbin.org/post'), { text: 'あ'*100_000 }.to_json, 'Content-Type' => 'application/json')"
ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
Traceback (most recent call last):
        19: from -e:1:in `<main>'
        18: from lib/ruby/2.6.0/net/http.rb:500:in `post'
        17: from lib/ruby/2.6.0/net/http.rb:605:in `start'
        16: from lib/ruby/2.6.0/net/http.rb:920:in `start'
        15: from lib/ruby/2.6.0/net/http.rb:502:in `block in post'
        14: from lib/ruby/2.6.0/net/http.rb:1281:in `post'
        13: from lib/ruby/2.6.0/net/http.rb:1493:in `send_entity'
        12: from lib/ruby/2.6.0/net/http.rb:1479:in `request'
        11: from lib/ruby/2.6.0/net/http.rb:1506:in `transport_request'
        10: from lib/ruby/2.6.0/net/http.rb:1506:in `catch'
         9: from lib/ruby/2.6.0/net/http.rb:1507:in `block in transport_request'
         8: from lib/ruby/2.6.0/net/http/generic_request.rb:123:in `exec'
         7: from lib/ruby/2.6.0/net/http/generic_request.rb:189:in `send_request_with_body'
         6: from lib/ruby/2.6.0/net/protocol.rb:247:in `write'
         5: from lib/ruby/2.6.0/net/protocol.rb:265:in `writing'
         4: from lib/ruby/2.6.0/net/protocol.rb:248:in `block in write'
         3: from lib/ruby/2.6.0/net/protocol.rb:275:in `write0'
         2: from lib/ruby/2.6.0/net/protocol.rb:275:in `each_with_index'
         1: from lib/ruby/2.6.0/net/protocol.rb:275:in `each'
lib/ruby/2.6.0/net/protocol.rb:280:in `block in write0': undefined method `bytesize' for nil:NilClass (NoMethodError)
This commit should fix Net::Protocol::BufferedIO#write when sending
large multi-byte string like following example.

```
$ uby -rnet/http -rjson -v -e "Net::HTTP.post(URI('http://httpbin.org/post'), { text: 'あ'*100_000 }.to_json, 'Content-Type' => 'application/json')"
ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
Traceback (most recent call last):
        19: from -e:1:in `<main>'
        18: from lib/ruby/2.6.0/net/http.rb:500:in `post'
        17: from lib/ruby/2.6.0/net/http.rb:605:in `start'
        16: from lib/ruby/2.6.0/net/http.rb:920:in `start'
        15: from lib/ruby/2.6.0/net/http.rb:502:in `block in post'
        14: from lib/ruby/2.6.0/net/http.rb:1281:in `post'
        13: from lib/ruby/2.6.0/net/http.rb:1493:in `send_entity'
        12: from lib/ruby/2.6.0/net/http.rb:1479:in `request'
        11: from lib/ruby/2.6.0/net/http.rb:1506:in `transport_request'
        10: from lib/ruby/2.6.0/net/http.rb:1506:in `catch'
         9: from lib/ruby/2.6.0/net/http.rb:1507:in `block in transport_request'
         8: from lib/ruby/2.6.0/net/http/generic_request.rb:123:in `exec'
         7: from lib/ruby/2.6.0/net/http/generic_request.rb:189:in `send_request_with_body'
         6: from lib/ruby/2.6.0/net/protocol.rb:247:in `write'
         5: from lib/ruby/2.6.0/net/protocol.rb:265:in `writing'
         4: from lib/ruby/2.6.0/net/protocol.rb:248:in `block in write'
         3: from lib/ruby/2.6.0/net/protocol.rb:275:in `write0'
         2: from lib/ruby/2.6.0/net/protocol.rb:275:in `each_with_index'
         1: from lib/ruby/2.6.0/net/protocol.rb:275:in `each'
lib/ruby/2.6.0/net/protocol.rb:280:in `block in write0': undefined method `bytesize' for nil:NilClass (NoMethodError)
```
@matzbot matzbot closed this in 1680a13 Dec 27, 2018
@ioquatix
Copy link
Member

@ioquatix ioquatix commented Dec 27, 2018

Ouch! Nice catch.

@bjeanes
Copy link

@bjeanes bjeanes commented Jan 8, 2019

I'm not sure if the same issue, but I am having request body corruption when sending large multi-byte bodies, but no apparent exception.

This script reproduces it for me with appropriate body:

# frozen_string_literal: true

require "net/http"
require "net/https"
require "uri"
require "json"

body = File.read("./test-data.txt")
uri = URI.parse "https://httpbin.org/anything"

Net::HTTP.start(uri.host, /^https/.match?(uri.scheme) ? 443 : 80, use_ssl: /^https/.match?(uri.scheme)) do |http|
  req = Net::HTTP::Post.new(uri.request_uri)
  req["Content-Type"] = "text/plain"
  req.body = body

  res = http.request(req)

  p JSON.parse(res.body)["data"] == body
end

This returns false but if I change the above URL to http:// it will return true.

I can't share the request body I'm using as it has sensitive data but if your own test body from this issue doesn't reproduce, I can try to reduce my sample to something shareable.

Is this likely a different symptom of the same issue or a different issue altogether?

@bjeanes
Copy link

@bjeanes bjeanes commented Jan 8, 2019

I just compiled 2.6 with this patch and it does fix the issue.

FWIW, this seems like a pretty major one. This brought down the entirety of all ElasticSearch bulk indexing for our site because the NDJSON bodies were corrupted to the point of being invalid JSON. Worse, some bodies were still valid JSON but the (large) fields had malformed content (i.e. corrupted semantics without corrupting syntax, though this was rarer).

EDIT: I'd like to leave a comment on https://bugs.ruby-lang.org/issues/15468 but I don't seem to see how to do that. I'll open a separate issue for this symptom if that is helpful.

@adamof
Copy link

@adamof adamof commented Jan 8, 2019

We are seeing this when we use Net:HTTP via Faraday and sending big requests.
lostisland/faraday#842

@zloyrusskiy
Copy link

@zloyrusskiy zloyrusskiy commented Jan 9, 2019

I think that it is necessary to add a test so that the regression does not repeat in the future

@amatsuda
Copy link
Member

@amatsuda amatsuda commented Jan 9, 2019

@zloyrusskiy See the actual commit by nobu 1680a13

matzbot pushed a commit that referenced this pull request Jan 12, 2019
	Fix Net::Protocol::BufferedIO#write when sending large multi-byte string

	This commit should fix Net::Protocol::BufferedIO#write when sending
	large multi-byte string like following example.

	```
	$ ruby -rnet/http -rjson -v -e "Net::HTTP.post(URI('http://httpbin.org/post'), { text: '?'*100_000 }.to_json, 'Content-Type' => 'application/json')"
	ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
	Traceback (most recent call last):
	        19: from -e:1:in `<main>'
	        18: from lib/ruby/2.6.0/net/http.rb:500:in `post'
	        17: from lib/ruby/2.6.0/net/http.rb:605:in `start'
	        16: from lib/ruby/2.6.0/net/http.rb:920:in `start'
	        15: from lib/ruby/2.6.0/net/http.rb:502:in `block in post'
	        14: from lib/ruby/2.6.0/net/http.rb:1281:in `post'
	        13: from lib/ruby/2.6.0/net/http.rb:1493:in `send_entity'
	        12: from lib/ruby/2.6.0/net/http.rb:1479:in `request'
	        11: from lib/ruby/2.6.0/net/http.rb:1506:in `transport_request'
	        10: from lib/ruby/2.6.0/net/http.rb:1506:in `catch'
	         9: from lib/ruby/2.6.0/net/http.rb:1507:in `block in transport_request'
	         8: from lib/ruby/2.6.0/net/http/generic_request.rb:123:in `exec'
	         7: from lib/ruby/2.6.0/net/http/generic_request.rb:189:in `send_request_with_body'
	         6: from lib/ruby/2.6.0/net/protocol.rb:247:in `write'
	         5: from lib/ruby/2.6.0/net/protocol.rb:265:in `writing'
	         4: from lib/ruby/2.6.0/net/protocol.rb:248:in `block in write'
	         3: from lib/ruby/2.6.0/net/protocol.rb:275:in `write0'
	         2: from lib/ruby/2.6.0/net/protocol.rb:275:in `each_with_index'
	         1: from lib/ruby/2.6.0/net/protocol.rb:275:in `each'
	lib/ruby/2.6.0/net/protocol.rb:280:in `block in write0': undefined method `bytesize' for nil:NilClass (NoMethodError)
	```

	[Fix GH-2058]

	From: Eito Katagiri <eitoball@gmail.com>


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_6@66799 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
@morgoth
Copy link

@morgoth morgoth commented Jan 21, 2019

In our case, while requesting some third party API, there was no error raised, but a valid http response returned (however it was a "bad request"), so it silently bitten us. Having 2.6.1 released with this fix would be much appreciated.

@ricardovj
Copy link

@ricardovj ricardovj commented Jan 23, 2019

This also silently caused a bunch of issues for us sending emails and it took us a few days to figure out it was related to the 2.6 release and we had to rollback. Releasing this will be much appreciated 🙏

@thefloweringash thefloweringash mentioned this pull request Jan 26, 2019
4 of 10 tasks
@oshanz
Copy link

@oshanz oshanz commented Jan 28, 2019

It's better to note this down this in the ruby 2.6.0 release page as a known issue

@ioquatix
Copy link
Member

@ioquatix ioquatix commented Jan 28, 2019

@oshanz can you make a PR against https://github.com/ruby/www.ruby-lang.org

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
9 participants