New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Net::Protocol::BufferedIO#write when sending large multi-byte string #2058

Closed
wants to merge 1 commit into
base: trunk
from

Conversation

6 participants
@eitoball
Copy link

eitoball commented Dec 27, 2018

This commit should fix Net::Protocol::BufferedIO#write when sending large multi-byte string like following example.

$ ruby -rnet/http -rjson -v -e "Net::HTTP.post(URI('http://httpbin.org/post'), { text: 'あ'*100_000 }.to_json, 'Content-Type' => 'application/json')"
ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
Traceback (most recent call last):
        19: from -e:1:in `<main>'
        18: from lib/ruby/2.6.0/net/http.rb:500:in `post'
        17: from lib/ruby/2.6.0/net/http.rb:605:in `start'
        16: from lib/ruby/2.6.0/net/http.rb:920:in `start'
        15: from lib/ruby/2.6.0/net/http.rb:502:in `block in post'
        14: from lib/ruby/2.6.0/net/http.rb:1281:in `post'
        13: from lib/ruby/2.6.0/net/http.rb:1493:in `send_entity'
        12: from lib/ruby/2.6.0/net/http.rb:1479:in `request'
        11: from lib/ruby/2.6.0/net/http.rb:1506:in `transport_request'
        10: from lib/ruby/2.6.0/net/http.rb:1506:in `catch'
         9: from lib/ruby/2.6.0/net/http.rb:1507:in `block in transport_request'
         8: from lib/ruby/2.6.0/net/http/generic_request.rb:123:in `exec'
         7: from lib/ruby/2.6.0/net/http/generic_request.rb:189:in `send_request_with_body'
         6: from lib/ruby/2.6.0/net/protocol.rb:247:in `write'
         5: from lib/ruby/2.6.0/net/protocol.rb:265:in `writing'
         4: from lib/ruby/2.6.0/net/protocol.rb:248:in `block in write'
         3: from lib/ruby/2.6.0/net/protocol.rb:275:in `write0'
         2: from lib/ruby/2.6.0/net/protocol.rb:275:in `each_with_index'
         1: from lib/ruby/2.6.0/net/protocol.rb:275:in `each'
lib/ruby/2.6.0/net/protocol.rb:280:in `block in write0': undefined method `bytesize' for nil:NilClass (NoMethodError)
Fix Net::Protocol::BufferedIO#write when sending large multi-byte string
This commit should fix Net::Protocol::BufferedIO#write when sending
large multi-byte string like following example.

```
$ uby -rnet/http -rjson -v -e "Net::HTTP.post(URI('http://httpbin.org/post'), { text: 'あ'*100_000 }.to_json, 'Content-Type' => 'application/json')"
ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
Traceback (most recent call last):
        19: from -e:1:in `<main>'
        18: from lib/ruby/2.6.0/net/http.rb:500:in `post'
        17: from lib/ruby/2.6.0/net/http.rb:605:in `start'
        16: from lib/ruby/2.6.0/net/http.rb:920:in `start'
        15: from lib/ruby/2.6.0/net/http.rb:502:in `block in post'
        14: from lib/ruby/2.6.0/net/http.rb:1281:in `post'
        13: from lib/ruby/2.6.0/net/http.rb:1493:in `send_entity'
        12: from lib/ruby/2.6.0/net/http.rb:1479:in `request'
        11: from lib/ruby/2.6.0/net/http.rb:1506:in `transport_request'
        10: from lib/ruby/2.6.0/net/http.rb:1506:in `catch'
         9: from lib/ruby/2.6.0/net/http.rb:1507:in `block in transport_request'
         8: from lib/ruby/2.6.0/net/http/generic_request.rb:123:in `exec'
         7: from lib/ruby/2.6.0/net/http/generic_request.rb:189:in `send_request_with_body'
         6: from lib/ruby/2.6.0/net/protocol.rb:247:in `write'
         5: from lib/ruby/2.6.0/net/protocol.rb:265:in `writing'
         4: from lib/ruby/2.6.0/net/protocol.rb:248:in `block in write'
         3: from lib/ruby/2.6.0/net/protocol.rb:275:in `write0'
         2: from lib/ruby/2.6.0/net/protocol.rb:275:in `each_with_index'
         1: from lib/ruby/2.6.0/net/protocol.rb:275:in `each'
lib/ruby/2.6.0/net/protocol.rb:280:in `block in write0': undefined method `bytesize' for nil:NilClass (NoMethodError)
```

@matzbot matzbot closed this in 1680a13 Dec 27, 2018

@ioquatix

This comment has been minimized.

Copy link
Member

ioquatix commented Dec 27, 2018

Ouch! Nice catch.

@bjeanes

This comment has been minimized.

Copy link

bjeanes commented Jan 8, 2019

I'm not sure if the same issue, but I am having request body corruption when sending large multi-byte bodies, but no apparent exception.

This script reproduces it for me with appropriate body:

# frozen_string_literal: true

require "net/http"
require "net/https"
require "uri"
require "json"

body = File.read("./test-data.txt")
uri = URI.parse "https://httpbin.org/anything"

Net::HTTP.start(uri.host, /^https/.match?(uri.scheme) ? 443 : 80, use_ssl: /^https/.match?(uri.scheme)) do |http|
  req = Net::HTTP::Post.new(uri.request_uri)
  req["Content-Type"] = "text/plain"
  req.body = body

  res = http.request(req)

  p JSON.parse(res.body)["data"] == body
end

This returns false but if I change the above URL to http:// it will return true.

I can't share the request body I'm using as it has sensitive data but if your own test body from this issue doesn't reproduce, I can try to reduce my sample to something shareable.

Is this likely a different symptom of the same issue or a different issue altogether?

@bjeanes

This comment has been minimized.

Copy link

bjeanes commented Jan 8, 2019

I just compiled 2.6 with this patch and it does fix the issue.

FWIW, this seems like a pretty major one. This brought down the entirety of all ElasticSearch bulk indexing for our site because the NDJSON bodies were corrupted to the point of being invalid JSON. Worse, some bodies were still valid JSON but the (large) fields had malformed content (i.e. corrupted semantics without corrupting syntax, though this was rarer).

EDIT: I'd like to leave a comment on https://bugs.ruby-lang.org/issues/15468 but I don't seem to see how to do that. I'll open a separate issue for this symptom if that is helpful.

@adamof adamof referenced this pull request Jan 8, 2019

Closed

Ruby 2.6 body size issue #842

@adamof

This comment has been minimized.

Copy link

adamof commented Jan 8, 2019

We are seeing this when we use Net:HTTP via Faraday and sending big requests.
lostisland/faraday#842

@zloyrusskiy

This comment has been minimized.

Copy link

zloyrusskiy commented Jan 9, 2019

I think that it is necessary to add a test so that the regression does not repeat in the future

@amatsuda

This comment has been minimized.

Copy link
Member

amatsuda commented Jan 9, 2019

@zloyrusskiy See the actual commit by nobu 1680a13

matzbot pushed a commit that referenced this pull request Jan 12, 2019

merge revision(s) 66582: [Backport #15468][Backport #15472]
	Fix Net::Protocol::BufferedIO#write when sending large multi-byte string

	This commit should fix Net::Protocol::BufferedIO#write when sending
	large multi-byte string like following example.

	```
	$ ruby -rnet/http -rjson -v -e "Net::HTTP.post(URI('http://httpbin.org/post'), { text: '?'*100_000 }.to_json, 'Content-Type' => 'application/json')"
	ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
	Traceback (most recent call last):
	        19: from -e:1:in `<main>'
	        18: from lib/ruby/2.6.0/net/http.rb:500:in `post'
	        17: from lib/ruby/2.6.0/net/http.rb:605:in `start'
	        16: from lib/ruby/2.6.0/net/http.rb:920:in `start'
	        15: from lib/ruby/2.6.0/net/http.rb:502:in `block in post'
	        14: from lib/ruby/2.6.0/net/http.rb:1281:in `post'
	        13: from lib/ruby/2.6.0/net/http.rb:1493:in `send_entity'
	        12: from lib/ruby/2.6.0/net/http.rb:1479:in `request'
	        11: from lib/ruby/2.6.0/net/http.rb:1506:in `transport_request'
	        10: from lib/ruby/2.6.0/net/http.rb:1506:in `catch'
	         9: from lib/ruby/2.6.0/net/http.rb:1507:in `block in transport_request'
	         8: from lib/ruby/2.6.0/net/http/generic_request.rb:123:in `exec'
	         7: from lib/ruby/2.6.0/net/http/generic_request.rb:189:in `send_request_with_body'
	         6: from lib/ruby/2.6.0/net/protocol.rb:247:in `write'
	         5: from lib/ruby/2.6.0/net/protocol.rb:265:in `writing'
	         4: from lib/ruby/2.6.0/net/protocol.rb:248:in `block in write'
	         3: from lib/ruby/2.6.0/net/protocol.rb:275:in `write0'
	         2: from lib/ruby/2.6.0/net/protocol.rb:275:in `each_with_index'
	         1: from lib/ruby/2.6.0/net/protocol.rb:275:in `each'
	lib/ruby/2.6.0/net/protocol.rb:280:in `block in write0': undefined method `bytesize' for nil:NilClass (NoMethodError)
	```

	[Fix GH-2058]

	From: Eito Katagiri <eitoball@gmail.com>


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_6@66799 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment