HTTP basic auth convert response body to string is hanging #152
Comments
|
I find it stucks in this while loop: https://github.com/tarcieri/http/blob/master/lib/http/response/body.rb#L40-L42 |
|
What http version is it? There was a bug with chunked responses (fixed in master and in 0.6.2). PS You don't need to get body explicitly, just call |
|
Thanks for reply. Sorry I was using 0.6.1. Update to 0.6.2 still hangs (already change to while (chunk = @client.readpartial)
@contents << chunk
end
No one calls def on_message_complete
@finished = true
end |
|
UPDATE: Actually I just need raw data that is consumed from the socket as is. |
|
@ixti Thanks! Did not know the http_parser, cool!!! I have put the data in this gist. https://gist.github.com/JuanitoFatas/b21a7f8e7c8523deb3b1 Thanks!!! |
|
Sorry I was a bit unclear :D I wanted to see the full data stream sent to socket. In other words somthing like this: require "http/response/parser"
module HTTP
class Response
class Parser
def add(data)
puts data.inspect
@parser << data
end
alias_method :<<, :add
end
end
endI'll try to do that myself once I'll get some time (need to get some kind of trial access to that server). |
|
Hi, My sample code : url = "http://188.165.196.212:8000/index.html"
user_agent = "Mozilla/5.0 (iPad; CPU OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) CriOS/30.0.1599.12 Mobile/11A465 Safari/8536.25"
accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
headers = {
"User-Agent" => user_agent,
accept:accept,
"DNT" => 1,
"Cache-Control" => "max-age=0",
"Accept-Encoding" => "gzip,deflate,sdch",
"Accept-Language"=> "en-US,en;q=0.8,fr;q=0.6",
}
r = HTTP.with(headers).get(url)
r.to_s # Hang and ruby process consums 100% of CPUWith the indication you give : here the sockets data : Nothing written after this. In Body.to_s chunk always == "" module HTTP
class Response
class Body
# Eagerly consume the entire body as a string
def to_s
return @contents if @contents
fail StateError, 'body is being streamed' unless @streaming.nil?
begin
@streaming = false
@contents = ''
binding.pry
while (chunk = @client.readpartial)
# @client.readpartial always return ""
@contents << chunk
end
rescue
@contents = nil
raise
end
@contents
end
end
end
end |
|
Now that's something interesting. require "yaml"
puts YAML.dump response.headers.to_h |
|
It's seem to me that http_parser never call on_message_complete.But http_parser core is in C. I'm not able to try to debug this to understand. Just a very quick and dirty monkey patch that work for me. module HTTP
class Response
class Parser
def chunk
chunk, @chunk = @chunk, nil
@finished = true if chunk.nil?
chunk
end
end
end
end |
|
Ugh. I think we should probably dig into http_parser and see what's up |
|
Yeah, I suspect issue is that content is neither chunked nor headers tells content length. |
|
@ixti The headers I get: 😀 |
|
As I said before, issue is exactly when HTTP Gem (and underlying http_parser) can't determine that response finished. Here's a simple example of a server that makes HTTP Gem run into infinitive loop: require "socket"
server = TCPServer.new 8080
loop do
Thread.start server.accept do |client|
client << "HTTP/1.1 200 OK\r\n"
client << "Content-Type: text/plain\r\n"
client << "\r\n"
client << "Hello!"
end
endSo, running HTTP.get("http://localhost:8080/").headers
# => #<HTTP::Headers {"Content-Type"=>"text/plain"}>
HTTP.get("http://localhost:8080/").to_s
# freeze...I don't think that HTTP Gem should handle this situation by default. For example Firefox is getting into infinitive loop as well. But we can have some kind of optional no more data timeout. Which will look something like this (just an idea, not a real code): def readpartial(...)
chunk = @socket.read(...)
if chunk
@no_data_at = nil
return chunk
end
now = Time.now.to_i
@no_data_at ||= now
fail_at = @no_data_at + @options[:wait_for_data].to_i
fail NoDataTimeot if fail_at < now
nil
endAgain this should not be handled by default at all. Only if explicitly specified in options. And it should be configurable, what to do on begin
puts HTTP.response_timeout(:fail_after => 10).get("http://defective-server.tld").to_s
rescue HTTP::NoDataTimeout => e
puts "Shit happens. Can't wait any longer... What we have so far is:"
puts e.response.to_s
end |
|
Content-Length being absent isn't a terribly unusual condition. The correct handling is typically to terminate the body when the connection is closed. That's what Oliviergg's monkey patch does. |
|
Unfortunately the |
|
The problem is that (!) this bug happens when connection is not closed by server. Like in the sample I have shared before. If you will change that broken server code to have |
|
Will verify this weekend, thank you @ixti !!! ❤️ |
|
@JuanitoFatas Any update on this? |
Just verified, now works like a charm. Thanks for the reminder. |
|
👍 |
Hello!
I am doing a basic auth, when I get the response body back. And convert to string. It hangs.
HTTP.auth(:basic, authenticate_params).get(FEED_URL)returns
And
HTTP.auth(:basic, authenticate_params).get(FEED_URL).bodyreturns
And
HTTP.auth(:basic, authenticate_params).get(FEED_URL).body.to_sHangs at
to_s.Any advices? Thank you!
The relevant code, please see here.
The text was updated successfully, but these errors were encountered: