Hello!
Thank you for all your work on Async and Falcon. We're in the process of migrating our largest Rails app from Puma to Falcon and the performance improvement is very impressive.
After deploying to production, I noticed a lot of dropped requests. According to the nginx error logs, Falcon closed the connection before sending a response. After a lot of debugging, I discovered that the problem seems to be that Protocol::HTTP doesn't allow multiple user agent headers. (From what I can see, it looks like it's mainly the Googlebot that sends the header multiple times.)
curl http://localhost:3000/en -H "User-Agent: foobar/2000" -H "User-Agent: foobar/2001"
curl: (52) Empty reply from server
Protocol::HTTP::DuplicateHeaderError: Duplicate singleton header key: "user-agent"
| Existing value: "foobar/2000"
| New value: "foobar/2001"This seems pretty harsh. Do you know what the reason for that is?
|
if hash.key?(key) |
|
raise DuplicateHeaderError.new(key, hash[key], value) |
|
end |
Puma (and nginx, as far as I can tell) handle this by concatenating the headers together, e.g. foobar/2000, foobar/2001.
Additionally, the header value that actually gets sent to Rack is foobar/2000,foobar/2001 anyway, because it gets concatenated in Protocol::Rack::Adapter::Generic.unwrap_headers.
I "fixed" this for our app by adding an initializer, but I was wondering if this problem wouldn't hit all users of Falcon that get Googlebot traffic.
# config/initializers/falcon.rb
Protocol::HTTP::Headers::POLICY["user-agent"] = Protocol::HTTP::Header::Split
Hello!
Thank you for all your work on Async and Falcon. We're in the process of migrating our largest Rails app from Puma to Falcon and the performance improvement is very impressive.
After deploying to production, I noticed a lot of dropped requests. According to the nginx error logs, Falcon closed the connection before sending a response. After a lot of debugging, I discovered that the problem seems to be that Protocol::HTTP doesn't allow multiple user agent headers. (From what I can see, it looks like it's mainly the Googlebot that sends the header multiple times.)
This seems pretty harsh. Do you know what the reason for that is?
protocol-http/lib/protocol/http/headers.rb
Line 348 in a2f2d1c
protocol-http/lib/protocol/http/headers.rb
Lines 455 to 457 in a2f2d1c
Puma (and nginx, as far as I can tell) handle this by concatenating the headers together, e.g.
foobar/2000, foobar/2001.Additionally, the header value that actually gets sent to Rack is
foobar/2000,foobar/2001anyway, because it gets concatenated inProtocol::Rack::Adapter::Generic.unwrap_headers.I "fixed" this for our app by adding an initializer, but I was wondering if this problem wouldn't hit all users of Falcon that get Googlebot traffic.