Change nested query parsing behavior of "foo[]" from {"foo" => [nil]} to {"foo" => []}

[PragTob]

Hello lovely folks,

first and foremost thanks for your excellent work on this project practically the entire ruby webdev eco system relies on 💚 🎉 🚀

Problem

I've come here today to propose a change in the parsing behavior of the query string param[].

There is a spec asserting the current behavior so I assume it's intentional

rack/test/spec_utils.rb

Lines 164 to 165 in 649c72b

	Rack::Utils.parse_nested_query("foo[]").
	must_equal "foo" => [nil]

I believe/hope the more correct test would be:

    Rack::Utils.parse_nested_query("foo[]").
      must_equal "foo" => []

I base this on three things:

• we otherwise have no way to indicate an empty array and an array with nil doesn't seem particularly useful (or intuitive) vs. an empty array
• the introduction was 12 years ago ( 6e330bd ) and might be a side effect of the implementation due to the splitting behavior exhibited here:

  rack/lib/rack/query_parser.rb

  Lines 67 to 72 in 649c72b

  	unless qs.nil? || qs.empty?
  	(qs || '').split(d ? (COMMON_SEP[d] || /[#{d}] */n) : DEFAULT_SEP).each do |p|
  	k, v = p.split('=', 2).map! { |s| unescape(s) }
  	normalize_params(params, k, v, param_depth_limit)
  	end

• the popular HTTP client library faraday (which her and many projects rely upon) changed it's way of serializing empty arrays to the query string above #800 correct handling parameter value empty array lostisland/faraday#801

Especially due to the last point this creates a sort of dissonance in the Ruby eco system where I send an empty array from the one side and it arrives as [nil] on the server side. Of course, the fix might also be on the faraday side.

I have not yet checked the HTTP specs or anything about this to see what's "officially correct"

To have complete traceability, I came here via ruby-grape/grape#2079 :)

Implementation

I'm not sure if it's the complete implementation but it seems to be easily fixable in

rack/lib/rack/query_parser.rb

Lines 102 to 105 in 649c72b

	elsif after == "[]"
	params[k] ||= []
	raise ParameterTypeError, "expected Array (got #{params[k].class.name}) for param `#{k}'" unless params[k].is_a?(Array)
	params[k] << v

all it needs is unless v.nil?:

      elsif after == "[]"
        params[k] ||= []
        raise ParameterTypeError, "expected Array (got #{params[k].class.name}) for param `#{k}'" unless params[k].is_a?(Array)
        params[k] << v unless v.nil?

All tests (with exception of the one mentioned above) pass with this change.

(I have the code, I found the old spec while implementing this/my spec was elsewhere)

Happy to make a PR should it be wanted!

Thanks for all your work! 🎉 🚀

[jeremyevans]

Can you get a browser to submit that via a form submission? As far as I'm aware, browsers will always include an equal sign, even if the right hand side would be empty. Looking at RFC 1866, it appears that a query string of foo[] is invalid. A valid query string would be foo[]=, which Rack will parse as {"foo"=>[""]}. If we change Rack's behavior in any way in this area, it should be to raise an exception for foo[], or other cases where there is no equal sign. That's my opinion, let's see what other committers think.

[PragTob]

Hi @jeremyevans 👋

Not sure you can get a form to submit like this, I haven't tried though. I looked at RFC3986 and it's... interesting.

So looking at the ABNF for URIs it seems valid that key can go without value:

   URI           = scheme ":" hier-part [ "?" query ] [ "#" fragment ]

   query         = *( pchar / "/" / "?" )
   pchar         = unreserved / pct-encoded / sub-delims / ":" / "@"

   unreserved    = ALPHA / DIGIT / "-" / "." / "_" / "~"
   pct-encoded   = "%" HEXDIG HEXDIG
   sub-delims    = "!" / "$" / "&" / "'" / "(" / ")"
                 / "*" / "+" / "," / ";" / "="

However, [ and ] seem invalid 🤷 (hope I didn't miss it) That's clearly not our lived reality though.

edit: Jeremy corrected me that of course they are escaped and hence valid

The RFC also says it's frequently used as key value, but not always:

However, as query components
are often used to carry identifying information in the form of
"key=value" pairs and one frequently used value is a reference to
another URI, it is sometimes better for usability to avoid percent-
encoding those characters.

Seems to me that this way [] is used is more of a thing of how web frameworks frequently use it more than an actual standard. Or I haven't found the standard yet, or it's just a recommendation...

[jeremyevans]

From a URL perspective, the query portion is opened ended. That's because different URL schemas could have different query formats. However, for HTTP, RFC 1866 applies, as the query for HTTP should be in application/x-www-form-urlencoded format. See https://tools.ietf.org/html/rfc1866#section-8.2.

Rack's specific handling of [] is not part of any standard. However, that doesn't change that foo[] (without the =) appears to not be valid format for application/x-www-form-urlencoded.

[jeremyevans]

Note that [] in the query part is supposed to be escaped in order to be valid according to the newer URL RFC (3986). So when submitted, the query should be submitted as: foo%5b%5d=. Rack will decode that to foo[]= before attempting to parse it. Rack and most webservers will accept unescaped [] in query, even though it isn't strictly valid. ruby-core even rejected a bug report to URI escape [] (https://bugs.ruby-lang.org/issues/9990), as using the unescaped version is not an issue in practice.

[ioquatix]

The problem with escaping [] is that it's pedantic and makes URLS super ugly. So I agree with Rails. The only thing I could add is that if foo[] is currently invalid, making it do the behaviour as outlined {foo: []} shouldn't be a breaking change.

So the only question is what is the use case and how did you create such a query string?

[ioquatix]

I'm okay with this change generally, but can you make a PR so we can review?

[jeremyevans]

@ioquatix can you describe briefly why are in favor of this change, considering that foo[] is not a valid query string? IMO, this is a bug introduced in faraday, and attempting to support it just encourages the bug to spread to more libraries.

[ioquatix]

• It shouldn't break existing code if it was previously undefined. However, now that I think about it, code could potentially depend on the currently unspecified behaviour.
• Query parsing is poorly defined so I'm not at odds with being more robust and having sensible interpretations of otherwise ambiguous strings.
• Throwing an exception could be a worse option.

Regarding this particular behaviour, if it's a bug in faraday, I'm less inclined to change it. But I don't think it's an unreasonable request. So, I want to see an actual PR so we can see what specs need to be changed and the potential impact. This is on the OP to deliver.

Regarding consistency, PHP behaves similar as Rack as implemented:

php > parse_str("foo[]", $result);
php > var_dump($result);
array(1) {
  ["foo"]=>
  array(1) {
    [0]=>
    string(0) ""
  }
}

The difference is PHP sets the variable to an empty string, so a lack of = is the same as =.

@PragTob can you survey other implementations to see what is the normal behaviour? Can you also comment on whether this is a bug with Faraday?

cc @olleolleolle

[jeremyevans]

I'm not totally opposed to foo[] being the same as foo[]=. The fact that it isn't currently the same is an implementation detail in my mind. I still think a warning in the next version followed by an exception in future versions is the better way to handle this.

As far as I can tell from the faraday issue, they chose this path because it appeared to work with Rails. I'm not sure whether the Rails behavior is intentional or just an implementation detail in Rails.

There's a big problem with accepting invalid input. Invalid input becomes cancerous. If a server accepts invalid input, clients will send invalid input considering it valid, and then try to use the invalid input with other servers, and complain to other servers that don't accept it. That's basically what is happening here.

See https://tools.ietf.org/id/draft-thomson-postel-was-wrong-03.html and https://news.ycombinator.com/item?id=9824638.

[ioquatix]

I agree that we shouldn't accept invalid input. But the RFC is not clear that this is invalid input? If we can confirm it's invalid according to the RFC, I would follow that.

Does the query parser throw exceptions for any other kind of input?

[jeremyevans]

@ioquatix https://tools.ietf.org/html/rfc1866#section-8.2 describes the application/x-www-form-urlencoded format. Section 8.2.1.2 states: The fields are listed in the order they appear in the document with the name separated from the value by '='. It does not appear to support key/value entries without an = separating the key and value. If a value is null (there is no value), then the whole pair should be omitted.

Yes, the parser already raises exceptions in some cases:

ruby -r rack -e 'p Rack::Utils.parse_nested_query("foo[bar]=&foo[]=")'
lib/rack/query_parser.rb:104:in `normalize_params': expected Array (got Rack::QueryParser::Params) for param `foo' (Rack::QueryParser::ParameterTypeError)

[jeremyevans]

It does appear the Rails behavior is by design: https://github.com/rails/rails/blob/master/actionpack/test/dispatch/request/query_string_parsing_test.rb#L96-L103

Fun fact, the stripping of nils from parameter arrays was introduced in Rails as a security fix for a bug in ActiveRecord: rails/rails@060c91c

This was later changed to just remove nils from arrays, leaving the array empty, and only because ActiveRecord had been fixed to handle things correctly: rails/rails@8f8ccb9

[PragTob]

👋

Hey thanks for all the input and thanks Jeremy for the corrections to my erroneous view of the RFC 💚

I can look at some implementations in other languages and see what they do sure :)

I can provide a PR, I already had/have the code - it only changes 2 lines and has everything passing so it's easy.

As for Faraday, it was a bugfix for lostisland/faraday#800 which itself was basically a bug fix for a json_api_client issue: JsonApiClient/json_api_client#278 - so it was definitely an intended bugfix. I'll check the JSON API spec to see if I can quickly find if they consider that valid or if it was just a wrong interpretation.

as an aside, I'd like it if there was a way to represent an empty array vs. array with an empty string. Standard conformity of course trumps how I feel though 😅

Also, thanks for the quick feedback everyone! 🚀