Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Keep query values data type when sorting #318

Merged
merged 13 commits into from

5 participants

tjsousa Bartosz Blimke Leonard Garvey Jure Triglav Trent Ogren
tjsousa

Query values can be an array when the :flat_array notation option is used on the query mapper, so this conversion to hash is removing repeated query params (which can exist).

This is a fix for bblimke/webmock#227

tjsousa tjsousa Keep query values data type when sorting
Query values can be an array when the :flat_array notation option is used on the query mapper, so this conversion to hash is removing repeated query params (which can exist).

This is a fix for bblimke/webmock#227
2bf5f5c
Bartosz Blimke
Owner

Thank you for this fix!

How can one set flat_array notation on query mapper if WebMock api doesn't offer this option?

In order to pull this request without any additional work on my side, I'd need unit test + acceptance test.
I can work on it on my own, when I have time, but not sure when that will be :)

tjsousa

Only after reading sporkmonger/addressable#77 did I understand the query mapper in use was borrowed from a previous version of Addressable (which provided the option), for backward compatibility reasons.

I now added a commit with WebMock configuration variable that you can set globally in your code, or when using allow_net_connect! and disable_net_connect! as an option.

I had to dig this rabbit hole but I'm not really into the details of WebMock, so is this the proper way of doing it?

Bartosz Blimke
Owner

Adding this option to allow_net_connect and disable_net_connect doesn't make sense, as it's not related to them in any way. I'd add a separate method to the api.

Bartosz Blimke
Owner

@tjsousa do you plan any more work on this pull request or is it complete?

tjsousa

It was meant to be for up review again with those last two commits passing on Travis CI.

At the moment, I don't have much time for additional work on this one. Do you think it is useful for now?

Leonard Garvey

This particular issue affects me with a 3rd party API I have to work with. The API accepts duplicate querystring parameters. I'd appreciate if this could be merged in and am happy to assist in any way if this is needed. Right now I'm pointing my Gemfile to the fixed version from @tjsousa

Jure Triglav

Spent an hour trying to figure out why the URL WebMock is reporting does not contain my query parameters.

This is the URL (note the multiple occurences of fq):

http://api.plos.org/search?fq=cross_published_journal_key%3APLoSCompBiol&q=everything%3Abiology&fq=doc_type:full&fq=!article_type_facet:%22Issue%20Image%22&fl=id,pmid,publication_date,received_date,accepted_date,title,cross_published_journal_name,author_display,editor_display,article_type,affiliate,subject,financial_disclosure&wt=json&facet=false&rows=25&hl=false

These are the parameters with CGI.parse (note the array for fq param):

CGI.parse(URI.parse(a).query)
=> {"fq"=>["cross_published_journal_key:PLoSCompBiol", "doc_type:full", "!article_type_facet:\"Issue Image\""],
 "q"=>["everything:biology"],
 "fl"=>["id,pmid,publication_date,received_date,accepted_date,title,cross_published_journal_name,author_display,editor_display,article_type,affiliate,subject,financial_disclosure"],
 "wt"=>["json"],
 "facet"=>["false"],
 "rows"=>["25"],
 "hl"=>["false"]}

And this is the URL that WebMock is reporting (note the single occurence of fq param):

WebMock::NetConnectNotAllowedError: Real HTTP connections are disabled. Unregistered request: GET http://api.plos.org/search?facet=false&fl=id,pmid,publication_date,received_date,accepted_date,title,cross_published_journal_name,author_display,editor_display,article_type,affiliate,subject,financial_disclosure&fq=!article_type_facet:%22Issue%20Image%22&hl=false&q=everything:biology&rows=25&wt=json with headers {'Accept'=>'*/*', 'Accept-Encoding'=>'gzip;q=1.0,deflate;q=0.6,identity;q=0.3', 'Host'=>'api.plos.org', 'User-Agent'=>'Ruby'}

What is the main reason this doesn't work out of the box and does this PR address it? Is it something that would possibly introduce a backwards incompatibility, perhaps for people who have stubbed their request based on what WebMock said, which is in fact an incomplete URL as their URL contains multiple parameters with the same name, and now these stubs would suddenly start failing?

Anyway, I'd be interested in doing any additional work that needs to be done to get this merged.

Jure Triglav jure referenced this pull request from a commit in articlemetrics/alm-report
Jure Triglav jure Add a spec for limiting results to specific journals with PLOS's API,…
… and then because of the way this works, i.e. using multiple instances of the fq parameter, going through and fixing all existing PLOS Solr specs/mocks, because WebMock does not support multiple parameters with the same name (bblimke/webmock#318 (comment)).
b9b4bd7
Bartosz Blimke bblimke merged commit 870464e into from
Bartosz Blimke
Owner

@jure "What is the main reason this doesn't work out of the box" - the default notation used to serialize/deserialize urls is subscript and it doesn't support duplicate parameters names.

The one that supports it :flat_array, is incompatible with :subscript

The changes are now merged. You can do the following to support duplicate keys in parameters.

WebMock::Config.instance.query_values_notation = :flat_array
Jure Triglav

Fantastic! Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Oct 24, 2013
  1. tjsousa

    Keep query values data type when sorting

    tjsousa authored
    Query values can be an array when the :flat_array notation option is used on the query mapper, so this conversion to hash is removing repeated query params (which can exist).
    
    This is a fix for bblimke/webmock#227
  2. tjsousa
  3. Trent Ogren tjsousa

    Failing unit test demonstrating WebMock ignoring params

    misfo authored tjsousa committed
    For params specified multiple times all but the last value is ignored.
    
    Issue #227
  4. tjsousa
  5. tjsousa
  6. tjsousa
  7. tjsousa
  8. tjsousa
  9. tjsousa
  10. tjsousa
  11. tjsousa
  12. tjsousa
  13. tjsousa
This page is out of date. Refresh to see the latest.
1  lib/webmock/config.rb
View
@@ -6,5 +6,6 @@ class Config
attr_accessor :allow_localhost
attr_accessor :allow
attr_accessor :net_http_connect_on_start
+ attr_accessor :query_values_notation
end
end
2  lib/webmock/http_lib_adapters/httpclient_adapter.rb
View
@@ -132,7 +132,7 @@ def build_webmock_response(httpclient_response)
def build_request_signature(req, reuse_existing = false)
uri = WebMock::Util::URI.heuristic_parse(req.header.request_uri.to_s)
- uri.query = WebMock::Util::QueryMapper.values_to_query(req.header.request_query) if req.header.request_query
+ uri.query = WebMock::Util::QueryMapper.values_to_query(req.header.request_query, :notation => WebMock::Config.instance.query_values_notation) if req.header.request_query
uri.port = req.header.request_uri.port
uri = uri.omit(:userinfo)
12 lib/webmock/request_pattern.rb
View
@@ -95,7 +95,7 @@ def add_query_params(query_params)
elsif rSpecHashIncludingMatcher?(query_params)
WebMock::Matchers::HashIncludingMatcher.from_rspec_matcher(query_params)
else
- WebMock::Util::QueryMapper.query_to_values(query_params)
+ WebMock::Util::QueryMapper.query_to_values(query_params, :notation => Config.instance.query_values_notation)
end
end
@@ -109,7 +109,7 @@ def to_s
class URIRegexpPattern < URIPattern
def matches?(uri)
WebMock::Util::URI.variations_of_uri_as_strings(uri).any? { |u| u.match(@pattern) } &&
- (@query_params.nil? || @query_params == WebMock::Util::QueryMapper.query_to_values(uri.query))
+ (@query_params.nil? || @query_params == WebMock::Util::QueryMapper.query_to_values(uri.query, :notation => Config.instance.query_values_notation))
end
def to_s
@@ -124,7 +124,7 @@ def matches?(uri)
if @pattern.is_a?(Addressable::URI)
if @query_params
uri.omit(:query) === @pattern &&
- (@query_params.nil? || @query_params == WebMock::Util::QueryMapper.query_to_values(uri.query))
+ (@query_params.nil? || @query_params == WebMock::Util::QueryMapper.query_to_values(uri.query, :notation => Config.instance.query_values_notation))
else
uri === @pattern
end
@@ -136,8 +136,8 @@ def matches?(uri)
def add_query_params(query_params)
super
if @query_params.is_a?(Hash) || @query_params.is_a?(String)
- query_hash = (WebMock::Util::QueryMapper.query_to_values(@pattern.query) || {}).merge(@query_params)
- @pattern.query = WebMock::Util::QueryMapper.values_to_query(query_hash)
+ query_hash = (WebMock::Util::QueryMapper.query_to_values(@pattern.query, :notation => Config.instance.query_values_notation) || {}).merge(@query_params)
+ @pattern.query = WebMock::Util::QueryMapper.values_to_query(query_hash, :notation => WebMock::Config.instance.query_values_notation)
@query_params = nil
end
end
@@ -201,7 +201,7 @@ def body_as_hash(body, content_type)
when :xml then
Crack::XML.parse(body)
else
- WebMock::Util::QueryMapper.query_to_values(body)
+ WebMock::Util::QueryMapper.query_to_values(body, :notation => Config.instance.query_values_notation)
end
end
2  lib/webmock/request_stub.rb
View
@@ -81,7 +81,7 @@ def self.from_request_signature(signature)
if signature.body.to_s != ''
body = if signature.url_encoded?
- WebMock::Util::QueryMapper.query_to_values(signature.body)
+ WebMock::Util::QueryMapper.query_to_values(signature.body, :notation => Config.instance.query_values_notation)
else
signature.body
end
24 lib/webmock/util/query_mapper.rb
View
@@ -38,9 +38,8 @@ class QueryMapper
# #=> [['one', 'two'], ['one', 'three']]
def self.query_to_values(query, options={})
query.force_encoding('utf-8') if query.respond_to?(:force_encoding)
- defaults = {:notation => :subscript}
- options = defaults.merge(options)
- if ![:flat, :dot, :subscript, :flat_array].include?(options[:notation])
+ notation = options[:notation] || :subscript
+ if ![:flat, :dot, :subscript, :flat_array].include?(notation)
raise ArgumentError,
"Invalid notation. Must be one of: " +
"[:flat, :dot, :subscript, :flat_array]."
@@ -60,7 +59,7 @@ def self.query_to_values(query, options={})
end
end
return nil if query == nil
- empty_accumulator = :flat_array == options[:notation] ? [] : {}
+ empty_accumulator = :flat_array == notation ? [] : {}
return ((query.split("&").map do |pair|
pair.split("=", 2) if pair && !pair.empty?
end).compact.inject(empty_accumulator.dup) do |accumulator, (key, value)|
@@ -70,18 +69,18 @@ def self.query_to_values(query, options={})
if value != true
value = Addressable::URI.unencode_component(value.gsub(/\+/, " "))
end
- if options[:notation] == :flat
+ if notation == :flat
if accumulator[key]
raise ArgumentError, "Key was repeated: #{key.inspect}"
end
accumulator[key] = value
- elsif options[:notation] == :flat_array
+ elsif notation == :flat_array
accumulator << [key, value]
else
- if options[:notation] == :dot
+ if notation == :dot
array_value = false
subkeys = key.split(".")
- elsif options[:notation] == :subscript
+ elsif notation == :subscript
array_value = !!(key =~ /\[\]$/)
subkeys = key.split(/[\[\]]+/)
end
@@ -103,7 +102,7 @@ def self.query_to_values(query, options={})
end
accumulator
end).inject(empty_accumulator.dup) do |accumulator, (key, value)|
- if options[:notation] == :flat_array
+ if notation == :flat_array
accumulator << [key, value]
else
accumulator[key] = value.kind_of?(Hash) ? dehash.call(value) : value
@@ -118,7 +117,8 @@ def self.query_to_values(query, options={})
# An empty Hash will result in a nil query.
#
# @param [Hash, #to_hash, Array] new_query_values The new query values.
- def self.values_to_query(new_query_values)
+ def self.values_to_query(new_query_values, options = {})
+ notation = options[:notation] || :subscript
if new_query_values == nil
return nil
@@ -159,14 +159,14 @@ def self.values_to_query(new_query_values)
value.sort!
buffer = ""
value.each do |key, val|
- new_parent = "#{parent}[#{key}]"
+ new_parent = notation != :flat_array ? "#{parent}[#{key}]" : parent
buffer << "#{to_query.call(new_parent, val)}&"
end
return buffer.chop
elsif value.is_a?(Array)
buffer = ""
value.each_with_index do |val, i|
- new_parent = "#{parent}[#{i}]"
+ new_parent = notation != :flat_array ? "#{parent}[#{i}]" : parent
buffer << "#{to_query.call(new_parent, val)}&"
end
return buffer.chop
7 lib/webmock/util/uri.rb
View
@@ -14,8 +14,8 @@ module CharacterClasses
NORMALIZED_URIS = Hash.new do |hash, uri|
normalized_uri = WebMock::Util::URI.heuristic_parse(uri)
if normalized_uri.query_values
- sorted_query_values = sort_query_values(WebMock::Util::QueryMapper.query_to_values(normalized_uri.query) || {})
- normalized_uri.query = WebMock::Util::QueryMapper.values_to_query(sorted_query_values)
+ sorted_query_values = sort_query_values(WebMock::Util::QueryMapper.query_to_values(normalized_uri.query, :notation => Config.instance.query_values_notation) || {})
+ normalized_uri.query = WebMock::Util::QueryMapper.values_to_query(sorted_query_values, :notation => WebMock::Config.instance.query_values_notation)
end
normalized_uri = normalized_uri.normalize #normalize! is slower
normalized_uri.query = normalized_uri.query.gsub("+", "%2B") if normalized_uri.query
@@ -74,7 +74,8 @@ def self.is_uri_localhost?(uri)
private
def self.sort_query_values(query_values)
- Hash[*query_values.sort.inject([]) { |values, pair| values + pair}]
+ sorted_query_values = query_values.sort
+ query_values.is_a?(Hash) ? Hash[*sorted_query_values.inject([]) { |values, pair| values + pair}] : sorted_query_values
end
def self.uris_with_inferred_port_and_without(uris)
2  spec/acceptance/httpclient/httpclient_spec_helper.rb
View
@@ -9,7 +9,7 @@ def http_request(method, uri, options = {}, &block)
c.ssl_config.verify_mode = OpenSSL::SSL::VERIFY_NONE
c.set_basic_auth(nil, uri.user, uri.password) if uri.user
params = [method, "#{uri.omit(:userinfo, :query).normalize.to_s}",
- WebMock::Util::QueryMapper.query_to_values(uri.query), options[:body], options[:headers] || {}]
+ WebMock::Util::QueryMapper.query_to_values(uri.query, :notation => WebMock::Config.instance.query_values_notation), options[:body], options[:headers] || {}]
if HTTPClientSpecHelper.async_mode
connection = c.request_async(*params)
connection.join
18 spec/acceptance/shared/request_expectations.rb
View
@@ -159,6 +159,24 @@
end
end
+ context "when using flat array notation" do
+ before :all do
+ WebMock::Config.instance.query_values_notation = :flat_array
+ end
+
+ it "should satisfy expectation if request includes different repeated query params in flat array notation" do
+ lambda {
+ stub_request(:get, "http://www.example.com/?a=1&a=2")
+ http_request(:get, "http://www.example.com/?a=1&a=2")
+ a_request(:get, "http://www.example.com/?a=1&a=2").should have_been_made
+ }.should_not raise_error
+ end
+
+ after :all do
+ WebMock::Config.instance.query_values_notation = nil
+ end
+ end
+
it "should fail if request was made more times than expected" do
lambda {
http_request(:get, "http://www.example.com/")
15 spec/unit/request_pattern_spec.rb
View
@@ -220,6 +220,21 @@ def match(request_signature)
:query => RSpec::Mocks::ArgumentMatchers::HashIncludingMatcher.new({"a" => ["b", "d"]})).
should_not match(WebMock::RequestSignature.new(:get, "www.example.com?a[]=b&a[]=c&b=1"))
end
+
+ context "when using query values notation as flat array" do
+ before :all do
+ WebMock::Config.instance.query_values_notation = :flat_array
+ end
+
+ it "should not match when repeated query params are not the same as declared as string" do
+ WebMock::RequestPattern.new(:get, "www.example.com", :query => "a=b&a=c").
+ should match(WebMock::RequestSignature.new(:get, "www.example.com?a=b&a=c"))
+ end
+
+ after :all do
+ WebMock::Config.instance.query_values_notation = nil
+ end
+ end
end
end
29 spec/unit/util/uri_spec.rb
View
@@ -197,6 +197,35 @@
uri = WebMock::Util::URI.normalize_uri(uri_string)
WebMock::Util::QueryMapper.query_to_values(uri.query).should == {"load"=>{"include"=>[{"staff"=>"email"},"business_name"]}}
end
+
+ context "when query notation is set to :flat_array" do
+ before :all do
+ WebMock::Config.instance.query_values_notation = :flat_array
+ end
+
+ it "should successfully handle repeated paramters" do
+ uri_string = "http://www.example.com:80/path?target=host1&target=host2"
+ uri = WebMock::Util::URI.normalize_uri(uri_string)
+ WebMock::Util::QueryMapper.query_to_values(uri.query, :notation => WebMock::Config.instance.query_values_notation).should == [['target', 'host1'], ['target', 'host2']]
+ end
+ end
+ end
+
+ describe "sorting query values" do
+
+ context "when query values is a Hash" do
+ it "returns an alphabetically sorted hash" do
+ sorted_query = WebMock::Util::URI.sort_query_values({"b"=>"one", "a"=>"two"})
+ sorted_query.should == {"a"=>"two", "b"=>"one"}
+ end
+ end
+
+ context "when query values is an Array" do
+ it "returns an alphabetically sorted array" do
+ sorted_query = WebMock::Util::URI.sort_query_values([["b","two"],["a","one_b"],["a","one_a"]])
+ sorted_query.should == [["a","one_a"],["a","one_b"],["b","two"]]
+ end
+ end
end
describe "stripping default port" do
Something went wrong with that request. Please try again.