Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Correct handling of "/:name.?:format?" and "/:user@?:host?" #492

Merged
merged 16 commits into from

4 participants

@floere

This pull request adds the correct handling of the following URL path patterns to Sinatra. These are the patterns which this is about:

 Pattern             | Old Regexp                                               | Example                          | Should Be
 "/:name.?:format?"  | /^\/([^\/?#]+)(?:\.|%2E)?([^\/?#]+)?$/                   | "/foo.bar"                       | ["foo", "bar"]
 "/:name.?:format?"  | /^\/([^\/?#]+)(?:\.|%2E)?([^\/?#]+)?$/                   | "/foo%2Ebar"                     | ["foo", "bar"]                  
 "/:user@?:host?"    | /^\/([^\/?#]+)(?:@|%40)?([^\/?#]+)?$/                    | "/foo@bar"                       | ["foo", "bar"]                    
 "/:user@?:host?"    | /^\/([^\/?#]+)(?:@|%40)?([^\/?#]+)?$/                    | "/foo.foo@bar"                   | ["foo.foo", "bar"]                
 "/:user@?:host?"    | /^\/([^\/?#]+)(?:@|%40)?([^\/?#]+)?$/                    | "/foo@bar.bar"                   | ["foo", "bar.bar"]

Why? / History

This is the result of a long discussion starting here:
https://twitter.com/#!/konstantinhaase/status/182480862326165504

The discussion being here:
https://gist.github.com/2154980

What? / Code changes

I mainly touched the base.rb file, specifically the Base#compile method. My plan was to generalize some of the other cases to keep the needed changes to the code low. This plan resulted in 2 other patterns -> regexps being changed (and imho improved).

I added a quite comprehensive spec in the file compile_test.rb, which can be run with ruby test/compile_test.rb.

Possible discussion worthy points

Using /^...$/ instead of /\A...\z/ regexps

One bigger change is found in switching from /^...$/ to /\A...\z/, i.e. switching from line-based matching to global matching with anchoring at the beginning of the string. Two reasons:

  1. I believe it to be "more correct".
  2. It is thought to be more performant (for some cases, see http://www.ruby-doc.org/core-1.9.3/Regexp.html).

If this is a problem, please discuss.

Pattern "/:name.?:format?" question

I was wondering about one pattern (see header) – on an example of "/.bar" is supposed to result in [".bar", nil]. This seems strange to me – it seems to me as if it should result in a non-match, i.e. nil.
I changed the tests to reflect my line of thought. If this is untrue, please tell me why and I'll change the tests and add the commit to this pull request.

Thanks for reading! If there are any cleanups to be done, please let me know.

@rkh rkh commented on the diff
lib/sinatra/base.rb
@@ -1302,17 +1302,21 @@ def compile!(verb, path, block, options = {})
def compile(path)
keys = []
if path.respond_to? :to_str
- pattern = path.to_str.gsub(/[^\?\%\\\/\:\*\w]/) { |c| encoded(c) }
+ ignore = ""
+ pattern = path.to_str.gsub(/[^\?\%\\\/\:\*\w]/) do |c|
+ ignore << escaped(c).join if c.match(/[\.@]/)
@rkh Owner
rkh added a note

I think we might also want this for other symbols, like :, so maybe remove the if or something?

@floere
floere added a note

Sure, I can do that. Do you have more example patterns -> regexps?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@rkh
Owner

With this patch, does "/:foo" still parse "/foo.bar"?

@floere

(That is my answer :) )

@floere

As a note, the tests basically contain the list here: https://gist.github.com/2154980#gistcomment-168954 (with some additions)

The case "/:foo" parses "/foo.bar" wasn't in there which is why I added it.

@floere

How are we going to proceed with this? Are there any stopping points from merging? If yes, what are they?

@rkh
Owner

Sorry, it's mainly a time management thing, I'd like to go over the code, again, refactor the tests, play around with it, run it in some real apps, and such. I want this to be part of the 1.4.0 release, as it is a behavioral change, but I'm also planning to have a 1.3.3 release first.

@ests

+1, this is the functionality I would love to see merged too. Consuming API webservice I've written in Sinatra, by using AcitveResource was a bit of pain, because by default AR generates request URIs like "/user/1.json" and ect.

@rkh
Owner

This will be merged in for 1.4.0. I will take care of this during or after RailsConf.

@floere

@rkh Thanks for the info. If you need changes (i.e. how do the test need to be refactored?), don't hesitate to tell me.

@rkh
Owner

I don't like the giant_array.each { ... } approach and would rather have some minimal DSL there.

@floere

I'm surprised. What's the reasoning? Just personal style?

@rkh
Owner

Readability, easier to grab what's going on for someone new joining the project. You actually like those arrays nested in arrays nested in arrays with a bunch of code right below it?

@floere

Let's move this in a constructive direction.

DSL example:

pattern_converts_into "/:name.?:format?", /^\/([^\/?#]+)(?:\.|%2E)?([^\/?#]+)?$/
pattern_resolves "/:name.?:format?", "/foo.bar", ["foo", "bar"]

Let's go from here.

@rkh
Owner

That's better, but I was thinking of maybe actually generating routes and testing if a requests goes through and what params looks like. That's the only way to actually make sure it works.

Something like this:

parses_pattern("/:name.?:format?", "/foo.bar", "name" => "foo", "format" => "bar")
does_not_parse("/:name.:format", "/")

Not sure about the method names.

@rkh rkh closed this
@rkh rkh reopened this
@rkh rkh closed this
@rkh rkh reopened this
@rkh rkh closed this
@rkh rkh reopened this
@floere

Undecided? ;)

@rkh rkh closed this
@rkh rkh reopened this
@rkh rkh closed this
@rkh rkh reopened this
@rkh rkh closed this
@rkh rkh reopened this
@rkh rkh closed this
@rkh rkh reopened this
@rkh rkh closed this
@rkh rkh reopened this
@rkh
Owner

Ah, sorry, this was not related to the Pull Request. We tried to figure out why github is not sending out pubsub notifications for some repos/pull requests.

@floere

No worries.

@travisbot

This pull request fails (merged dfa8409 into b882ab3).

@floere

Ok. This pull request seems to be 1.9.2 only. Shall I have a look?

@rkh
Owner

That'd be great. Ignore the Puma failure (fixed in master).

@travisbot

This pull request fails (merged 540b171 into b882ab3).

@floere

@travisbot Not sure who fails here ;)

Error: #<NativeException: org.virtualbox_4_1.VBoxException:
The function "powerDown" returned an error condition:
"The virtual machine is being powered down"  (0x80bb0002)>
@travisbot

This pull request fails (merged be59b2b into b882ab3).

@travisbot

This pull request passes (merged 9f08499 into b882ab3).

@travisbot

This pull request fails (merged 98ef21b into b882ab3).

@travisbot

This pull request fails (merged 57fc08d into b882ab3).

@floere

@travisbot Well, it "fails": SIGSEGV (0xb) at pc=0xb19c633b, pid=2081, tid=3078609776

Anyway, @rkh, I believe we're good to go afaics.

@travisbot

This pull request passes (merged 57babc5 into b882ab3).

@travisbot

This pull request fails (merged 6462a00 into b882ab3).

@ests

Having notifications turned on is no longer nice, got spammed by Mr. Travisbot ;)

@travisbot

This pull request fails (merged de21260 into b882ab3).

@floere

@ests Heh, I agree. See below "Disable notifications for this Pull Request".

@floere

Also, the order of comments is out of sync. The one that passes (3rd last) is the latest one.

@rkh
Owner

Sorry, we are looking into optimizing @travisbot comments.

@floere

Eh, no worries. I wonder though why they can be that much out of order. Isn't some sort of queue used?

@rkh
Owner

Not sure. We have them in RabbitMQ. We had some issues with Heroku and other services being down a lot the last few days, maybe something went wrong there.

@floere

Even the it's strange it's out of order, don't you think? OTOH, it's not that important (to me).

@floere

Anything stopping this one from being pulled?

@rkh
Owner

Me spending too much time at conferences. I'll look into this this week, potentially today.

@floere

Hehe, thanks for the quick response. I was mainly wondering whether I can remove this from my TODO list.

@floere

Still spending too much time at conferences?

The changes (https://github.com/sinatra/sinatra/pull/492/files) are pretty small – and beautify the tests quite a bit. If I can help with the review, let me know.

@travisbot

This pull request fails (merged e59b62e into b882ab3).

@floere

Again, it doesn't really fail – it segfaults (ie. we don't know whether it would fail).

@rkh rkh merged commit 762967f into from
@rkh
Owner

Thanks for all the effort. :)

@floere

My pleasure. Note that since the pattern behaviour changes, some apps might break who have misused the patterns that now work correctly. (I did not know where to describe that though, and forgot to ask, I'm afraid)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Mar 23, 2012
  1. @floere

    set up "working" tests for path pattern compilation

    floere authored
    Note: We can optimize the tests by compiling only once for each group of examples
  2. @floere
  3. @floere

    clarify variable name

    floere authored
  4. @floere
  5. @floere
  6. @floere
Commits on Jun 17, 2012
  1. @floere
Commits on Jun 18, 2012
  1. @floere
  2. @floere

    .

    floere authored
  3. @floere

    marked test that fails in 1.8

    floere authored
  4. @floere
  5. @floere

    unless nil? -> if

    floere authored
  6. @floere
  7. @floere
  8. @floere

    at least it fails already

    floere authored
Commits on Jul 12, 2012
  1. @floere

    remove accidental indentation

    floere authored
This page is out of date. Refresh to see the latest.
Showing with 151 additions and 4 deletions.
  1. +12 −4 lib/sinatra/base.rb
  2. +139 −0 test/compile_test.rb
View
16 lib/sinatra/base.rb
@@ -1302,17 +1302,21 @@ def compile!(verb, path, block, options = {})
def compile(path)
keys = []
if path.respond_to? :to_str
- pattern = path.to_str.gsub(/[^\?\%\\\/\:\*\w]/) { |c| encoded(c) }
+ ignore = ""
+ pattern = path.to_str.gsub(/[^\?\%\\\/\:\*\w]/) do |c|
+ ignore << escaped(c).join if c.match(/[\.@]/)
@rkh Owner
rkh added a note

I think we might also want this for other symbols, like :, so maybe remove the if or something?

@floere
floere added a note

Sure, I can do that. Do you have more example patterns -> regexps?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ encoded(c)
+ end
pattern.gsub!(/((:\w+)|\*)/) do |match|
if match == "*"
keys << 'splat'
"(.*?)"
else
keys << $2[1..-1]
- "([^/?#]+)"
+ "([^#{ignore}/?#]+)"
end
end
- [/^#{pattern}$/, keys]
+ [/\A#{pattern}\z/, keys]
elsif path.respond_to?(:keys) && path.respond_to?(:match)
[path, path.keys]
elsif path.respond_to?(:names) && path.respond_to?(:match)
@@ -1328,10 +1332,14 @@ def compile(path)
def encoded(char)
enc = URI.escape(char)
- enc = "(?:#{Regexp.escape enc}|#{URI.escape char, /./})" if enc == char
+ enc = "(?:#{escaped(char, enc).join('|')})" if enc == char
enc = "(?:#{enc}|#{encoded('+')})" if char == " "
enc
end
+
+ def escaped(char, enc = URI.escape(char))
+ [Regexp.escape(enc), URI.escape(char, /./)]
+ end
public
# Makes the methods defined in the block and in the Modules given
View
139 test/compile_test.rb
@@ -0,0 +1,139 @@
+# I like coding: UTF-8
+require File.expand_path('../helper', __FILE__)
+
+class CompileTest < Test::Unit::TestCase
+
+ def self.converts pattern, expected_regexp
+ it "generates #{expected_regexp.source} from #{pattern}" do
+ compiled, _ = compiled pattern
+ assert_equal expected_regexp, compiled
+ end
+ end
+ def self.parses pattern, example, expected_params
+ it "parses #{example} with #{pattern} into params #{expected_params}" do
+ compiled, keys = compiled pattern
+ match = compiled.match(example)
+ fail %Q{"#{example}" does not parse on pattern "#{pattern}".} unless match
+
+ # Aggregate e.g. multiple splat values into one array.
+ #
+ params = keys.zip(match.captures).reduce({}) do |hash, mapping|
+ key, value = mapping
+ hash[key] = if existing = hash[key]
+ existing.respond_to?(:to_ary) ? existing << value : [existing, value]
+ else
+ value
+ end
+ hash
+ end
+
+ assert_equal(expected_params, params)
+ end
+ end
+ def self.fails pattern, example
+ it "does not parse #{example} with #{pattern}" do
+ compiled, _ = compiled pattern
+ match = compiled.match(example)
+ fail %Q{"#{pattern}" does parse "#{example}" but it should fail} if match
+ end
+ end
+ def compiled pattern
+ app ||= mock_app {}
+ compiled, keys = app.send(:compile, pattern)
+ [compiled, keys]
+ end
+
+ converts "/", %r{\A/\z}
+ parses "/", "/", {}
+
+ converts "/foo", %r{\A/foo\z}
+ parses "/foo", "/foo", {}
+
+ converts "/:foo", %r{\A/([^/?#]+)\z}
+ parses "/:foo", "/foo", "foo" => "foo"
+ parses "/:foo", "/foo.bar", "foo" => "foo.bar"
+ parses "/:foo", "/foo%2Fbar", "foo" => "foo%2Fbar"
+ fails "/:foo", "/foo?"
+ fails "/:foo", "/foo/bar"
+ fails "/:foo", "/"
+ fails "/:foo", "/foo/"
+
+ converts "/föö", %r{\A/f%C3%B6%C3%B6\z}
+ parses "/föö", "/f%C3%B6%C3%B6", {}
+
+ converts "/:foo/:bar", %r{\A/([^/?#]+)/([^/?#]+)\z}
+ parses "/:foo/:bar", "/foo/bar", "foo" => "foo", "bar" => "bar"
+
+ converts "/hello/:person", %r{\A/hello/([^/?#]+)\z}
+ parses "/hello/:person", "/hello/Frank", "person" => "Frank"
+
+ converts "/?:foo?/?:bar?", %r{\A/?([^/?#]+)?/?([^/?#]+)?\z}
+ parses "/?:foo?/?:bar?", "/hello/world", "foo" => "hello", "bar" => "world"
+ parses "/?:foo?/?:bar?", "/hello", "foo" => "hello", "bar" => nil
+ parses "/?:foo?/?:bar?", "/", "foo" => nil, "bar" => nil
+ parses "/?:foo?/?:bar?", "", "foo" => nil, "bar" => nil
+
+ converts "/*", %r{\A/(.*?)\z}
+ parses "/*", "/", "splat" => ""
+ parses "/*", "/foo", "splat" => "foo"
+ parses "/*", "/foo/bar", "splat" => "foo/bar"
+
+ converts "/:foo/*", %r{\A/([^/?#]+)/(.*?)\z}
+ parses "/:foo/*", "/foo/bar/baz", "foo" => "foo", "splat" => "bar/baz"
+
+ converts "/:foo/:bar", %r{\A/([^/?#]+)/([^/?#]+)\z}
+ parses "/:foo/:bar", "/user@example.com/name", "foo" => "user@example.com", "bar" => "name"
+
+ converts "/test$/", %r{\A/test(?:\$|%24)/\z}
+ parses "/test$/", "/test$/", {}
+
+ converts "/te+st/", %r{\A/te(?:\+|%2B)st/\z}
+ parses "/te+st/", "/te+st/", {}
+ fails "/te+st/", "/test/"
+ fails "/te+st/", "/teeest/"
+
+ converts "/test(bar)/", %r{\A/test(?:\(|%28)bar(?:\)|%29)/\z}
+ parses "/test(bar)/", "/test(bar)/", {}
+
+ converts "/path with spaces", %r{\A/path(?:%20|(?:\+|%2B))with(?:%20|(?:\+|%2B))spaces\z}
+ parses "/path with spaces", "/path%20with%20spaces", {}
+ parses "/path with spaces", "/path%2Bwith%2Bspaces", {}
+ parses "/path with spaces", "/path+with+spaces", {}
+
+ converts "/foo&bar", %r{\A/foo(?:&|%26)bar\z}
+ parses "/foo&bar", "/foo&bar", {}
+
+ converts "/:foo/*", %r{\A/([^/?#]+)/(.*?)\z}
+ parses "/:foo/*", "/hello%20world/how%20are%20you", "foo" => "hello%20world", "splat" => "how%20are%20you"
+
+ converts "/*/foo/*/*", %r{\A/(.*?)/foo/(.*?)/(.*?)\z}
+ parses "/*/foo/*/*", "/bar/foo/bling/baz/boom", "splat" => ["bar", "bling", "baz/boom"]
+ fails "/*/foo/*/*", "/bar/foo/baz"
+
+ converts "/test.bar", %r{\A/test(?:\.|%2E)bar\z}
+ parses "/test.bar", "/test.bar", {}
+ fails "/test.bar", "/test0bar"
+
+ converts "/:file.:ext", %r{\A/([^\.%2E/?#]+)(?:\.|%2E)([^\.%2E/?#]+)\z}
+ parses "/:file.:ext", "/pony.jpg", "file" => "pony", "ext" => "jpg"
+ parses "/:file.:ext", "/pony%2Ejpg", "file" => "pony", "ext" => "jpg"
+ fails "/:file.:ext", "/.jpg"
+
+ converts "/:name.?:format?", %r{\A/([^\.%2E/?#]+)(?:\.|%2E)?([^\.%2E/?#]+)?\z}
+ parses "/:name.?:format?", "/foo", "name" => "foo", "format" => nil
+ parses "/:name.?:format?", "/foo.bar", "name" => "foo", "format" => "bar"
+ parses "/:name.?:format?", "/foo%2Ebar", "name" => "foo", "format" => "bar"
+ fails "/:name.?:format?", "/.bar"
+
+ converts "/:user@?:host?", %r{\A/([^@%40/?#]+)(?:@|%40)?([^@%40/?#]+)?\z}
+ parses "/:user@?:host?", "/foo@bar", "user" => "foo", "host" => "bar"
+ parses "/:user@?:host?", "/foo.foo@bar", "user" => "foo.foo", "host" => "bar"
+ parses "/:user@?:host?", "/foo@bar.bar", "user" => "foo", "host" => "bar.bar"
+
+ # From https://gist.github.com/2154980#gistcomment-169469.
+ #
+ # converts "/:name(.:format)?", %r{\A/([^\.%2E/?#]+)(?:\(|%28)(?:\.|%2E)([^\.%2E/?#]+)(?:\)|%29)?\z}
+ # parses "/:name(.:format)?", "/foo", "name" => "foo", "format" => nil
+ # parses "/:name(.:format)?", "/foo.bar", "name" => "foo", "format" => "bar"
+ fails "/:name(.:format)?", "/foo."
+end
Something went wrong with that request. Please try again.