Use raw_path instead of path in the router to be able to differentiate requests which include %2F

[ChiliJohnson]

I proposed this functionality in #1827 and added a bit of color in there, this was a quick-and-dirty change so I don't really have a good sense for any far-reaching implications of this change. But it was relatively-simple to implement so I thought I'd give it a shot.

Using raw_path instead of the pre-parsed path in the router would enable the it to distinguish between paths which include URL-encoded slashes, e.g. between my%2Froute/example and my/route/example

[tomchristie]

Compare and contrast with a more well established Python web framework.

What does Flask do?

[Kludex]

@ChiliJohnson Are you still interested in this PR?

[ChiliJohnson]

Yes I'm definitely still interested!

As far as Flask / Werkzeug / WSGI go, it seems that PEP 3333 implicitly requires that the path variable (used for routing in Flask/Werkzeug) be pre-unquoted by time they reach the app. So as a result, Flask also has this inability to distinguish between / and %2F at the app level.

Here are a few relevant discussions about this in WSGI land:

• Routing considers quoted slashes (%2F) as slashes. pallets/werkzeug#477
• Add RAW_URI to environ (as Gunicorn does) pallets/werkzeug#1048
• http/wsgi.py: do not unquote path before injecting into environ benoitc/gunicorn#1211

I guess the question here is do we want Starlette to match this behavior with the WSGI-based projects, or use this feature as a point of differentiation from them?

It seems like it could be nice to give app developers more control in how they route requests, but at the same time I can see benefit to Starlette acting more like the other players in this space

[adriangb]

My vote would be to move forward with this change. I like the idea of giving developers more power.

One thing I’m not immediately clear on is what does this break? Can it brake any Starlette or FastAPI applications? Will this require changes in FastAPI itself?

[Kludex]

In any case, the decision other web framework made have influence in our decisions here...

Besides that, it doesn't look like there's many people interested in this over the years...

[ChiliJohnson]

This would break web apps relying on the fact that /path/some/example and /path%2Fsome/example currently match a single "/path/{param:path}" route, which given that most URLs are probably constructed in a consistent way, likely means that there isn't a huge surface area for affected FastAPI / Starlette apps.

A complicating factor here is that there's the potential that an upstream web server or reverse proxy may be pre-unquoting the URL before we even know about it. In the case of Uvicorn + Starlette (including the changes from this PR) this doesn't happen, but I can see how it would be possible; and in that case the behavior of routing should remain exactly the same

[kefir-]

As far as Flask / Werkzeug / WSGI go, it seems that PEP 3333 implicitly requires that the path variable (used for routing in Flask/Werkzeug) be pre-unquoted by time they reach the app. So as a result, Flask also has this inability to distinguish between / and %2F at the app level.

Is PEP 3333 (implicitly) in conflict with this explicit description in RFC 3986 aka STD 66? I should think the explicit RFC/STD takes precedence here.

2.4. When to Encode or Decode

[snip]

When a URI is dereferenced, the components and subcomponents
significant to the scheme-specific dereferencing process (if any)
must be parsed and separated before the percent-encoded octets within
those components can be safely decoded, as otherwise the data may be
mistaken for component delimiters. The only exception is for
percent-encoded octets corresponding to characters in the unreserved
set, which can be decoded at any time. For example, the octet
corresponding to the tilde ("~") character is often encoded as "%7E"
by older URI processing implementations; the "%7E" can be replaced by
"~" without changing its interpretation.

[kefir-]

Interpreting %2F as / is also against W3C recommendations, as described in Example 2 at https://www.w3.org/Addressing/URL/4_URI_Recommentations.html

Screenshot:

[ChiliJohnson]

Thanks for the great research @kefir- ! From what you've posted it definitely seems like the current behavior of the router is in breach of these core web standards; I guess the question now is is there too much inertia coming from all the apps out there which maybe (intentionally or unintentionally) depend on this incorrect behavior? Maybe it makes sense to release this as part of a new major version?

[kefir-]

I'll add another source. That is RFC 3986 section 2.2, which states:

URIs that differ in the replacement of a reserved character with its corresponding percent-encoded octet are not equivalent.

/ is a reserved character.

[fclairamb]

I have a concrete example of why this is a problem. I'm creating an HTTP proxy to simplify authentication around AWS's codeartifact and some URLs can contain / in their name.
This service hosts packages that can sometimes contain "/". A request to the package /npm/$repository/@angular%2Fforms/ is proxied to /npm/$repository/@angular/forms/ and becomes a 400.

So far the only way I found to work around this is to identify those queries and URL-encode the decoded URL string (which should have never been decoded in the first place).

Argument extracted from the path should indeed be URL-decoded but the Request.url property should not be decoded.

Please fix this.

[Kludex]

Related: django/asgiref#51

[Kludex]

@simonw If you have time, could you give me your opinion here? 🙏

[Kludex]

• Let's continue the discussion on Use the ASGI `raw_path` in the router #1827. It doesn't seem we have many people interested in this.