[Routing] UTF-8 support for parameter requirements

[c960657]

Q	A
Branch?	2.7
Bug fix?	yes
New feature?	no
BC breaks?	no
Deprecations?	no
Tests pass?	yes
Fixed tickets
License	MIT
Doc PR

Assume I have a route /{foo} and a requirement \pL. I would expect this to match the request URL /%C3%A9 (URL-encoded version of /é), but it doesn't.

In UrlGenerator::doGenerate() it is generally assumed that the variables in $mergedParams are UTF-8-encoded (because they are passed to rawurlencode() that escapes each octet in the UTF-8-encoded string as described in RFC 3986). However, the requirement checkout using preg_match() does not use the u modifier, so the requirement . does not match the above URL, but .. does. In other words, the pattern should match the bytes in the UTF-8 encoded string, not the string itself (so to speak).

This patch adds the u modifier but preserves backwards compatibility with patterns made that circumvents the current limitation. In the master branch, the second preg_match() should be omitted.

This patch is slightly more liberal in validating parameter requirements, so it may break BC in some edge cases if people have worked around the existing limitation.

I know that the Routing component does not claim to be fully UTF-8 compliant, but this a least takes it a bit closer.

[Tobion]

Ref #5236

You changed the UrlGenerator to use the u modifier for regexes but the UrlMatcher/Compiler still doesn't use the u modifer, see https://github.com/symfony/symfony/blob/master/src/Symfony/Component/Routing/RouteCompiler.php#L166
So that seems inconsistent and buggy now because you could generate a URL that doesn't match.

[Tobion]

Status: Needs Work

[nicolas-grekas]

Adding u to the router matcher would break URLs that are not encoded as UTF-8, which is perfectly legal. How should we deal with that?

[c960657]

@Tobion You are right. I'll look into that also.

@nicolas-grekas I suggest we check both variants similarly to hos this PR does in the generator, possibly depending on the value of the default_charset or input_encoding ini setting or some explicit configuration setting on the router.

You are right that non-UTF-8 URLs are valid. However, supporting such URLs with Symfony seems like an uphill struggle. AFAICT only UTF-8 is supported for the static parts of the routes when using the Yaml and XML file format (because the Component\Yaml and PHP's DOM extension always return UTF-8-encoded strings). Browsers seems to assume that URL's are UTF-8-encoded when pretty-printing URLs e.g. in the address field. So I wonder whether this is worth supporting. Does Symfony really work well with other encodings?

If we decide to drop support for this in future versions, we could still detect non-UTF-8 strings in incoming requests and convert them to UTF-8 accordingly.

Anyway, dropping support for non-UTF-8 is not necessary in order to accomplish the goals of this PR.

[nicolas-grekas]

@c960657 IMHO we should not auto-magically deal with charsets. Let's add an option to enable UTF-8 on routes, either on individual routes or globally on the app, and add the u modifier accordingly.

[c960657]

I have made a new PR against the master branch: #19562

The default encoding is UTF-8. It can be changed with an application-wide setting if one needs to support legacy URLs (I can't think of other reasons for not using UTF-8).