[4.x]: Page with non-ASCII characters in the slug returns 200

[Romanavr]

What happened?

Description

More context for this problem can be found in the following issue, putyourlightson/craft-blitz#730

The main problem for me is that it's possible to visit a page with non-ASCII characters, while slug is set to ASCII only, pure English.
Also, 'limitAutoSlugsToAscii' => true.

As I understand, Craft in-depth checking against non-ASCII chars and replacing them while looking for necessary route.
I would like to have an option to avoid such behaviour as it causes problem in my case(with the Blitz plugin).
At least, it makes sense to do this for cases when limitAutoSlugsToAscii is set to true

Steps to reproduce

1. Create any page with the english slug, without ASCII symbols, https://putyourlightson.com/plugins (plugins)
2. Visit this page by replacing some characters with non-ASCII characters, for example, https://putyourlightson.com/plugi%C5%84s (plugińs)

Expected behavior

The page says 200 and loads the original content

Actual behavior

It should return a 404 and say that no such page was found.

Craft CMS version

Pro 4.12.0

PHP version

8.2.13

Operating system and version

Ubuntu

Database type and version

MySQL8.0.39

Image driver and version

No response

Installed plugins and versions

• Amazon S3: 2.2.1
• Blitz: 4.23.0
• Button Box: 4.2.4
• Dumper: 5.0.1
• Embedded Assets: 4.1.2
• Fetch: 2.0.0-beta.1
• Guest Entries: 4.0.1
• ImageOptimize: 4.0.9
• Linkit: 4.0.4.1
• Mailgun: 3.1.0
• Redactor: 3.1.0
• School calendar: 5.0.7
• Sentry Logger: 4.1.6
• SEOmatic: 4.1.2
• Spoon: 4.0.4
• Sprig: 2.13.0
• SuperSort: 4.0.0
• Super Table: 3.0.14
• Tags: 2.0.0
• Vite: 4.0.10

[i-just]

Hi, thanks for reaching out!

First of all, let me clarify that the only purpose of limitAutoSlugsToAscii is to enforce the auto-generated slugs to contain only ASCII characters. Once that’s done, there’s nothing stopping a user from amending the slug to contain non-ASCII characters.

That said, I see the behaviour you’re referring to. It comes from your database (from MySQL), not Craft, and it varies depending on the selected charset and collation for your tables.

When a web request is made and it reaches Craft, Craft will first figure out where to route it.

In this case, we’re talking about having an entry with a slug of plugins. So, when UrlManager->_getMatchedElementRoute() is reached, it’ll eventually work its way down to attempting to find an element by URI.
To do so, it’ll perform a query that attempts to find an element that has the elements_sites.uri set to the requested path. Querying the elements_sites table where uri is plugins and then performing the same request for uri plugińs can return the exact same element depending on what charset and collation you’re using.

For example, if I use charset utf8mb4 and collation utf8mb4_0900_ai_ci, querying where uri is plugins and then where uri is plugińs returns the exact same element.
If I change my collation to, e.g. utf8mb4_pl_0900_ai_ci, querying where uri is plugins will return an entry, but querying where uri is plugińs will not.

I hope this helps clarify things!

I’ll close this now, but feel free to reply if you run into any further issues.

[Romanavr]

Hi, thanks for reaching out!

First of all, let me clarify that the only purpose of limitAutoSlugsToAscii is to enforce the auto-generated slugs to contain only ASCII characters. Once that’s done, there’s nothing stopping a user from amending the slug to contain non-ASCII characters.

That said, I see the behaviour you’re referring to. It comes from your database (from MySQL), not Craft, and it varies depending on the selected charset and collation for your tables.

When a web request is made and it reaches Craft, Craft will first figure out where to route it.

In this case, we’re talking about having an entry with a slug of plugins. So, when UrlManager->_getMatchedElementRoute() is reached, it’ll eventually work its way down to attempting to find an element by URI. To do so, it’ll perform a query that attempts to find an element that has the elements_sites.uri set to the requested path. Querying the elements_sites table where uri is plugins and then performing the same request for uri plugińs can return the exact same element depending on what charset and collation you’re using.

For example, if I use charset utf8mb4 and collation utf8mb4_0900_ai_ci, querying where uri is plugins and then where uri is plugińs returns the exact same element. If I change my collation to, e.g. utf8mb4_pl_0900_ai_ci, querying where uri is plugins will return an entry, but querying where uri is plugińs will not.

I hope this helps clarify things!

I’ll close this now, but feel free to reply if you run into any further issues.

Absolutely, thank you very much for the detailed explanation!