Long list of whitelisted chararacters in .htaccess cause 500 error

[gideonso]

Short description of the issue

@ryancramerdesign

For utf8 page name, we need to whitelist characters both in config.php and .htaccess 16A or 16B. But there is a very long list of Chinese Characters. I found that if the number of characters put in the .htaccess file exceeds certain number causes 500 error. Is there any way to whitelist all the utf8 character with a more convenient way?

Adrian Mak has proposed to use ut8 code range to whitelist all utf8 characters. Please see his post here: https://processwire.com/talk/topic/12776-pw-3012-support-for-extended-utf8-page-namesurls/?page=2&tab=comments#comment-146652

Expected behavior

Page loads without a 500 error

Actual behavior

Page shows a 500 eror

• ProcessWire version: latest version 3.0.123
• (Optional) PHP version: 7.1
• (Optional) MySQL version: MariaDB 10.3.11

[ryancramerdesign]

@gideonso The .htaccess file is already using pattern matching (regular expression) so it is not necessary to have all the characters in there. You can define ranges. For instance, our default...

RewriteCond %{REQUEST_URI} "^/~?[-_.a-zA-Z0-9/]*$"

....means that any upper or lower ASCII alpha character is allowed, as are digits and hyphen, underscore or period. You can do the same with the characters that you want to define. You'll probably need to use UTF-8 character class ranges, which I think define them as hex numbers preceded by a backslash. To be honest I don't know much about that, other than that it's possible to do in your .htaccess, but I've just never had the need arise with the sites I work on.

If you find tracking down the right UTF-8 pattern is too much trouble, you could also comment out that particular .htaccess condition. It should run fine without it. It just means PW will receive potentially more URLs that would otherwise have gone to a 403 or 404, and PW will render the 404 instead. As for $config->pageNameWhitelist, it is also optional—leaving it blank will let it support any characters you want. But if you already have that part working with your characters, I would suggest keeping it, as I think having a whitelist is always better than not. However, I'm not currently aware of any issues with leaving these blank if/when necessary.

[netcarver]

/remind me to review this issue in 1 week

[reminders]

@netcarver set a reminder for Feb 19th 2019

[gideonso]

@ryancramerdesign Do you mean I can add utf8 code range to the htaccess file like the following?

RewriteCond %{REQUEST_URI} "^/~?[a-zA-Z0-9-._/-\x{4e00}-\x{9fa5}]*$"

Gideon

[ryancramerdesign]

@gideonso I don't know the technical details of how to express such a range in an .htaccess regular expression, but it looks like it's worth a try. If that's not it, it may take some more trial and error, and searching around for the solution. I'd be curious to hear when you find the right RewriteCond to match your needs here. But ultimately if you don't find it, you'll probably want to just remove the RewriteCond completely. The RewriteCond is an optimization to filter out requests that PW wouldn't be able to serve, but not technically necessary, as PW will still throw the appropriate 404 when it receives a request that doesn't match a URL in the system.

[gideonso]

@ryancramerdesign Thanks. I will test what is the best expression to use. By the way, is there any possibility to use utf8 code range in config.php to whitelist characters?? As you mentioned in your previous post, it is better to have the whitelist than leave it blank.

[ryancramerdesign]

@gideonso Ranges aren't supported with the $config property. Though with good reason, as it uses a faster matching technique than a regular expression could. While it is better to have this defined than not to, if the quantity of characters is so large that it's not practical to maintain, then I think it would be preferable to leave it blank. If there are characters you want to exclude from URLs, you can always do it from .htaccess or from inspecting the $_SERVER['REQUEST_URI'] property from PHP and throwing a Wire404Exception() if it contains something you want to disallow.