-
Notifications
You must be signed in to change notification settings - Fork 90
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Regex engine: allow unicode character ranges #332
Comments
A reasonable workaround is to (roughly) invert the ascii character class
So I guess if the issue is in the library, doing nothing is a very reasonable resolution. |
The service does indeed delegate the regular expression filtering to the POSIX library. It uses a UTF-8 locale if available. Should we close the issue or mark it as invalid or do something else? |
It might be worth to have our own regex parser. This would allow to correct behaviour like the above. It would also solve some potential security issues. Hence, I keep this one with this direction as "enhancement". |
I don't see much reason for a home grown solution, which has to implement http://www.unicode.org/reports/tr18/ in the end. There are already widely adopted solutions available for this particular requirement:
See: |
It's demo time! http://overpass-turbo.eu/s/kZx Commit: in feature/regexp_icu branch: master...mmd-osm:feature/regexp_icu |
BTW: Another interesting use case which is currently not supported: find nodes with [bbox:{{bbox}}][regexp:ICU];
(
node[~"^addr:.*"~"."];
-
node[~"^(?!addr:).*$"~"."];
);
out meta; |
A use case for proper Unicode support would be to easily find those streets where the |
For info re Myanmar there have been conflicts in OSM re which coding system to use (see https://en.wikipedia.org/wiki/Burmese_Wikipedia#Challenges in wikipedia for some info). This was about 14 months ago. I can't say what the current state of the OSM data is there; just mentioning it to provide a bit of background. |
It's still an issue. I asked the Yangonese OpenStreetMap community about it a month ago. If I understood them correctly, Burmese smartphone vendors even modify Android in a way that it is compatible to this non-Unicode Zawgyi font, apparently making it incompatible to Unicode then. I am still hoping that the issue will solve itself over time as they slowly migrate to Unicode. The alternative would be to add into all the editors for OSM an automatic conversion from Zawgyi to Unicode (and to be honest, I don't think it can be detected automatically if the input is Zawgyi). |
Another use case here, analyzing unlikely name:en tags coming from Maps.me users: mapsme/omim#7262 |
On IRC someone wondered about finding objects with mixed language names and I thought the very simple approach might work:
The answer is:
So I guess the regex implementation doesn't support such character ranges?
The text was updated successfully, but these errors were encountered: