Unicode identifiers are not considered

[FichteFoll]

As I was made aware by sublimehq/sublime_text#460, unicode identifiers are currently not considered. The following function call is not recognized as such:

sâomething(arg=22)

[MattDMo]

This is probably true for just about every regex that matches an identifier:

[a-zA-Z_][a-zA-Z_0-9]*

I'm not sure how to work around it, though. Does Sublime honor [[:unicode:]]?

[FichteFoll]

http://www.geocities.jp/kosako3/oniguruma/doc/RE.txt

\p{Word}, [\p{Alpha}_], [[:word:]] or [[:alpha:]_] sound useful, but I'm not sure what exactly they match. Also you need to consider that they must not begin with numbers, such as [[[:alpha:]_]&&[^0-9]].

[facelessuser]

Yep, but I think most, if not all of sublime's syntax languages ignore unicode. Heck even some other editors ignore them as well. But it would definitely be nice to have them though. I imagine this would be a pretty big patch though, because you will have to modify a lot of places for variables, functions, and class names etc. I could see this being worth the effort if you are from a non-english speaking country.

[FichteFoll]

[A-Za-z_][A-Za-z0-9_]* is what's used mostly for identifier matches. You could simply replace them with the new match and then search for potential cases where you missed them.

[MattDMo]

@FichteFoll is Sublime's internal regex engine Oniguruma or PCRE? I know the YAML-tmLanguage files use Oniguruma, but the snippets reference links to the Boost::regex docs. I'm pretty sure the Find and Replace dialogs also use Boost, if I'm not mistaken.

[facelessuser]

@MattDMo From what I understand, Sublime does use different regex engines for different things (which I find kind of annoying), so yes, you would probably have to update snippets differently.

[FichteFoll]

@guillermooo could know more about this since I just assumed the docs were right in that regard and use Oniguruma. I never really validified that statement.

[facelessuser]

jps has mentioned that both boost and oniguruma are used, though he doesn't specifically mention what snippets use: http://www.sublimetext.com/forum/viewtopic.php?f=3&t=6354#p28594

You should be able to verify pretty quick which syntax that snippets recognize though.

Edit: he did mention snippets, my bad.

.tmLanguage files use the Oniguruma regex library, while the Find panel and snippets use boost regex.

[MattDMo]

Thanks for the link, Isaac. That explains a lot, and will help writing new regexes for PI and some other languages I'm working on.

[FichteFoll]

Here is a syntax definition that handles unicode: https://github.com/JoshuaWise/javascript-ultimate

I'm not sure if these are really all the identifiers you need, but they sure ain't little. Would be a nice use case for https://packagecontrol.io/packages/SyntaxHighlightTools because that allows to create some kind of regex repository that you can re-use throughout the whole syntax definition.
(Sadly it doesn't work for me currently.)

[MattDMo]

@FichteFoll thanks for the link. I'm reopening this issue for now and removing wontfix while I evaluate JS Ultimate and see if we can borrow anything from there.