New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unicode whitespace support #86
Comments
Are you running an old version of rascal? I seem to remember fixing this
|
Rascal plugin 0.5.2 on Juno |
okay, so reading this again.
so workaround: add the unicode ranges from the new regarding 3, try if this is still an issue with current release, since 0.5.2 is about 4 months old. |
kthx, works now! btw, it turned out I already ran 0.5.4. At least my Eclipse says so, while the website said 0.5.2 was the last one. |
Davy Landman notifications@github.com wrote:
Do we have \s as in Perl? Unless we have an (old fashioned) language definition that says otherwise, treating any unicode space as space is probably better than enumerating the characters. I see Vadim has forgotten formfeed, for example, which is commonly accepted as space (heavily used in Gnu code). -anya-anya |
Well, using unicode ranges should do the trick, since I hope that any possible whitespace character should be found in the whitespace category in the unicode standard. |
aka "fix utf plz"
In particular, the nasty \uC2A0 (unbreakable space) character just made me waste quite a lot of time and even made me believe for a second that I have a problem in my grammar.
Details: if the layout/whitespace definition is the usual [\ \n\t\r]+ !>> [\ \n\t\r], and this character is in the input stream, a parse error occurs. However, if we add it to the layout definition, the same error occurs (my guess is that it is somehow normalised to the normal space). In fact, even replaceAll(inputString,"\uC2A0"," ") does not solve the problem (again, my guess is that some normalisations occur somewhere).
I know enough Perl to overcome this for now, but I love Rascal much more ;)
The text was updated successfully, but these errors were encountered: