Allow non-ASCII identifiers #2457
Conversation
|
||
> **<sup>Lexer:<sup>** | ||
> IDENTIFIER_OR_KEYWORD: | ||
> XID_Start XID_Continue<sup>\*</sup> |
sfackler
Jun 3, 2018
Member
How does this compare to Go's identifier grammar: https://golang.org/ref/spec#Identifiers? It doesn't use XID_Start and XID_Continue, but instead the "Letter" and "Number, decimal digit" character classes.
How does this compare to Go's identifier grammar: https://golang.org/ref/spec#Identifiers? It doesn't use XID_Start and XID_Continue, but instead the "Letter" and "Number, decimal digit" character classes.
pyfisch
Jun 3, 2018
•
Author
Contributor
XID_Start includes "Letter" but also includes "Number, Letter" (for example Roman numerals like Ⅻ) and a few characters for compatibility and to ensure that XID_Start is closed under NFKC.
XID_Continue includes all of XID_Start and "Number, decimal digit" , marks to build combining characters from parts (e
+ ^
→ ê), 10 "Punctuation, Connector" and again a few for compatibility and NFKC.
There is golang/go#194 where combining characters are discussed as they are needed to write some Asian languages. See also: https://golang.org/doc/faq#unicode_identifiers
XID_Start includes "Letter" but also includes "Number, Letter" (for example Roman numerals like Ⅻ) and a few characters for compatibility and to ensure that XID_Start is closed under NFKC.
XID_Continue includes all of XID_Start and "Number, decimal digit" , marks to build combining characters from parts (e
+ ^
→ ê), 10 "Punctuation, Connector" and again a few for compatibility and NFKC.
There is golang/go#194 where combining characters are discussed as they are needed to write some Asian languages. See also: https://golang.org/doc/faq#unicode_identifiers
|
||
Two identifiers X, Y are considered to be equal if there [NFKC forms][TR15] are equal: NFKC(X) = NFKC(Y). | ||
|
||
A `unicode_idents` lint is added to the compiler. This lint is `allow` by default. The lint checks if any identifier in the current context contains a codepoint with a value equal to or greater than 0x80 (outside ASCII range). Not only locally defined identifiers are checked but also those imported from other crates and modules into the current context. |
est31
Jun 3, 2018
Contributor
I don't think the lint should be allow
by default. It should at least be warn by default. Starting to use non ascii idents should be a conscious choice, not an accidental one.
As for checking imported idents as well, I think this should be a separate lint. You might want to be able to import something from a foreign language crate but not want to have foreign language idents in your own code.
I don't think the lint should be allow
by default. It should at least be warn by default. Starting to use non ascii idents should be a conscious choice, not an accidental one.
As for checking imported idents as well, I think this should be a separate lint. You might want to be able to import something from a foreign language crate but not want to have foreign language idents in your own code.
shingtaklam1324
Jun 4, 2018
•
Reposting my comment from #2455 (comment), I think that there should be different lints, with different levels, as some languages should be allow/warn by default, and some others should be deny by default.
Reposting my comment from #2455 (comment), I think that there should be different lints, with different levels, as some languages should be allow/warn by default, and some others should be deny by default.
* Identifiers starting with numbers or "non letters": `42_the_answer`, `third√of7`, `◆◆◆`, ... | ||
* Emojis: 🙂, 🦀, 💩, ... | ||
|
||
Similar Unicode identifiers are normalized: `a1` and `a₁` refer to the same variable. This also applies to accented characters which can be represented in different ways. |
mark-i-m
Jun 3, 2018
Member
FWIW, this sentence does not render properly on my phone. I get a1
and a<box>
...
One other drawback that is not mentioned in the RFC is that unicode characters don't render properly on all terminals and fonts.
FWIW, this sentence does not render properly on my phone. I get a1
and a<box>
...
One other drawback that is not mentioned in the RFC is that unicode characters don't render properly on all terminals and fonts.
I wanted to emphasize how much of an ergonomics regression it would be for me if this was added and actually used. The combination of "hard to type" and "often incorrectly rendered" would make most libraries that use this useless for me. (Needless to say, I would be very unhappy if non-ascii characters were ever used in libstd)... |
I think the NFKC form restriction here sounds essential, so very nice catch there. We should keep the skeleton test proposed here too. Yet, I'm not convinced it addresses homograph attacks because confusions might arise with identifiers not in scope. We cannot run the skeleton test across all scopes obviously. In practice, there are almost no scenarios where more than a few scripts get used together, even when doing mathematics, so enabling each script separately sounds viable, ala
If the compiler encounters a allowable but currently disallowed script, then it might issue a warning giving both the line and column, as well as propose the correct |
> | ||
> For some languages, common transliteration systems exist (in particular, for the Latin-based writing systems). For other languages, users have larger difficulties to use Latin to write their native words. | ||
Additionally some math oriented projects may want to use identifiers closely resembling mathematical writing. |
dscorbett
Jun 3, 2018
NFKC removes the mathematical distinction of font style, as in ⟨ℍ⟩ vs. ⟨ℋ⟩.
NFKC removes the mathematical distinction of font style, as in ⟨ℍ⟩ vs. ⟨ℋ⟩.
pyfisch
Jun 4, 2018
Author
Contributor
@dscorbett yeah the idea is that while you can use stylized characters you should not go completely overboard and use all of them in the same formula. This applies also to the distinguished in natural language orthographies.
@dscorbett yeah the idea is that while you can use stylized characters you should not go completely overboard and use all of them in the same formula. This applies also to the distinguished in natural language orthographies.
`XID_Start` and `XID_Continue` are used as defined in the aforementioned standard. The definition of identifiers is forward compatible with each successive release of Unicode as only appropriate new characters are added to the classes but none are removed. | ||
|
||
Two identifiers X, Y are considered to be equal if there [NFKC forms][TR15] are equal: NFKC(X) = NFKC(Y). |
dscorbett
Jun 3, 2018
NFKC folds together some characters distinguished in natural language orthographies, such as Tifinagh ⟨ⵡ⟩ and ⟨ⵯ⟩.
NFKC folds together some characters distinguished in natural language orthographies, such as Tifinagh ⟨ⵡ⟩ and ⟨ⵯ⟩.
> **<sup>Lexer:<sup>** | ||
> IDENTIFIER_OR_KEYWORD: | ||
> XID_Start XID_Continue<sup>\*</sup> | ||
> | `_` XID_Continue<sup>+</sup> |
dscorbett
Jun 3, 2018
The guide-level explanation says that identifiers may contain “more letters, digits and some connecting punctuation”, but XID_Continue does not include connecting punctuation.
The guide-level explanation says that identifiers may contain “more letters, digits and some connecting punctuation”, but XID_Continue does not include connecting punctuation.
pyfisch
Jun 4, 2018
Author
Contributor
XID_Continue includes characters from the Pc class called "Punctuation, Connector".
XID_Continue includes characters from the Pc class called "Punctuation, Connector".
dscorbett
Jun 4, 2018
So it does. I must have been looking at XID_Start.
So it does. I must have been looking at XID_Start.
# Motivation | ||
[motivation]: #motivation | ||
|
||
Rust is written by many people who are not fluent in the English language. Using identifiers in ones native language eases writing and reading code for these developers. |
dscorbett
Jun 3, 2018
Some languages use U+200C ZERO WIDTH NON-JOINER and U+200D ZERO WIDTH JOINER, which are not in XID_Continue. Should they be allowed too? See section 2.3 of UAX #31.
Some languages use U+200C ZERO WIDTH NON-JOINER and U+200D ZERO WIDTH JOINER, which are not in XID_Continue. Should they be allowed too? See section 2.3 of UAX #31.
shingtaklam1324
Jun 4, 2018
To have proper support then they should be allowed, as it affects the rendering of text, which may cause it to have different meanings (I'm not sure, but it seems likely to me). However, this can cause issues like in C++ and Swift for example, different number of zero-width joiners or spaces are different identifiers, and that leads to a readability nightmare.
To have proper support then they should be allowed, as it affects the rendering of text, which may cause it to have different meanings (I'm not sure, but it seems likely to me). However, this can cause issues like in C++ and Swift for example, different number of zero-width joiners or spaces are different identifiers, and that leads to a readability nightmare.
dscorbett
Jun 4, 2018
That is why UAX #31 suggests allowing them only in certain positions, to avoid having multiple distinct identifiers look identical.
That is why UAX #31 suggests allowing them only in certain positions, to avoid having multiple distinct identifiers look identical.
Manishearth
Jun 5, 2018
Member
So AIUI these are used for:
-
Forcing explicit viramas in Indic scripts in consonant clusters. There is a semantic difference but it's exceedingly minor and mostly comes in play for Sanskrit. It's a pretty minor difference; AIUI it's used in certain kinds of compound words and is very easily omitted
-
Certain vowel presentational forms in Bengali and Oriya. Also minor.
-
Forcing letters to take different (word-medial, etc) forms in the Perso-Arabic script, used for:
-
Arabic affixes, when shown in isolation or when used with non-arabic words. Imagine you had to write something like "Rust's" where "Rust" is in the Latin script but the 's is in Arabic, you need the Arabic suffix to not use a word-initial text form. An example of this is the ب prefix preposition
-
AFAICT these are all somewhat optional (though preferred). The abbreviations one might be the most used.
(that said, there are much larger problems with using an RTL script in rust)
So AIUI these are used for:
-
Forcing explicit viramas in Indic scripts in consonant clusters. There is a semantic difference but it's exceedingly minor and mostly comes in play for Sanskrit. It's a pretty minor difference; AIUI it's used in certain kinds of compound words and is very easily omitted
-
Certain vowel presentational forms in Bengali and Oriya. Also minor.
-
Forcing letters to take different (word-medial, etc) forms in the Perso-Arabic script, used for:
-
Arabic affixes, when shown in isolation or when used with non-arabic words. Imagine you had to write something like "Rust's" where "Rust" is in the Latin script but the 's is in Arabic, you need the Arabic suffix to not use a word-initial text form. An example of this is the ب prefix preposition
-
AFAICT these are all somewhat optional (though preferred). The abbreviations one might be the most used.
(that said, there are much larger problems with using an RTL script in rust)
khaledhosny
Jun 5, 2018
ZWNJ has important use in Persian at least https://en.m.wikipedia.org/wiki/Zero-width_non-joiner
ZWNJ has important use in Persian at least https://en.m.wikipedia.org/wiki/Zero-width_non-joiner
Examples of valid identifiers are: | ||
|
||
* English language words: `color`, `image_width`, `line2`, `Photo`, `_unused`, ... | ||
* ASCII words in foreign languages: `die_eisenbahn`, `el_tren`, `artikel_1_grundgesetz` |
dscorbett
Jun 3, 2018
Since the primary beneficiaries are developers who prefer not using English identifiers, these examples wouldn’t be in foreign languages for all readers.
Since the primary beneficiaries are developers who prefer not using English identifiers, these examples wouldn’t be in foreign languages for all readers.
pyfisch
Jun 4, 2018
Author
Contributor
Yeah I tried to follow the spirit of the section to explain as if you were "teaching it to another Rust programmer" and since the explanation in English I found it appropriate to refer to these languages as foreign. (Do you know a better word?)
Normally I would say: "Du kannst den Variablen auch deutsche Namen geben, Umlaute funktionieren auch." 😉
Yeah I tried to follow the spirit of the section to explain as if you were "teaching it to another Rust programmer" and since the explanation in English I found it appropriate to refer to these languages as foreign. (Do you know a better word?)
Normally I would say: "Du kannst den Variablen auch deutsche Namen geben, Umlaute funktionieren auch."
dscorbett
Jun 4, 2018
How about “other”? Compare the last example, which says “other scripts” instead of “foreign scripts”.
How about “other”? Compare the last example, which says “other scripts” instead of “foreign scripts”.
SimonSapin
Jun 4, 2018
Contributor
I think that “non-English languages” works. But that distinction from the previous point isn’t really relevant in the first place. Identifiers made of ASCII letters that do not form a “real” word in any language are also valid. Maybe replace both points with “ASCII letters and digits”?
I think that “non-English languages” works. But that distinction from the previous point isn’t really relevant in the first place. Identifiers made of ASCII letters that do not form a “real” word in any language are also valid. Maybe replace both points with “ASCII letters and digits”?
# Reference-level explanation | ||
[reference-level-explanation]: #reference-level-explanation | ||
|
||
Identifiers in Rust are based on the [Unicode® Standard Annex #31 Unicode Identifier and Pattern Syntax][TR31]. Rust compilers shall use at least Revision 27 of the standard. |
dscorbett
Jun 3, 2018
The revision should be specified exactly. Otherwise, the same identifier in the same Rust version could be valid in one compiler but another.
The revision should be specified exactly. Otherwise, the same identifier in the same Rust version could be valid in one compiler but another.
pyfisch
Jun 4, 2018
Author
Contributor
I don't want to lock Rust compilers to a specific Unicode revision. An alternative would be to state the supported Unicode revision for each Rust edition.
I don't want to lock Rust compilers to a specific Unicode revision. An alternative would be to state the supported Unicode revision for each Rust edition.
dscorbett
Jun 4, 2018
New versions of Rust should use new versions of Unicode, of course, but any given version of Rust should have an unambiguous definition of identifiers. Your alternative is good.
Here is why the current text is wrong. Let’s say, for example, that revision 28 adds ⟨⍙⟩ to XID_Start, just in time for Rust 1.40.0. Would ⟨⍙abc⟩ be a valid identifier in Rust 1.40.0? It would be valid in compilers that use revision 28, but not in compilers that still use revision 27. Both compilers would be correct and yet disagree about something as basic as identifier validity.
New versions of Rust should use new versions of Unicode, of course, but any given version of Rust should have an unambiguous definition of identifiers. Your alternative is good.
Here is why the current text is wrong. Let’s say, for example, that revision 28 adds ⟨⍙⟩ to XID_Start, just in time for Rust 1.40.0. Would ⟨⍙abc⟩ be a valid identifier in Rust 1.40.0? It would be valid in compilers that use revision 28, but not in compilers that still use revision 27. Both compilers would be correct and yet disagree about something as basic as identifier validity.
SimonSapin
Jun 4, 2018
Contributor
FWIW we’ve already been updating Unicode for the standard library (e.g. char::to_lowercase
), and have https://doc.rust-lang.org/std/char/constant.UNICODE_VERSION.html to indicate which version is in use.
FWIW we’ve already been updating Unicode for the standard library (e.g. char::to_lowercase
), and have https://doc.rust-lang.org/std/char/constant.UNICODE_VERSION.html to indicate which version is in use.
SimonSapin
Jun 4, 2018
Contributor
UAX 31 should be referenced for information/context, but I think that its revision number can be removed entirely. This RFC defines identifier syntax based on the XID_Start
and XID_Continue
properties, which are specified exactly for a given version of Unicode (e.g. 10.0.0).
To maintain Rust’s stability promise, we need to ensure that:
- If a given string is a valid identifier in a given version of Rust, it needs to stay a valid identifier in later versions of Rust.
- If two strings are valid identifiers in a given version of Rust, whether they compare equal after normalization needs to be unchanged in later versions of Rust.
My reading of UAX 31 and 15 is that we can do that and still update the to newer versions of Unicode in newer versions of Rust backward-compatibly without involving Rust editions. We should have some way communicate to users the version mapping, and I think that the existing std::char::UNICODE_VERSION
(with docs archive like https://doc.rust-lang.org/1.15.0/std/char/constant.UNICODE_VERSION.html) is already satisfactory for that.
This translates 1. above into the requirement UAX31-R1b Stable Identifier:
To meet this requirement, an implementation shall guarantee that identifiers are stable across versions of the Unicode Standard: that is, once a string qualifies as an identifier, it does so in all future versions.
Per https://www.unicode.org/reports/tr31/#Backward_Compatibility I believe that XID_Start XID_Continue* | "_" XID_Continue+
meets that requirement. (Since XID_Start / XID_Continue include Other_ID_Start / Other_ID_Continue respectively. I’d appreciate if someone could double-check and confirm my understanding.)
Additionally, code points that are not assigned in a given Unicode version cannot be in XID_Start
or XID_Continue
in that Unicode version. And https://www.unicode.org/reports/tr15/#Versioning defines:
It is crucial that Normalization Forms remain stable over time. That is, if a string that does not have any unassigned characters is normalized under one version of Unicode, it must remain normalized under all future versions of Unicode.
Updating to a newer Unicode does not change the result of NFKC(X)
or NFKC(Y)
, so NFKC(X) = NFKC(Y)
is also unchanged and requirement 2. above is also met.
Let’s say, for example, that revision 28 adds ⟨⍙⟩ to XID_Start, just in time for Rust 1.40.0. Would ⟨⍙abc⟩ be a valid identifier in Rust 1.40.0?
It would a Unicode version rather than in a revision of UAX 31 that adds it, but yes, if Rust 1.40.0 is the one that upgrades to that version of Unicode (regardless of when that version of Unicode was released), then ⍙abc
would be a valid ident in Rust 1.40.0 but not 1.39.0. This is similar to using new a language feature newly in 1.40.0.
UAX 31 should be referenced for information/context, but I think that its revision number can be removed entirely. This RFC defines identifier syntax based on the XID_Start
and XID_Continue
properties, which are specified exactly for a given version of Unicode (e.g. 10.0.0).
To maintain Rust’s stability promise, we need to ensure that:
- If a given string is a valid identifier in a given version of Rust, it needs to stay a valid identifier in later versions of Rust.
- If two strings are valid identifiers in a given version of Rust, whether they compare equal after normalization needs to be unchanged in later versions of Rust.
My reading of UAX 31 and 15 is that we can do that and still update the to newer versions of Unicode in newer versions of Rust backward-compatibly without involving Rust editions. We should have some way communicate to users the version mapping, and I think that the existing std::char::UNICODE_VERSION
(with docs archive like https://doc.rust-lang.org/1.15.0/std/char/constant.UNICODE_VERSION.html) is already satisfactory for that.
This translates 1. above into the requirement UAX31-R1b Stable Identifier:
To meet this requirement, an implementation shall guarantee that identifiers are stable across versions of the Unicode Standard: that is, once a string qualifies as an identifier, it does so in all future versions.
Per https://www.unicode.org/reports/tr31/#Backward_Compatibility I believe that XID_Start XID_Continue* | "_" XID_Continue+
meets that requirement. (Since XID_Start / XID_Continue include Other_ID_Start / Other_ID_Continue respectively. I’d appreciate if someone could double-check and confirm my understanding.)
Additionally, code points that are not assigned in a given Unicode version cannot be in XID_Start
or XID_Continue
in that Unicode version. And https://www.unicode.org/reports/tr15/#Versioning defines:
It is crucial that Normalization Forms remain stable over time. That is, if a string that does not have any unassigned characters is normalized under one version of Unicode, it must remain normalized under all future versions of Unicode.
Updating to a newer Unicode does not change the result of NFKC(X)
or NFKC(Y)
, so NFKC(X) = NFKC(Y)
is also unchanged and requirement 2. above is also met.
Let’s say, for example, that revision 28 adds ⟨⍙⟩ to XID_Start, just in time for Rust 1.40.0. Would ⟨⍙abc⟩ be a valid identifier in Rust 1.40.0?
It would a Unicode version rather than in a revision of UAX 31 that adds it, but yes, if Rust 1.40.0 is the one that upgrades to that version of Unicode (regardless of when that version of Unicode was released), then ⍙abc
would be a valid ident in Rust 1.40.0 but not 1.39.0. This is similar to using new a language feature newly in 1.40.0.
dscorbett
Jun 4, 2018
Right. It was a rhetorical question to show why specifying the Unicode version exactly is necessary.
Right. It was a rhetorical question to show why specifying the Unicode version exactly is necessary.
Manishearth
Jun 4, 2018
Member
Bear in mind, the unicode annexes also do get updated every few versions, often to accommodate for new kinds of code points but sometimes to just fix things.
Bear in mind, the unicode annexes also do get updated every few versions, often to accommodate for new kinds of code points but sometimes to just fix things.
|
||
Note: The confusable detection is set to `warn` instead of `deny` to enable forward compatibility. The list of confusable characters will be extended in the future and programs that were once valid would fail to compile. | ||
|
||
The confusable detection algorithm is based on [Unicode® Technical Standard #39 Unicode Security Mechanisms Section 4 Confusable Detection][TR39Confusable]. For every distinct identifier X in the current scope execute the function `skeleton(X)`. If there exist two distinct identifiers X and Yin the same crate where `skeleton(X) = skeleton(Y)` report it. |
dscorbett
Jun 3, 2018
skeleton
does not handle default-ignorable code points, but many are in XID_Continue. The lint should delete them before running skeleton
.
skeleton
does not handle default-ignorable code points, but many are in XID_Continue. The lint should delete them before running skeleton
.
pyfisch
Jun 4, 2018
Author
Contributor
I see why this would be a good idea but if we decide to do it I would like to consult with the Unicode people first to find out why they did not do it in this way.
I see why this would be a good idea but if we decide to do it I would like to consult with the Unicode people first to find out why they did not do it in this way.
dscorbett
Jun 4, 2018
I suspect they didn’t do it that way because UTS #39’s confusability data’s “primary goal is to include characters that would be Status=Allowed”, and default-ignorable code points are Status=Restricted.
I suspect they didn’t do it that way because UTS #39’s confusability data’s “primary goal is to include characters that would be Status=Allowed”, and default-ignorable code points are Status=Restricted.
eternaleye
Jun 5, 2018
Typo: Yin
-> Y in
(missing space)
Typo: Yin
-> Y in
(missing space)
Examples of invalid identifiers are: | ||
|
||
* Keywords: `impl`, `fn`, `_` (underscore), ... | ||
* Identifiers starting with numbers or "non letters": `42_the_answer`, `third√of7`, `◆◆◆`, ... |
dscorbett
Jun 3, 2018
third√of7
does not start with a number or non-letter.
third√of7
does not start with a number or non-letter.
1. Require all identifiers to be in NFKC or NFC form. | ||
2. Two identifiers are only equal if their codepoints are equal. | ||
3. Perform NFC mapping instead of NFKC mapping for identifiers. | ||
4. Only a number of common scripts could be supported. |
dscorbett
Jun 3, 2018
Other possible variants are the restriction levels of UTS #39.
Other possible variants are the restriction levels of UTS #39.
pyfisch
Jun 4, 2018
Author
Contributor
This is listed as variant 5.
This is listed as variant 5.
Swift supports unicode identifier as a another point of comparison for the exact allowed behavor: https://developer.apple.com/library/content/documentation/Swift/Conceptual/Swift_Programming_Language/LexicalStructure.html#//apple_ref/doc/uid/TP40014097-CH30-ID412 @mark-i-m are those concerns about widespread use of non-ascii identifiers borne out in your experience in other languages that allow non-ascii identifiers? |
Yes, in particular,
|
As well as the other concerns that have been raised in the thread, there is something which I experienced while doing some work in Ruby, which is that for some languages, there are multiple encoding systems, and all of which have significant use. One example is Chinese, which has Unicode, and also GBK which is used about as much as Unicode. This can cause issues as code encoded in GBK is valid code right now with ASCII idents, but with this addition, if a user decides to use GBK to encode their (library for example), then the user uses Unicode, this can cause issues. This is a valid concern as AFAIK, GBK is a superset of Unicode 1.1, and it should be supported somehow. Rendering these kinds of code on terminals which do not support past their designed and required character ranges (ASCII, GBK ...), can be an issue, as they cause random glyphs to occur in place of the actual characters. |
Raise two more questions. Suggest restriction levels as an alternative design. Describe the Go language identifier syntax.
In my experience it is quite rare to accidentially use non ASCII idents. Almost all developers are aware of the distinction between basic latin (ASCII) and other characters so they wont introduce them if they don't intend to do so. Typos containing Unicode characters are also unlikely as one usually needs to make the same mistake at least twice (definition and use).
I did not propose different lints based on how the recommended script list or enabling each script individually to limit implementation complexity and make it easier for each developer to use the scripts he/she wants to use. Limiting scripts is intended to improve security. As there is already a Unicode on/off switch the problem applies only to projects that want to use non select Latin scripts but are afraid that other scripts sneak in. Regarding GBK and other encodings. Rustc currently only accepts UTF-8 (and thus also ASCII-only) encoded files. Because of this an editor to write Rust code should be set to UTF-8 and files in GBK need to converted to UTF-8.
This must be very annoying. When I need to use Greek letters I open my Greek letter sheet in a separate window and copy the characters as needed which works reasonably well. On the other hand it did not seem to annoy you enough to look up a better way: 1 or 2 |
|
To disallow any Unicode identifiers in a project (for example to ease collaboration or for security reasons) limiting the accepted identifiers to ASCII add this lint to the `lib.rs` or `main.rs` file of your project: | ||
|
||
```rust | ||
#![forbid(unicode_idents)] |
SimonSapin
Jun 4, 2018
Contributor
To split hairs: this should be non_ascii_idents
.
Rust source files are always Unicode. The U+0061 to U+007A range is part of Unicode.
To split hairs: this should be non_ascii_idents
.
Rust source files are always Unicode. The U+0061 to U+007A range is part of Unicode.
Manishearth
Jun 4, 2018
Member
Came here to say this 😄
Came here to say this
|
||
For open source crates it is recommended to write them in English and use ASCII-only. An exception should be made if the application domain (e.g. math) benefits from Unicode and the target audience (e.g. for a crate interfacing with Russian passports) is comfortable with the used language and characters. Additionally crates should provide an ASCII-only API. | ||
|
||
Private projects can use any script and language the developer(s) desire. It is still a good idea (as with any language feature) not to overuse it. |
SimonSapin
Jun 4, 2018
Contributor
I would perhaps weaken the language in these two paragraphs, with phrases like “suggested” and “should consider”. English is indeed the de-facto international language and ASCII-only idents are indeed more friendly to an international audience (if only for typing), but it is not a Rust RFC’s place to judge which other concerns are or are not acceptable reasons to do otherwise, or how much use is overuse.
I would perhaps weaken the language in these two paragraphs, with phrases like “suggested” and “should consider”. English is indeed the de-facto international language and ASCII-only idents are indeed more friendly to an international audience (if only for typing), but it is not a Rust RFC’s place to judge which other concerns are or are not acceptable reasons to do otherwise, or how much use is overuse.
`XID_Start` and `XID_Continue` are used as defined in the aforementioned standard. The definition of identifiers is forward compatible with each successive release of Unicode as only appropriate new characters are added to the classes but none are removed. | ||
|
||
Two identifiers X, Y are considered to be equal if there [NFKC forms][TR15] are equal: NFKC(X) = NFKC(Y). |
SimonSapin
Jun 4, 2018
Contributor
s/there/their/ ?
s/there/their/ ?
But a PR sent to you could contain special characters without you noticing. Surely, the confusable check might catch it, but it might also have a bug. I am not generally against giving an option to allow unicode letters, but this danger still looms as long as the lint is default on. Another point: when I want a codebase to be in german, most times I don't, I don't think I'd want to allow non-german characters or non latin scripts. Here, narrowing the lint down to a specific script would be really useful. |
* Are crates with Unicode names allowed and can they be published to crates.io? | ||
* Are `unicode_idents` and `confusable_unicode_idents` good names? | ||
* Should [ZWNJ and ZWJ be allowed in identifiers][TR31Layout]? | ||
* Should *rustc* accept files in a different encoding than *UTF-8*? |
SimonSapin
Jun 4, 2018
Contributor
This one is easy: no. Why would it? UTF-8 by definition supports all of Unicode. Also, source files encodings seem off-topic for this RFC.
This one is easy: no. Why would it? UTF-8 by definition supports all of Unicode. Also, source files encodings seem off-topic for this RFC.
Sorry I’m splitting hairs again, but please replace “Unicode idents” with “Non-ASCII idents” in various places (PR title, RFC filename, lint name, etc.) Rust syntax is already based on Unicode. ASCII-range letters are part of Unicode. |
`XID_Start` and `XID_Continue` are used as defined in the aforementioned standard. The definition of identifiers is forward compatible with each successive release of Unicode as only appropriate new characters are added to the classes but none are removed. | ||
|
||
Two identifiers X, Y are considered to be equal if there [NFKC forms][TR15] are equal: NFKC(X) = NFKC(Y). |
SimonSapin
Jun 4, 2018
Contributor
I’d like this to go further and specify:
- Parsers for Rust syntax normalize idents to NFKC
- APIs such as
proc_macro::Ident::new
normalize to NFKC
- As a consequence, identifiers are considered equal if their NFKC forms are equal (module hygiene concerns, which are out of scope for this RFC), and APIs such as
proc_macro::Ident::to_string
return a normalized string.
I’d like this to go further and specify:
- Parsers for Rust syntax normalize idents to NFKC
- APIs such as
proc_macro::Ident::new
normalize to NFKC - As a consequence, identifiers are considered equal if their NFKC forms are equal (module hygiene concerns, which are out of scope for this RFC), and APIs such as
proc_macro::Ident::to_string
return a normalized string.
So, first of all, this seems well-thought-out; if we're going to allow Unicode identifiers, this seems like the right approach to do so. That said, I personally would prefer not to do this at all, and to require that identifiers remain ASCII. Also, as another alternative to include (in the alternatives section): we could require that any identifiers exported from a crate must always be ASCII, but identifiers that remain entirely within the same crate may be Unicode. EDIT: this comment is obsolete, please see my comments starting from #2457 (comment) instead, which discuss specific concerns and tradeoffs. |
Maybe I'm missing something, maybe this is super complicated as well, but I think that confusability checks as well as normalisation algorithms are far harder to implement than a simple lookup to which script a character belongs to. |
Could you expand a bit more on what that preference is, and why? My preference it to keep idents ASCII and based on English words, and keep comments in English, in projects that I create, maintain, contribute to, or use. (And English is not my native language.) But that doesn’t require an ASCII-only restriction to be baked into the language. It’s more of a social question than a technical one in my opinion.
The crate boundary is rather arbitrary here. A single system with a region-specific purpose could be split into multiple crates, for example to manage recompilation speed. |
@SimonSapin It's a tradeoff between "should people be able to write identifiers in languages that can't be represented in ASCII" versus "should people be able to read arbitrary code". Both of those are important, and I don't want to discount either, but I'd favor the latter. Another useful alternative that I think seems worth documenting: allowing unicode identifiers, but leaving them as |
@joshtriplett Yes this is an important trade off, but it is up to the author of a given piece of code to make that choice. It is not our place to dictate what language people should speak or write in contexts we can’t even think of. Even if it were, a character set restriction is a poor way to do it. |
@SimonSapin anything that would speak against a deny by default lint? |
Thanks to SimonSapin for the suggestions.
Done. Managed to fit them in bullet points anyway |
How do we handle update to the Unicode standard ? Can update to the Unicode standard break our backward compatibility ? Especially for the proposed lints? |
No -- lints don't have strong stability requirements and the XID properties are designed with stability in mind so relying on them should work. |
Please note that such homoglyph attacks might be constructed from string literals as well, given that certain behavior depends critically on a string literal. Therefore I would strongly recommend to provide a way to deny non-ASCII input altogether. Secondly, from a security point of view it might be most transparent to unrecoverably deny non-ASCII input for a whole crate or module subtree. This way, comfortably by changing a line, the compiler/lint can show you all places where non-ASCII input occurs, no second tool is needed. |
@JohnBSmith The RFC already provides for such a mechanism for identifiers (but not string literals -- which seems like a very different proposition and completely unrelated to this RFC). You can simply write |
So far, the RFC does not make clear, whether |
AFAIK all |
Only deny can, forbid prevents allowing. (https://doc.rust-lang.org/reference/attributes.html#lint-check-attributes) |
Oooooh, TIL. Thanks for the pointer to the reference, I only knew about |
@ubsan mentioned that (This is mostly an implementation detail, but might as well not change things we don't need to) |
@Manishearth I actually don't know that that's true, I just dislike differentiating between On another thing entirely: frig yes this RFC is friggin awesome. I'm so excited that this is happening :D |
So long as no observable behavior changes wrt. macro_rules! m { ($i:ident) => {} }
m!(_); // error: no rules expected the token `_` |
Yeah, I mentioned in the RFC that this is just to match current behavior. |
The final comment period, with a disposition to merge, as per the review above, is now complete. |
Huzzah! This RFC has been merged! Tracking issue: rust-lang/rust#55467 |
Mmmm... I would say, if we would allow non-ASCII identifiers, I didn't see a reason to not allow emojis as well. Emojis are today's international language, when being well designed they can be very well presented the intensions of the code. fn check_☎(☎_no: &str) -> Result<(),Error> {
} Of cause people will abuse it. But we can at least design a limited set of such symbols? |
We now do allow non-ASCII identifiers; it already landed. The identifiers we allow are most of the subset recommended by the Unicode Consortium, as detailed in many prior posts in this thread. Emoji are not among the consortium's recommendations. IIUC, any proposal to add them needs its own RFC. |
Allow non-ASCII letters (such as accented characters, Cyrillic, Greek, Kanji, etc.) in Rust identifiers.
Rendered
Tracking issue
It uses the feedback from rust-lang/rust#28979, #2455 and was updated several times based on the comments received in this PR.
The different options for identifier Unicode support can be summarized in this list, ordered from least complex to most complex:
feature(non_ascii_idents)
)Level 2 already provides good Unicode support for most uses. Normalizing identifiers and confusable detection improves security and usability.