Tracking issue for non-ASCII identifiers (feature "non_ascii_idents")

[DemiMarie]

Non-ASCII identifiers are currently feature gated. Handling of them should be fixed and the feature gate removed.

[steveklabnik]

/cc @rust-lang/lang

[pnkfelix]

nominating

[nrc]

cc @SimonSapin

Apparently we implement this: http://www.unicode.org/reports/tr31/ or something like it.

I would like to see this stabilised, but it will take some work to persuade ourselves that we are doing the right thing.

[SimonSapin]

I have no idea what the right thing is here. In addition to Unicode recommendations, we might want to look at what other languages actually do, and what related bug reports or criticism they get. Or was this already done when the feature was first introduced?

[petrochenkov]

@SimonSapin
C and C++ use http://unicode.org/reports/tr31/#Alternative_Identifier_Syntax (with some minor restrictions) and I haven't seen any complaints about it on isocpp forums or issue lists :)
Overview of the problem: http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1518.htm
Implementation in Clang: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Lex/UnicodeCharSets.h?view=markup
cc #4928

There's also a problem with normalization of identifiers and mapping unicode mod names to the filesystem names (on OS X, IIRC), but I can't find the relevant link here it is: #2253. (In the worst case non-inline mods and extern crates can be forced to be ASCII)

[pnkfelix]

Yes #2253 is the big issue I know of that makes me worry about premature stabilization of non-unicode identifiers.

(The discussion there is more broad and arguably could be forked off into two threads; e.g. we could take one normalization path for identifiers and another for string literal contents.)

[pnkfelix]

we may want to migrate This discussion to the RFCS repo, e.g. at rust-lang/rfcs#802

[bstrie]

I agree that this is a feature that deserves to be put through the RFC process.

[aturon]

I've repurposed this issue to track stabilization (or deprecation, etc) of the non_ascii_idents feature gate.

[nikomatsakis]

After discussion in the lang team meeting, we decided that yes, an RFC would be the proper way forward here. We need something that collects the solutions from other languages, analyzes their pros/cons, and suggests the appropriate choice for Rust. This is controversial and complex enough that it should be brought to the community at large -- especially as many of us hacking on Rust on a daily basis don't have a lot of experience with non-ASCII anyhow.

[nikomatsakis]

triage: P-low

Marking as low as there is no RFC at present and hence no actionable content.

[huonw]

cc #7539

[kberov]

In JavaScript, Perl 5 and Perl 6 this feature is available.
JavaScript (Firefox 50)

function Слово(стойност) {
  this.стойност = стойност;
}
var здрасти = new Слово("Здравей, свят");
console.log(здрасти.стойност) //Здравей, свят

Perl >=5.12

use utf8;
{
  package Слово;
  sub new {
    my $self = bless {}, shift;
    $self->{стойност} = shift;
    $self
  }
};
my $здрасти = Слово->new("здравей, свят");
say ucfirst($здрасти->{стойност}); #Здравей, свят

Perl6 (this is not just next version of Perl. This is a new language)

class Слово {
  has $.стойност;
}

my $здрасти = Слово.new(стойност => 'здравей, свят');
say $здрасти.tc; #Здравей, свят

I would be happy to see it in Rust too.

[SimonSapin]

For what it’s worth identifiers in ECMAScript 2015 are based on the Default Identifier Syntax from Unicode Standard Annex #31.

Perl with use utf8; uses the regexp below, with XID_Start and XID_Continue presumably also from UAX # 31.

/ (?[ ( \p{Word} & \p{XID_Start} ) + [_] ])
        (?[ ( \p{Word} & \p{XID_Continue} ) ]) *    /x

[kberov]

Yes! Thanks @SimonSapin!

[SimonSapin]

For Python it’s <XID_Start> <XID_Continue>*.

So it looks like many programming languages that allow non-ASCII identifiers are based on the same standard, but in the details they each do something slightly different…

[mjbshaw]

I would personally love to see support for math-related identifiers. For example, ∅ (and set operators, like ∩ and ∪). Translating equations from research papers/specifications into code is often a terrible process resulting in verbose and difficult to read code. Being able to use the same identifiers in the code that are in the paper's math equations would simplify implementation and would make the code easier to check and compare against the paper's equations.

[DoumanAsh]

What's point of this feature exactly? Aside from adding possibility to create truly ugly mix of different languages in your code(english is the only truly international language), it gives no benefits to language functionality wise. Or is it support of unicode for the sake of supporting unicode?

[SamWhited]

I don’t understand this point. XID_Start and XID_Continue are derived properties.

I might have misunderstood UAX 31 then; it looked to me like it required a specific Unicode version. Re-reading I can't see where I got that from though.

[ryankurte]

Not sure if this is the right place to post this, but some interesting issues are are likely to appear with linting of mathematical symbols. Easily avoided by writing out variable names, but could be important if better correlation with real equations is a goal.

For example, Δ (uppercase) vs. δ (lowercase) in the following screenshot. The linter is not /wrong/, but it also imo doesn't really make sense to apply the snake case requirement here.

[ghost]

would it be possible to allow emoji in variable names even though they aren't XID Start/Continue, like in Swift?

[behnam]

@fwrs, Emojis are way more complicated now than non-Emoji characters.

Thanks to some vendors, now you can have Emoji joining (ZWJ) sequences that just keep changing their colors and small details, many of which are not necessarily visible to the naked eye.

Also, the definition of Emoji is expanding fast, every single year, which is not something a system-level programming languages that wants to be stable and reliable needs.

So, although it's cute, I don't think it sits well with Rust goals. But, rust-based scripting/educational languages may benefit from allowing Emojis, depending on their goals.

[ketsuban]

@ryankurte There's a semantic problem in your example—you're transcribing mathematical formulae, but you used U+0394 GREEK CAPITAL LETTER DELTA rather than U+2206 INCREMENT. The former is a letter of the Greek alphabet, and as such has casemapping; the latter is a mathematical symbol and does not.

[steveklabnik]

I'd like to cross-link this comment: #4928 (comment)

[gnzlbg]

I haven't seen the possibility of enabling homoglyph-based attacks here (If somebody mentioned them please ignore the noise), but I just filled a clippy issue to request a lint that warns on code like this:

#![feature(non_ascii_idents)]
fn main() {
    let a = 2;
    let а = 3;
    assert_eq!(a, 2);  // OK
    assert_eq!(а, 3);  // OK
}

In a nutshell, those two as are different unicode characters so the second let binding does not shadow the first one, and both asserts pass (the playground doesn't seem to support unicode identifiers though so the only way to try this is locally; works for me).

This "feature" can be used to introduce exploits in Rust programs that are harder to detect, in particular given that shadowing let bindings are considered idiomatic Rust by many, myself included.

P.S.: this "feature" might be useful in underhanded Rust contests, although that #![feature(non_ascii_idents)] should raise some eyebrows :)

[ketsuban]

@gnzlbg I believe there's already some support for confusables detection to stop people swapping out your semicolons for Greek question marks and such, but I don't know if it applies to identifiers. If it does, then that solves that problem; if it doesn't, at least we have the tooling to do it ready to go.

I'm a little concerned that this is a candidate for being closed and the code removed from the compiler because it's not had significant movement for a while and requires an RFC. I care a fair amount about Rust being a language of the 21st century, which means Unicode, and about Rust being friendly to non-English-speaking programmers. What I lack is the ability to actually write an RFC.

[gnzlbg]

@ketsuban

I believe there's already some support for confusables detection to stop people swapping out your semicolons for Greek question marks and such, but I don't know if it applies to identifiers.

yes, I think that, as suggested by @oli-obk in the clippy issue, Rust implementation would instead just use the latest official confusable list:

http://www.unicode.org/Public/security/revision-06/confusables.txt

homoglyph-based attacks can be prevented. This list would need to be kept in sync though, but that is something that can be automated as part of the build system.

[gnzlbg]

@ketsuban

If you care about this, there are other languages that support unicode in their identifiers, and these languages have processes similar to the RFC process. You could start by checking those. Who knows, maybe you can just merge them together with the feedback in this issue, and get a pre-RFC in the internals forum going? From that point on, it is just about incorporating/arguing feedback with others, and before you know it you will have an RFC ready.

[mitsuhiko]

In a way I hope we stick with ASCII identifiers forever. Handling unicode identifiers is such a massive interoperability pain. Some of the more bizarre examples of NFKC mappings is that things like this map to the same identifier:

>>> ℌ = 1
>>> H
1
>>> Ⅸ = 42
>>> IX
42
>>> ℕ = 23
>>> N
23
>>> import math
>>> ℯ = math.e
>>> e
2.718281828459045
>>> ℨ = 2
>>> Z
2

[Serentty]

@mitsuhiko The real world has that kind of pain. We can't just ignore this problem because it's hard to deal with and involves a feature that you personally have no use for.

[Ixrec]

Also, the current RFC explicitly proposes NFC over NFKC, after a lot of discussion about examples very similar to those.

[Centril]

Closing in favor of #55467.