Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: spec: allow combining characters in identifiers #20706

Open
rsc opened this issue Jun 16, 2017 · 22 comments
Open

proposal: spec: allow combining characters in identifiers #20706

rsc opened this issue Jun 16, 2017 · 22 comments

Comments

@rsc
Copy link
Contributor

@rsc rsc commented Jun 16, 2017

Forking from #16033, which had two related but different proposals in it. The proposal for this issue, by @robpike:

On a related note, some writing systems - Devanagari is one (see #5167) require combining characters. The current identifier rules forbid combining characters; perhaps that should be relaxed, although that will require a canonicalization rule for combining characters. Unicode does have a definition for identifiers (http://unicode.org/reports/tr31/); perhaps Go should use it. Note that the addition of combining characters, allied with the export proposal above, would make it possible to export Devanagari identifiers.

@gopherbot gopherbot added this to the Proposal milestone Jun 16, 2017
@rsc
Copy link
Contributor Author

@rsc rsc commented Jun 16, 2017

Re canonicalization, one possibility Rob and I discussed at one point was to require in the spec that implementations canonicalize during comparisons to establish whether two identifiers are the same but also to have gofmt canonicalize to generate its output (the former is required for the latter to be semantically safe). Then source code is consistent but the compilers will deal if not.

@griesemer
Copy link
Contributor

@griesemer griesemer commented Jun 16, 2017

@rsc Is this Go 2 or would you consider this for Go 1?

@rsc
Copy link
Contributor Author

@rsc rsc commented Jun 17, 2017

For Go 2.

@rsc rsc added the Go2 label Jun 17, 2017
@rsc
Copy link
Contributor Author

@rsc rsc commented Jun 17, 2017

Merging #5167 in here. From suraj@barkale.com in 2013:

My suggestion is to amend Go specification by allowing combining-mark & non-spacing-mark characters in identifiers.

This will be similar to Java identifier rules given at http://docs.oracle.com/javase/1.4.2/docs/api/java/lang/Character.html#isJavaIdentifierPart(char).
A character may be part of a Java identifier if any of the following are true:

  • it is a letter
  • it is a currency symbol (such as '$')
  • it is a connecting punctuation character (such as '_')
  • it is a digit
  • it is a numeric letter (such as a Roman numeral character)
  • it is a combining mark
  • it is a non-spacing mark
  • isIdentifierIgnorable returns true for the character

@bakul
Copy link

@bakul bakul commented Jul 19, 2017

Note that the Java id rule link is invalid now.

The first char of an identifier is typically more constrained (e.g. no digit).

I propose that as far as Indic languages are concerned, an identifier may not start with one of various sign chars, digits, dependent vowels or "virama". An identifier may start with a currency sign or an "om" sign is allowed or an independent vowel (vowel letters) or a consonant.

@robpike
Copy link
Contributor

@robpike robpike commented Jul 19, 2017

@bakul It's precisely that kind of minute precision we'd like to avoid. The rule must be very simple to express, as it is now. We are seeking a new but also simple-to-express rule that admits more classes of identifier without requiring natural-language-specific detail.

@bakul
Copy link

@bakul bakul commented Jul 19, 2017

Would "१२३" (123 in devanagari) be considered a number or an identifier? If the latter, that would be strange!

If you don't want nitpicky rules, a simpler rule may be that the first char satisfies unicode.IsLetter() and the succeeding chars satisfy unicode.IsLetter() or IsDigit() or IsMark(). Plus whatever is needed for CJK.

@bakul
Copy link

@bakul bakul commented Jul 19, 2017

I should add: along with identifiers, numbers should also be expressible in other languages (but I expect that'll be even more unpopular!)

@bakul
Copy link

@bakul bakul commented Sep 19, 2018

In case it makes a difference, the Swift programming language allows legitimate Indic words as identifiers. From Swift Lexical Structure:

Identifiers begin with an uppercase or lowercase letter A through Z, an underscore (_), a noncombining alphanumeric Unicode character in the Basic Multilingual Plane, or a character outside the Basic Multilingual Plane that isn’t in a Private Use Area. After the first character, digits and combining Unicode characters are also allowed.

Most indian language words use combining chars so it would be good to fix this.

If these identifiers are not exportable, that is fine. Prefixing with an uppercase letter from some other script is ugly but that can't be helped! I am tempted to suggest using the section symbol (§) or some such symbol as an additional exportable start char of an identifier.

@bcmills
Copy link
Member

@bcmills bcmills commented Nov 29, 2018

that will require a canonicalization rule for combining characters.

See #27896 for a proposal specifically about canonicalization.

That potentially affects existing programs, since (for example) μ and µ are both already allowed (and treated as distinct identifiers) in Go source code.

@aarzilli
Copy link
Contributor

@aarzilli aarzilli commented Nov 29, 2018

Re canonicalization, one possibility Rob and I discussed at one point was to require in the spec that implementations canonicalize during comparisons to establish whether two identifiers are the same but also to have gofmt canonicalize to generate its output (the former is required for the latter to be semantically safe). Then source code is consistent but the compilers will deal if not.

I would prefer that non-normalized identifier were rejected so I wouldn't have to worry that my grep/editor/browser uses the same normalization strategy as gofmt/compiler when looking for identifiers even for sources that weren't visited by gofmt.

@mpvl
Copy link
Contributor

@mpvl mpvl commented Dec 7, 2018

@bcmills: forcing NFKC is not backwards compatible. If we break backwards compatibility, I would prefer to simply disallow any character with a decomposition type "font" and permit all others as is.

@bcmills
Copy link
Member

@bcmills bcmills commented Dec 7, 2018

If we break backwards compatibility, I would prefer to simply disallow any character with a decomposition type "font" and permit all others as is.

I think that would lead to a pretty unfortunate user experience. Consider the snippet:

	var jalapeño = "🌶️"
	fmt.Println(jalapeño)

Without any sort of normalization, the otherwise-equivalent identifiers jalapeño and jalapeño refer to two completely different variables, and as far as I am aware none of the characters involved have decomposition type font.

@griesemer
Copy link
Contributor

@griesemer griesemer commented Feb 5, 2019

In https://blog.golang.org/go2-here-we-come we propose to adopt a version of Unicode's TR-31 specification.

@gocs
Copy link

@gocs gocs commented May 6, 2019

w̶e̸ ̵m̵i̴g̵h̷t̸ ̷h̴a̸v̷e̴ ̴a̶ ̴p̶r̸o̷b̸l̷e̴m̷

do we allow zalgo text and alike here?

@bcmills
Copy link
Member

@bcmills bcmills commented May 6, 2019

@gocs, you can already write intentionally-obfuscated code today (for example, mixing Latin and Cyrillic vowels). Can you give specific examples where you believe the current restriction has prevented someone from writing an unreasonable identifier they would have chosen otherwise?

@eric-hawthorne
Copy link

@eric-hawthorne eric-hawthorne commented Jul 16, 2019

Arbitrary unicode identifiers in a programming language are a bad idea. By allowing arbitrary nonsensical or deliberately misleading mixings of characters from different natural languages, and also because of the non 1-1 mapping from glyph of the character to encoding, it will just allow programs to become more incomprehensible and unmaintainable, considered as a whole corpus of programs.
It could also discourage international free/open source code sharing.

Identifiers should stay ASCII + digits, maybe with a handful of greek letters thrown in as a concession to those wishing to express mathematics, engineering, and physics.

In this day and age, strings and comments in the language should be able to contain arbitrary unicode characters, but the code should be treated as standardized and constrained and simple formal expression of math and logic, in a small, simple alphabet.

The program expression philosophy especially for free/open source should be more like how international aviation has standardized on English communications, for safe interoperability.

Sometimes less is more. Remember the ghost of PL/1. Worst programming language in the known universe because it tried to be everything to everyone.

I know go from the get-go has permitted some unicode in identifiers, so this is water under the bridge.
But don't increase the torrent of incomprehensible flotsam and jetsam now.

@taralx
Copy link

@taralx taralx commented Aug 1, 2019

I disagree that gofmt should canonicalize. What I think it should do is unify - change all uses of an identifier to match the character sequence used at the declaration.

The only case that is a bit odd then is what to do about shadowing declarations - do we unify those too, or let them be unrelated?

@bcmills
Copy link
Member

@bcmills bcmills commented Aug 2, 2019

@taralx, the problem with unifying to the declaration site is that it is hostile to byte-oriented searches. If I declare a method AñadirJalapeños, and you declare a method AñadirJalapeños, then either they're the same method (and can both be used as the same interface) but with different representations in the source code, or they're different methods but due to encoding only.

Neither of those options seems particularly better than canonicalizing the source.

@taralx
Copy link

@taralx taralx commented Aug 2, 2019

Okay, so interfaces make that more difficult. My concern is that many canonicalization schemes result in aggressive switching from one code block to another (e.g. reducing superscript numbers to regular numbers or full-width symbols to half-width symbols). If the plan is to use only mild canonicalization that does not do this, then this will not be an issue for me, at least.

@bakul
Copy link

@bakul bakul commented Oct 28, 2020

What is the current thinking on this? On re-reading this thread, the first comment here by Russ captures exactly what I want (vis-a-vis Indic languages). What is the issue with simply following Unicode TR31? It talks about normalization (not canonicalization) in section 5. What specific issues does Go run into beyond what is discussed there? Since many languages do not have upper/lower case distinctions you'd need something to indicate either what is a public or private ID. But beyond that I don't see anything special in Go. Note that Python, Swift, Nim & at least one implementation of Scheme are following it fine. I can always switch to one of these languages for some Sanskrit related work I am doing but it would be really nice if Go2 supported such identifiers!

@ianlancetaylor ianlancetaylor added this to Incoming in Proposals Oct 28, 2020
@ianlancetaylor ianlancetaylor removed this from Incoming in Proposals Oct 28, 2020
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 28, 2020

As far as I know Unicode TR31 is still the current thinking. But it will take some work to get there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet