Definition identifiers conforming to Unicode Standard Annex #31?

[joakim]

Programming languages typically do this for identifiers (Rust for example).

CommonMark only specifies some restrictions and refers to Unicode case folding when normalizing reference links.

I see two alternatives:

1. Do like CommonMark
   • require at least one non-whitespace character
   • disallow starting with ^, ? and -
   • otherwise allow any characters except [ and ]
2. Do like programming languages
   • conform to UAX31 with a profile allowing for whitespace and syntax characters except [ and ] in the middle

I don't know what the benefits/drawbacks are for either of these. Input would be appreciated.

Here's a stab at UAX31 conformity:

Identifiers must conform to the Unicode Standard Annex #31 for Unicode Version 15.0.0 (UAX31-C1), observing the following requirements (UAX31-C2):

• UAX31-R1: Using the following profile of UAX31-R1-2:
  • Start := XID_Start
  • Continue := XID_Continue + Pattern_Syntax - U+005B - U+005D
  • Medial := Pattern_White_Space
• UAX31-R1b: Stable identifiers across versions of the Unicode Standard
• UAX31-R5: Equivalent case-insensitive identifiers with full case folding

[joakim]

One benefit of the UAX31 profile is that it restricts the first character (Start) to not belonging to Pattern_Syntax. That set includes ^ (note), ? (abbreviation) and - (comment), which must be reserved anyway, but also many other characters for possible future extensions to the definition syntax.

On the other hand, it disallows some common characters like $, #, +, -, =, /, @ that one might want to use at the start of an identifier.

[joakim]

On second thought, nope. Unicode just complicates things.

A compromise

Reserve non-alphanumeric printable ASCII characters for future use by disallowing as the first character of identifiers.

In other words, ASCII codes 33-47, 58-64, 91, 93-96 and 123-126:

!"#$%&'()*+,-/:;<=>?@[]^_`{|}~

Note that \ (92) is not reserved. That means one could still start an identifier with a reserved character by simply escaping it.

[\- not a comment -]: a snippet

This is [\- not a comment -]

Best of both worlds?