add Unicode.julia_chartransform Julia-parser normalization #42561
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
stevengj
added
domain:unicode
This was referenced Oct 8, 2021
|
Cool. Though is "charmap" the best term for this? It's often used to refer to the full list of all possible characters. Maybe something about "mapping" or "transformation" would avoid confusion? |
|
Thank you for working on this. |
stevengj
force-pushed
the
sgj/julia_charmap
branch
from
109d55e
to
f07680f
Compare
|
@nalimilan, renamed |
nalimilan
reviewed
Oct 9, 2021
stevengj
changed the title
nalimilan
reviewed
Oct 10, 2021
|
CI failures seem unrelated (it's been a long time since I've seen a green CI!), so this PR should be ready for review. |
Co-authored-by: Milan Bouchet-Valat <nalimilan@club.fr>
Co-authored-by: Milan Bouchet-Valat <nalimilan@club.fr>
Co-authored-by: Milan Bouchet-Valat <nalimilan@club.fr>
Co-authored-by: Milan Bouchet-Valat <nalimilan@club.fr>
stevengj
force-pushed
the
sgj/julia_charmap
branch
from
0dbc3fb
to
7601943
Compare
vtjnash
approved these changes
Oct 15, 2021
| @@ -4,6 +4,50 @@ module Unicode | |||
|
|
|||
| export graphemes, isequal_normalized | |||
|
|
|||
| """ | |||
| Unicode.julia_chartransform(c::Union{Char,Integer}) | |||
There was a problem hiding this comment.
Thoughts about naming this something like parsertransform?
There was a problem hiding this comment.
BTW, do you have an opinion on my last comment above? #42561 (comment)
There was a problem hiding this comment.
I think the callback function is good
There was a problem hiding this comment.
I don't have strong feelings on the name…
vtjnash
reviewed
Oct 15, 2021
Co-authored-by: Jameson Nash <vtjnash@gmail.com>
Co-authored-by: Jameson Nash <vtjnash@gmail.com>
Co-authored-by: Jameson Nash <vtjnash@gmail.com>
stevengj
commented
Oct 15, 2021
stevengj
commented
Oct 15, 2021
c42f
added a commit
to JuliaLang/JuliaSyntax.jl
that referenced
this pull request
Jan 21, 2022
* Add Unicode NFC normalization of all identifiers * Add Julia-specific normalizations of lookalikes of - ⋅ ε μ Based on the same functionality from the Unicode stdlib in Julia 1.8 JuliaLang/julia#42561
LilithHafner
pushed a commit
to LilithHafner/julia
that referenced
this pull request
Feb 22, 2022
LilithHafner
pushed a commit
to LilithHafner/julia
that referenced
this pull request
Mar 8, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Finally addresses a request by @Keno in #19464 (comment): provides a
Unicode.julia_chartransformfunction exporting the custom character normalization used by the Julia parser, which can be used as achartransformkeyword toUnicode.normalize.(More generally, the user can pass any custom codepoint mapping to
normalizevia thechartransformkeyword.)This should be useful for any package that needs to reproduce the normalization performed by the Julia parser: they can now do
Unicode.normalize(string, compose=true, stable=true, chartransformUnicode.julia_chartransform).