Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

eliminate dependency on Base.is_id_char and julia-specific utf8proc #377

Open
stevengj opened this issue Nov 1, 2023 · 2 comments · May be fixed by #381
Open

eliminate dependency on Base.is_id_char and julia-specific utf8proc #377

stevengj opened this issue Nov 1, 2023 · 2 comments · May be fixed by #381

Comments

@stevengj
Copy link
Member

stevengj commented Nov 1, 2023

As discussed in #372, in order to make the JuliaSyntax parsing independent of the version of Julia in which it runs, it needs to:

  1. Add a dependency on utf8proc_jll, rather than using the utf8proc bundled with Julia. This is important because the supported version of Unicode (for character classification) depends on utf8proc.
  2. Re-implement Base.is_id_start_char and Base.is_id_char to identify which characters can appear in identifiers (or technically, which characters can begin graphemes, now that handle ZWJ and emoji sequences, don't break identifiers within graphemes #372 is merged … this could simplify the rules slightly since we no longer need to explicitly check for combining marks).
@c42f
Copy link
Member

c42f commented Nov 1, 2023

Thanks for outlining what needs to be done here :)

Just wondering - do you think you'll have bandwidth to work on this yourself or should I or someone else take it on? I feel like tackling it relatively soon because I'd like to be able to bump JuliaSyntax version which is used in Base, even on Julia-1.10.x without introducing new features. And I belated realized that #372 breaks the ability to do that.

@stevengj
Copy link
Member Author

stevengj commented Nov 4, 2023

I'm working on it.

@stevengj stevengj linked a pull request Nov 5, 2023 that will close this issue
KristofferC pushed a commit to JuliaLang/julia that referenced this issue Nov 13, 2023
This fixes a whole bunch of small but annoying bugs, as described in the
JuliaSyntax-0.4.7 release notes

https://github.com/JuliaLang/JuliaSyntax.jl/releases/tag/v0.4.7

I've been careful about cutting the JuliaSyntax-0.4.7 release from
nonbreaking changes, so we should be able to backport this to 1.10.

---

Extended notes about compatibility
* The public keyword in
JuliaLang/JuliaSyntax.jl#320 is released in
JuliaSyntax-0.4.7 but JuliaSyntax is multi-version aware so this is
disabled when used as the default parser in Julia 1.10, but is enabled
in 1.11-DEV. So should be backportable.
* We aim for parsing to `Expr` to always be stable in JuliaSyntax and
independent of the host Julia `VERSION`, but we're not fully there yet
for 1.11 / 1.10 due to
JuliaLang/JuliaSyntax.jl#377. Thus some
careful management of the JuliaSyntax-0.4.x branch for now.
KristofferC pushed a commit to JuliaLang/julia that referenced this issue Nov 13, 2023
This fixes a whole bunch of small but annoying bugs, as described in the
JuliaSyntax-0.4.7 release notes

https://github.com/JuliaLang/JuliaSyntax.jl/releases/tag/v0.4.7

I've been careful about cutting the JuliaSyntax-0.4.7 release from
nonbreaking changes, so we should be able to backport this to 1.10.

---

Extended notes about compatibility
* The public keyword in
JuliaLang/JuliaSyntax.jl#320 is released in
JuliaSyntax-0.4.7 but JuliaSyntax is multi-version aware so this is
disabled when used as the default parser in Julia 1.10, but is enabled
in 1.11-DEV. So should be backportable.
* We aim for parsing to `Expr` to always be stable in JuliaSyntax and
independent of the host Julia `VERSION`, but we're not fully there yet
for 1.11 / 1.10 due to
JuliaLang/JuliaSyntax.jl#377. Thus some
careful management of the JuliaSyntax-0.4.x branch for now.

(cherry picked from commit 85d7cca)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants