[JuliaLowering] Refactor scope resolution pass #60316

mlechu · 2025-12-04T20:24:05Z

Fixes

Soft scope

We currently treat flisp's '(block (softscope true)) as a scope that "is
soft", but this isn't correct. It should behave more like a toggle, where if the
scope surrounding (softscope true) is the top-level thunk, neutral scopes not
protected by a hard scope become permeable to already-defined globals. I've
added K"softscope" for this.

Fixes JuliaLang/JuliaLowering.jl#101.
JuliaLowering.activate!() should be much more usable in the REPL now, as
globals won't be accidentally eaten by the "soft scope."

Shadowing behaviour

Found while cleaning up the lhs-resolution step with the change above. This PR
allows static parameters to be shadowed by globals and locals as long as they're
explicit (and not in the same scope). flisp allows this with globals, and the
explicit locals that desugar to local-def forms (which JuliaLowering doesn't
have).

This change is more permissive than flisp in the local case, since after looking
into why shadowing was disallowed I realized it was just was just to prevent
assignment to the static parameter (#32623). The flisp fix leads to some funny
behaviour:

julia> function f(x::T) where T
     let
         global T # remove this T and the other two will fight
         let; local T; end
     end
     end
f (generic function with 1 method)

Bindings/LambdaBindings

In figuring out how to use the variable bookkeeping system, I ran into inaccuracies.

LambdaBindings is currently a per-lambda map from unique variable to four
flags: captured, read, assigned, and called. I think (but correct me if
I'm wrong @c42f) this was a misinterpretation of the holy text: in flisp
lowering, a local variable captured by some other lambda does show up in both
lambdas' variable lists, but is the same underlying object, and flag mutations
on one variable are seen by all lambdas.

I tried to think of other reasons for tracking vinfo per lambda within lowering,
but if we're doing something about the capture-boxing issue, we need something
more complex anyway.

This PR moves all vinfo to BindingInfo and deletes the incorrect bookkeeping
in LambdaBindings. We still need to have a per-lambda flag for capturedness
(different from the variable-level capt vinfo flag). With the added flags,
I've just made BindingInfo mutable since our previous workflow (BindingId is an
index into a vector; mutate the vector) doesn't give us the benefits of
immutability anyway.

Enhancements

Scopes are retained until the end of the pass, so consumers like JETLS can
answer questions like "what names are available at my cursor?" Recreating
this previously-discarded information was a bit hacky! [1] [2]
Hopefully enough tests and explanatory comments to make up for the large diff

TODO

Our use of K"local" in desugaring is dubious in some places
I've added expr_compat_mode to the scope analysis context, but we still need
to implement flisp hygiene exemptions for globals (see note in test/scopes.jl).
Reviewing the IR changes, the #self# argument still has extra flags set in
some cases. This is an existing desugaring bug with a comment that took me
too long to find: we shouldnn't be using the same #self# binding for
multiple methods defined by one function body

"get" is clearer to me that the binding is the output rather than the input, that it definitely exists, and that the ID resolves uniquely

I know it's a goal to make some of these clearer, but using the same string as Expr (and using the fall-through case in conversion) will make some planned code movement easier

topolarity

I still need to do another once-over of the core pass logic, but this should be useful (mostly very minor) feedback in the mean time.

topolarity · 2025-12-05T19:07:46Z

JuliaLowering/src/bindings.jl

+    binfo.is_assigned_once   && print(io, ", is_assigned_once")
+    binfo.is_captured        && print(io, ", is_captured")
+    binfo.is_always_defined  && print(io, ", is_always_defined")
+    binfo.is_used_undef      && print(io, ", is_used_undef")


Doesn't this no longer round-trip when pasting into, e.g., the REPL?

Maybe worth it for compactness, but would be nice to restore.

topolarity · 2025-12-09T19:30:31Z

JuliaLowering/test/scopes.jl

+    # errors like the conflict cases (two of the same decl should never do
+    # anything, and the user might be expecting two variables).
+    @testset "global,global" begin
+        s = "function (); global g; global g; 1; end"


JuliaLowering seems to accept this example, but flisp does not:

julia> JuliaLowering.include_string(TestMod,""" function (); global g; const g = 1; end """) #anon###0 (generic function with 1 method) julia> function (); global g; const g = 1; end ERROR: syntax: `global const` declaration not allowed inside function around REPL[5]:1 Stacktrace: [1] top-level scope @ REPL[5]:1

somehow global g gets JL to look the other way

topolarity · 2025-12-09T19:37:41Z

JuliaLowering/test/scopes.jl

+
+# For each distinct outer and inner scope, and each kind of variable in the
+# outer scope, set the same name to true from the inner scope
+@testset "Behaviour of `=` in local scope (shadow or assign-existing)" begin


I love exhaustive testing like this, even if it's a bit "shallow" w.r.t. the AST depth.

Really nice!

topolarity · 2025-12-09T19:59:23Z

JuliaLowering/src/bindings.jl

+    const id::IdTag                 # Unique integer identifying this binding
+    const name::String
+    const kind::Symbol              # :local :global :argument :static_parameter
+    const node_id::Int              # ID of associated K"BindingId" node in the syntax graph


For my own understanding, is node_id <-> id a 1:1 mapping?

topolarity · 2025-12-09T20:08:18Z

JuliaLowering/src/bindings.jl

    # Reconstruct the SyntaxTree for this binding. We keep only the node_id
    # here, because that's got a concrete type. Whereas if we stored SyntaxTree
    # that would contain the type of the graph used in the pass where the
    # bindings were created and we'd need to call reparent(), etc.


I'm confused by these comments.. this implementation seems very similar to reparent()

I know they're not yours, but can you explain where it's different?

topolarity · 2025-12-09T20:22:27Z

JuliaLowering/src/closure_conversion.jl


 # Compute fields for a closure type, one field for each captured variable.
 function closure_type_fields(ctx, srcref, closure_binds, is_opaque)
    capture_ids = Vector{IdTag}()


Another micro-optimization, but Vector{IdTag} should probably be BitSet
(since these are small and contiguous)

topolarity · 2025-12-09T20:29:00Z

JuliaLowering/src/scope_analysis.jl

+    # assigned to at top level, or passing the defined-and-owned-global check.
+    defined_toplevel_globals::Set{NameKey}
+    enable_soft_scopes::Bool
+    expr_compat_mode::Bool


This looks unused?

topolarity · 2025-12-09T20:46:31Z

JuliaLowering/src/scope_analysis.jl

 end

-function _resolve_scopes(ctx, ex::SyntaxTree)
+function _resolve_scopes(ctx, ex::SyntaxTree, @nospecialize(scope))


Suggested change

function _resolve_scopes(ctx, ex::SyntaxTree, @nospecialize(scope))

function _resolve_scopes(ctx, ex::SyntaxTree, @nospecialize(scope::Union{Nothing,ScopeInfo}))

IIUC you assume this, so best to mark it
(should also dramatically help inference, given the @nospecialize)

topolarity · 2025-12-09T20:48:08Z

JuliaLowering/src/scope_analysis.jl

            end
        elseif etype == "global_toplevel_only"
-            if !ctx.scope_stack[end].is_toplevel_global_scope
+            if !is_top_scope(scope)


Won't this throw a MethodError is scope isa Nothing?

topolarity · 2025-12-09T20:53:18Z

JuliaLowering/src/scope_analysis.jl

+parent(ctx, scope::ScopeInfo) = is_top_scope(scope) ? nothing :
+    ctx.scopes[scope.parent_id]
+
+enclosing_lambda(ctx, lb::LambdaBindings) = ctx.scopes[lb.scope_id]


This method is somewhat confusing, since it's a weird name if you come to this operation by saying "I'd like to get the ScopeInfo corresponding to the lambda."

Suggested change

enclosing_lambda(ctx, lb::LambdaBindings) = ctx.scopes[lb.scope_id]

I'd see whether enclosing_lambda(ctx, scope(ctx, lb)) (or similar) is not too awkward instead

topolarity · 2025-12-09T21:04:47Z

JuliaLowering/src/scope_analysis.jl

+    # Every lexical scope, indexed by ScopeId
+    scopes::Vector{ScopeInfo}
+    # Current stack of scopes to look for names in, innermost scope last
+    scope_stack::Vector{ScopeId}


Is this identical to the sequence of scopes you get if you follow the parent(ctx, scope) chain?

topolarity

Seems like the right approach (and indeed closer to flisp) to update shared bindings across all lambdas simultaneously.

Changes are looking good to me. Nice work @mlechu!

Feel free to take or drop the nits / ideas, as desired.

topolarity · 2025-12-09T21:17:18Z

JuliaLowering/src/scope_analysis.jl

-            throw(LoweringError(arg, "Unexpected lambda arg kind"))
+# If `b` is local and not yet recorded in the lambda bindings, mark it as
+# `capt`.  Also, (if `capt==true`), add it to any parent lambdas.
+function record_lambda_var!(ctx, scope::Union{ScopeInfo, LambdaBindings}, b, capt)


Suggested change

function record_lambda_var!(ctx, scope::Union{ScopeInfo, LambdaBindings}, b, capt)

function record_lambda_var!(ctx, scope::ScopeInfo, b; is_captured)

topolarity · 2025-12-09T21:20:15Z

JuliaLowering/src/scope_analysis.jl

    end
-    id
+    b = _new_binding(ctx, ex, nk.name, bk; mod, kws...)
+    home_scope.vars[nk] = b.id


Suggested change

home_scope.vars[nk] = b.id

declaration_scope.vars[nk] = b.id

topolarity · 2025-12-09T22:16:04Z

JuliaLowering/src/scope_analysis.jl

+    # Usually, globals in the top scope are ignored.  These are globals that may
+    # be assigned to without the `global` keyword in soft scopes due to being
+    # assigned to at top level, or passing the defined-and-owned-global check.
+    defined_toplevel_globals::Set{NameKey}


I actually do prefer the old implicit name or implicitly_scoped_globals? "top-level global" is a bit redundant, IIUC?

topolarity · 2025-12-09T22:18:36Z

JuliaLowering/src/scope_analysis.jl

+    elseif old_k === new_k
+        (new_k === :global || new_k === :local) && return bid
+        throw(LoweringError(ex, "function $(_var_str(new_k)) name not unique"))
+    # See note in test/scopes.jl


can you add some identifier to the note you mean?

topolarity · 2025-12-09T22:30:20Z

JuliaLowering/src/scope_analysis.jl

+        # mutable nameless bindings may be introduced in desugaring
+        maybe_declare_in_scope!(ctx, scope, SyntaxTree(ctx.graph, node_id),
+                                get_binding(ctx, bid).kind)
    end


It feels like it should be possible to unify this with the logic for scope.assignments below, since semantically they aren't any different iiuc?

Not a required re-work by any means, but it seems like it might be nice

mlechu requested review from c42f and topolarity December 4, 2025 20:24

mlechu added the compiler:lowering Syntax lowering (compiler front end, 2nd stage) label Dec 4, 2025

mlechu added 4 commits December 8, 2025 11:44

bikeshed: rename lookup_binding to get_binding

55c9b00

"get" is clearer to me that the binding is the output rather than the input, that it definitely exists, and that the ID resolves uniquely

Small fix: kw in function (tuple=1); end syntax

2755559

Refactor scope resolution pass

d818562

K"use_softscope_if_toplevel" -> K"softscope"

47eb8e9

I know it's a goal to make some of these clearer, but using the same string as Expr (and using the fall-through case in conversion) will make some planned code movement easier

mlechu force-pushed the jl-moved-scope-refactor branch from 912ff3c to 47eb8e9 Compare December 8, 2025 19:54

topolarity reviewed Dec 9, 2025

View reviewed changes

topolarity approved these changes Dec 9, 2025

View reviewed changes

	function _resolve_scopes(ctx, ex::SyntaxTree, @nospecialize(scope))
	function _resolve_scopes(ctx, ex::SyntaxTree, @nospecialize(scope::Union{Nothing,ScopeInfo}))

	function record_lambda_var!(ctx, scope::Union{ScopeInfo, LambdaBindings}, b, capt)
	function record_lambda_var!(ctx, scope::ScopeInfo, b; is_captured)

Uh oh!

[JuliaLowering] Refactor scope resolution pass #60316

Are you sure you want to change the base?

[JuliaLowering] Refactor scope resolution pass #60316

Conversation

mlechu commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Fixes

Soft scope

Shadowing behaviour

Bindings/LambdaBindings

Enhancements

TODO

Uh oh!

topolarity left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

topolarity Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

topolarity left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mlechu commented Dec 4, 2025 •

edited

Loading

topolarity Dec 9, 2025 •

edited

Loading