Global variable scope rules lead to unintuitive behavior at the REPL/notebook

[IainNZ]

Example 1

This came up with a student who upgraded from 0.6 to 1.0 directly, so never even got a chance to see a deprecation warning, let alone find an explanation for new behavior:

julia> beforefor = true
true

julia> for i in 1:2
         beforefor = false
       end

julia> beforefor  # this is surprising bit
true

julia> beforeif = true
true

julia> if 1 == 1
         beforeif = false
       end
false

julia> beforeif  # Another surprise!
false

julia> function foo()
         infunc = true
         for i in 1:10
           infunc = false
         end
         @show infunc
       end
foo (generic function with 1 method)

julia> foo()  # "I don't get this"
infunc = false 

Example 2

julia> total_lines = 0
0

julia> list_of_files = ["a", "b", "c"]
3-element Array{String,1}:
 "a"
 "b"
 "c"

julia> for file in list_of_files
         # fake read file
         lines_in_file = 5
         total_lines += lines_in_file
       end
ERROR: UndefVarError: total_lines not defined
Stacktrace:
 [1] top-level scope at ./REPL[3]:4 [inlined]
 [2] top-level scope at ./none:0

julia> total_lines  # This crushs the students willingness to learn
0

I "get" why this happens in the sense that I think I can explain, with sufficient reference to the arcana in the manual about what introduces scopes and what doesn't, but I think that this is problematic for interactive use.

In example one, you get a silent failure. In example two, you get an error message that is very there-is-no-spoon. Thats roughly comparable to some Python code I wrote in a notebook at work today.

I'm not sure what the rules are in Python, but I do know that generally you can't assign to things at the global scope without invoking global. But at the REPL it does work, presumably because at the REPL the rules are different or the same logic as if they were all are in the scope of function is applied.

I can't language-lawyer the rules enough to propose the concrete change I would like, and based on Slack this isn't even necessarily perceived as an issue by some people, so I don't know where to go with this except to flag it.

Cross-refs:
#19324
https://discourse.julialang.org/t/repl-and-for-loops-scope-behavior-change/13514
https://stackoverflow.com/questions/51930537/scope-of-variables-in-julia

[IainNZ]

(Per @mlubin, this is the relevant change #19324)

[jekbradbury]

Stefan suggested here that one possibility to solve this issue is automatic wrapping of REPL entries in let blocks

[KristofferC]

But wouldn't that be confusing in that you couldn't do

a = 1

and use a after that? Unless global is inserted for all the toplevel assignments, I guess?

[StefanKarpinski]

The behavior wouldn't be just to wrap everything in a let block—it's more complicated than that. You need to let-bind any global that's assigned inside the expression and then extract the let-bound value to a global at the end of the expression.

[StefanKarpinski]

So you would turn a = 1 into something like a = let a; a = 1; end. And something like

for i in 1:2
    before = false
end

would be turned into this:

before = let before = before
    for i in 1:2
        before = false
    end
end

Frankly, I'm pretty annoyed that people are only giving this feedback now. This has change has been on master for ten months.

[piever]

I'm guilty of not having followed master very closed until recently, so this feedback is indeed a bit late. More than a concern for programmers (most for loops will be inside a function in library code) I'm afraid this is a concern for teaching. Often for loops are taught before functions or scopes (of course you need to understand scopes to really understand what's going on but in teaching things are often simplified).

Here it becomes a bit difficult to teach a beginner how to sum numbers from 1 to 10 without explaining functions or global variables.

[mlubin]

Frankly, I'm pretty annoyed that people are only giving this feedback now. This has change has been on master for ten months.

To be fair, Julia 0.7 was released 13 days ago. This is a new change for most Julia users.

[IainNZ]

Frankly, I'm pretty annoyed that people are only giving this feedback now. This has change has been on master for ten months

Unfortunately for those of us who can not handle living on the edge, its brand-new from our perspective.

[rickhg12hs]

Frankly, I'm pretty annoyed that people are only giving this feedback now. This has change has been on master for ten months.

And for those of us who have been encouraged to stay off the development branches, "it's brand-new from our perspective."

[KristofferC]

Can we please go back to focus on the issue at hand now, instead of having a meta discussion about how long people have had to test this. It is what it is right now, so let's look forward.

[ChrisRackauckas]

I'm guilty of not having followed master very closed until recently, so this feedback is indeed a bit late. More than a concern for programmers (most for loops will be inside a function in library code) I'm afraid this is a concern for teaching. Often for loops are taught before functions or scopes (of course you need to understand scopes to really understand what's going on but in teaching things are often simplified).

Here it becomes a bit difficult to teach a beginner how to sum numbers from 1 to 10 without explaining functions or global variables.

This is a big point. After finding out what the issue really is, it's surprising how little it actually shows up. It is less of an issue with a lot of Julia code in the wild and in tests, and it did reveal a lot of variables which were accidentally global (in both Julia Base's tests according to the original PR, and I noticed this on most of DiffEq's tests). In most cases it seems that the subtly wrong behavior isn't what you get (expecting a change in a loop), but rather expecting to be able to use a variable in a loop is what I've found to be the vast majority of where this shows up in updating test scripts to v1.0. So the good thing is that in most cases the user is presented with an error, and it's not difficult to fix.

The bad thing is that it's a little verbose to have to put global x inside of the loops, and now your REPL code is also different from the function code. Whether or not it's more intuitive behavior than before is a tough opinion because there were definitely some edge cases in hard/soft local scoping and so this is clearly easier to explain. But at the same time, while having a much more succinct explanation than the behavior of before, it's now easier to hit the edge cases where understanding scoping rules matters. 🤷‍♂️.

I for one would like to see the experiments with let blocking. This would keep the "you didn't really want so many globals" aspect of it, along with the simplified scoping explanation, while at the same time make REPL code behave like function interiors (which is seemingly what we've always wanted). Or inversely, making people specify variables they want to act as globals

global x = 5
for i = 1:5
  println(x+i)
end

could be a nice way to keep the explicitness, and would make the "REPL code is slow because of globals" be much more obvious. The downside is that once again throwing things into a function would not require the global markers.

But given how this tends to show up, it's not really gamebreaking or a showstopper. I'd classify it as a wart that should get a mention in any workshop but it's not like v1.0 is unusable because of it. I hope that changing this behavior isn't classified as breaking and require v2.0 though.

[ExpandingMan]

I'm not so sure I like the idea that the REPL should behave like a function interior. It clearly isn't, so I expect it to behave like global scope. To me the REPL not behaving like global scope would be potentially even more confusing than the discrepency that causes this issue.

Regardless, at the very least I think that the documentation should be somewhat more explicit about this issue. Casually reading the docs I would have assumed that you would need to use the local keyword to get the behavior occurs in global scope by default.

[piever]

I for one would like to see the experiments with let blocking. This would keep the "you didn't really want so many globals" aspect of it, along with the simplified scoping explanation, while at the same time make REPL code behave like function interiors (which is seemingly what we've always wanted)

If we're going for "REPL is the same as the inside of a function" we should also think about outer:

julia> i = 1
1

julia> for outer i = 1:10
       end
ERROR: syntax: no outer variable declaration exists for "for outer"

versus:

julia> function f()
          i = 0
          for outer i = 1:10
          end
          return i
       end
f (generic function with 1 method)

julia> f()
10

[stevengj]

Frankly, I'm pretty annoyed that people are only giving this feedback now. This has change has been on master for ten months.

People haven't been using master for interactive use or for teaching, they've been using it to upgrade packages, which are only minimally affected by this and are mostly written by experienced programmers.

(I was one of the few people who did give feedback in #19324, though, where I argued for the old behavior.)

A non-breaking way out of this would be to change back to the old behavior (ideally not by inserting implicit let blocks or anything — just restore the old code in julia-syntax.scm as an option) in the REPL. Or rather, to make it available in environments like IJulia that might want it, add a soft_global_scope=false flag to include, include_string, and Core.eval to restore the old behavior.

[StefanKarpinski]

(I was one of the few people who did give feedback in #19324, though, where I argued for the old behavior.)

Yes, and I greatly appreciate it. It doesn't much matter now since we made the choice, let it bake for ten months and have now released it with a long-term commitment to stability. So the only thing to do now is to focus on what to do going forward.

Having an option to choose between the old behavior and the new one is interesting but it feels very hacky. That means we not only sometimes have a scoping behavior that everyone apparently found incredibly confusing, but we don't always have it and whether we have it or not depends on a global flag. That feels pretty unsatisfactory, I'm afraid.