New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Global variable scope rules lead to unintuitive behavior at the REPL/notebook #28789

Open
IainNZ opened this Issue Aug 21, 2018 · 98 comments

Comments

Projects
None yet
@IainNZ
Member

IainNZ commented Aug 21, 2018

Example 1

This came up with a student who upgraded from 0.6 to 1.0 directly, so never even got a chance to see a deprecation warning, let alone find an explanation for new behavior:

julia> beforefor = true
true

julia> for i in 1:2
         beforefor = false
       end

julia> beforefor  # this is surprising bit
true

julia> beforeif = true
true

julia> if 1 == 1
         beforeif = false
       end
false

julia> beforeif  # Another surprise!
false

julia> function foo()
         infunc = true
         for i in 1:10
           infunc = false
         end
         @show infunc
       end
foo (generic function with 1 method)

julia> foo()  # "I don't get this"
infunc = false 

Example 2

julia> total_lines = 0
0

julia> list_of_files = ["a", "b", "c"]
3-element Array{String,1}:
 "a"
 "b"
 "c"

julia> for file in list_of_files
         # fake read file
         lines_in_file = 5
         total_lines += lines_in_file
       end
ERROR: UndefVarError: total_lines not defined
Stacktrace:
 [1] top-level scope at ./REPL[3]:4 [inlined]
 [2] top-level scope at ./none:0

julia> total_lines  # This crushs the students willingness to learn
0

I "get" why this happens in the sense that I think I can explain, with sufficient reference to the arcana in the manual about what introduces scopes and what doesn't, but I think that this is problematic for interactive use.

In example one, you get a silent failure. In example two, you get an error message that is very there-is-no-spoon. Thats roughly comparable to some Python code I wrote in a notebook at work today.

I'm not sure what the rules are in Python, but I do know that generally you can't assign to things at the global scope without invoking global. But at the REPL it does work, presumably because at the REPL the rules are different or the same logic as if they were all are in the scope of function is applied.

I can't language-lawyer the rules enough to propose the concrete change I would like, and based on Slack this isn't even necessarily perceived as an issue by some people, so I don't know where to go with this except to flag it.

Cross-refs:
#19324
https://discourse.julialang.org/t/repl-and-for-loops-scope-behavior-change/13514
https://stackoverflow.com/questions/51930537/scope-of-variables-in-julia

@IainNZ

This comment has been minimized.

Member

IainNZ commented Aug 21, 2018

(Per @mlubin, this is the relevant change #19324)

@jekbradbury

This comment has been minimized.

Contributor

jekbradbury commented Aug 21, 2018

Stefan suggested here that one possibility to solve this issue is automatic wrapping of REPL entries in let blocks

@KristofferC

This comment has been minimized.

Contributor

KristofferC commented Aug 21, 2018

But wouldn't that be confusing in that you couldn't do

a = 1

and use a after that? Unless global is inserted for all the toplevel assignments, I guess?

@StefanKarpinski

This comment has been minimized.

Member

StefanKarpinski commented Aug 21, 2018

The behavior wouldn't be just to wrap everything in a let block—it's more complicated than that. You need to let-bind any global that's assigned inside the expression and then extract the let-bound value to a global at the end of the expression.

@StefanKarpinski

This comment has been minimized.

Member

StefanKarpinski commented Aug 21, 2018

So you would turn a = 1 into something like a = let a; a = 1; end. And something like

for i in 1:2
    before = false
end

would be turned into this:

before = let before = before
    for i in 1:2
        before = false
    end
end

Frankly, I'm pretty annoyed that people are only giving this feedback now. This has change has been on master for ten months.

@piever

This comment has been minimized.

piever commented Aug 21, 2018

I'm guilty of not having followed master very closed until recently, so this feedback is indeed a bit late. More than a concern for programmers (most for loops will be inside a function in library code) I'm afraid this is a concern for teaching. Often for loops are taught before functions or scopes (of course you need to understand scopes to really understand what's going on but in teaching things are often simplified).

Here it becomes a bit difficult to teach a beginner how to sum numbers from 1 to 10 without explaining functions or global variables.

@mlubin

This comment has been minimized.

Member

mlubin commented Aug 21, 2018

Frankly, I'm pretty annoyed that people are only giving this feedback now. This has change has been on master for ten months.

To be fair, Julia 0.7 was released 13 days ago. This is a new change for most Julia users.

@IainNZ

This comment has been minimized.

Member

IainNZ commented Aug 21, 2018

Frankly, I'm pretty annoyed that people are only giving this feedback now. This has change has been on master for ten months

Unfortunately for those of us who can not handle living on the edge, its brand-new from our perspective.

@rickhg12hs

This comment has been minimized.

Contributor

rickhg12hs commented Aug 21, 2018

Frankly, I'm pretty annoyed that people are only giving this feedback now. This has change has been on master for ten months.

And for those of us who have been encouraged to stay off the development branches, "it's brand-new from our perspective."

@KristofferC

This comment has been minimized.

Contributor

KristofferC commented Aug 21, 2018

Can we please go back to focus on the issue at hand now, instead of having a meta discussion about how long people have had to test this. It is what it is right now, so let's look forward.

@ChrisRackauckas

This comment has been minimized.

Contributor

ChrisRackauckas commented Aug 21, 2018

I'm guilty of not having followed master very closed until recently, so this feedback is indeed a bit late. More than a concern for programmers (most for loops will be inside a function in library code) I'm afraid this is a concern for teaching. Often for loops are taught before functions or scopes (of course you need to understand scopes to really understand what's going on but in teaching things are often simplified).

Here it becomes a bit difficult to teach a beginner how to sum numbers from 1 to 10 without explaining functions or global variables.

This is a big point. After finding out what the issue really is, it's surprising how little it actually shows up. It is less of an issue with a lot of Julia code in the wild and in tests, and it did reveal a lot of variables which were accidentally global (in both Julia Base's tests according to the original PR, and I noticed this on most of DiffEq's tests). In most cases it seems that the subtly wrong behavior isn't what you get (expecting a change in a loop), but rather expecting to be able to use a variable in a loop is what I've found to be the vast majority of where this shows up in updating test scripts to v1.0. So the good thing is that in most cases the user is presented with an error, and it's not difficult to fix.

The bad thing is that it's a little verbose to have to put global x inside of the loops, and now your REPL code is also different from the function code. Whether or not it's more intuitive behavior than before is a tough opinion because there were definitely some edge cases in hard/soft local scoping and so this is clearly easier to explain. But at the same time, while having a much more succinct explanation than the behavior of before, it's now easier to hit the edge cases where understanding scoping rules matters. 🤷‍♂️.

I for one would like to see the experiments with let blocking. This would keep the "you didn't really want so many globals" aspect of it, along with the simplified scoping explanation, while at the same time make REPL code behave like function interiors (which is seemingly what we've always wanted). Or inversely, making people specify variables they want to act as globals

global x = 5
for i = 1:5
  println(x+i)
end

could be a nice way to keep the explicitness, and would make the "REPL code is slow because of globals" be much more obvious. The downside is that once again throwing things into a function would not require the global markers.

But given how this tends to show up, it's not really gamebreaking or a showstopper. I'd classify it as a wart that should get a mention in any workshop but it's not like v1.0 is unusable because of it. I hope that changing this behavior isn't classified as breaking and require v2.0 though.

@ExpandingMan

This comment has been minimized.

Contributor

ExpandingMan commented Aug 21, 2018

I'm not so sure I like the idea that the REPL should behave like a function interior. It clearly isn't, so I expect it to behave like global scope. To me the REPL not behaving like global scope would be potentially even more confusing than the discrepency that causes this issue.

Regardless, at the very least I think that the documentation should be somewhat more explicit about this issue. Casually reading the docs I would have assumed that you would need to use the local keyword to get the behavior occurs in global scope by default.

@piever

This comment has been minimized.

piever commented Aug 21, 2018

I for one would like to see the experiments with let blocking. This would keep the "you didn't really want so many globals" aspect of it, along with the simplified scoping explanation, while at the same time make REPL code behave like function interiors (which is seemingly what we've always wanted)

If we're going for "REPL is the same as the inside of a function" we should also think about outer:

julia> i = 1
1

julia> for outer i = 1:10
       end
ERROR: syntax: no outer variable declaration exists for "for outer"

versus:

julia> function f()
          i = 0
          for outer i = 1:10
          end
          return i
       end
f (generic function with 1 method)

julia> f()
10
@stevengj

This comment has been minimized.

Member

stevengj commented Aug 21, 2018

Frankly, I'm pretty annoyed that people are only giving this feedback now. This has change has been on master for ten months.

People haven't been using master for interactive use or for teaching, they've been using it to upgrade packages, which are only minimally affected by this and are mostly written by experienced programmers.

(I was one of the few people who did give feedback in #19324, though, where I argued for the old behavior.)

A non-breaking way out of this would be to change back to the old behavior (ideally not by inserting implicit let blocks or anything — just restore the old code in julia-syntax.scm as an option) in the REPL. Or rather, to make it available in environments like IJulia that might want it, add a soft_global_scope=false flag to include, include_string, and Core.eval to restore the old behavior.

@StefanKarpinski

This comment has been minimized.

Member

StefanKarpinski commented Aug 21, 2018

(I was one of the few people who did give feedback in #19324, though, where I argued for the old behavior.)

Yes, and I greatly appreciate it. It doesn't much matter now since we made the choice, let it bake for ten months and have now released it with a long-term commitment to stability. So the only thing to do now is to focus on what to do going forward.

Having an option to choose between the old behavior and the new one is interesting but it feels very hacky. That means we not only sometimes have a scoping behavior that everyone apparently found incredibly confusing, but we don't always have it and whether we have it or not depends on a global flag. That feels pretty unsatisfactory, I'm afraid.

@stevengj

This comment has been minimized.

Member

stevengj commented Aug 21, 2018

Having an option to choose between the old behavior and the new one is interesting but it feels very hacky.

If someone implements an "unbreak me" soft-scope AST transformation, it will be very tempting to use it in IJulia, OhMyREPL, etcetera, at which point you get the even more problematic situation in which the default REPL is seen as broken.

@StefanKarpinski

This comment has been minimized.

Member

StefanKarpinski commented Aug 21, 2018

That's not what I'm saying. Clearly we should use the same solution in all those contexts. But implementing it as two different variations on scoping rules seems less clean than implementing it as a code transformation with one set of scoping rules. But perhaps those are functionally equivalent. However, it seems easier to explain in terms of the new simpler scoping rules + a transformation that takes REPL-style input and transforms it before evaluating it.

@StefanKarpinski

This comment has been minimized.

Member

StefanKarpinski commented Aug 21, 2018

That could be done as Meta.globalize(m::Module, expr::Expr) that transforms an expression by automatically annotating any globals which exist in the module as global if they are assigned inside of any top-level non-function scope. Of course, I think that's equivalent to what the old parser did, but a bit more transparent since you can call Meta.globalize yourself and see what the REPL will evaluate.

@stevengj

This comment has been minimized.

Member

stevengj commented Aug 21, 2018

That could be done as Meta.globalize(m::Module, expr::Expr) that transforms an expression by automatically annotating any globals which exist in the module as global if they are assigned inside of any top-level non-function scope.

I actually started looking into implementing something like this a few minutes ago. However, it looks like it would be much easier to implement as an option in julia-syntax.jl:

  • Writing an external AST transformation is possible, but it seems like there are lots of tricky corner cases — you basically have to re-implement the scoping rules — whereas we already had the code to get it right in julia-syntax.scm.
  • It's even more tricky for something like IJulia that currently uses include_string to evaluate a whole block of code and get the value of the last expression. Not only would we have to switch to parsing expression by expression, but some hackery may be needed in order to preserve the original line numbers (for error messages etcetera). (Though I found a hack for ChangePrecision.jl for this sort of thing that may work here also.)
  • Not to mention of the case of people that include external files, which would not be caught by your AST transformation.

However, it seems easier to explain in terms of the new simpler scoping rules + a transformation that takes REPL-style input and transforms it before evaluating it.

I seriously doubt this would be easier to explain to new users than just saying that the rules are less picky for interactive use or for include with a certain flag.

@stevengj

This comment has been minimized.

Member

stevengj commented Aug 21, 2018

Here is a rough draft of a globalize(::Module, ast) implementation: https://gist.github.com/stevengj/255cb778efcc72a84dbf97ecbbf221fe

@stevengj

This comment has been minimized.

Member

stevengj commented Aug 21, 2018

Okay, I've figured out how to implement a globalize_include_string function that preserves line-number information, and have added it to my gist.

A possible (non-breaking) way forward, if people like this approach:

  1. Release a SoftGlobalScope.jl package with the globalize etc. functions.
  2. Use SoftGlobalScope in IJulia (and possibly Juno, vscode, and OhMyREPL).
  3. Fold the SoftGlobalScope functions into a future release of the REPL stdlib package and use it in the REPL.

Or is it practical to roll it into REPL.jl immediately? I'm not completely clear on how stdlib updates work in 1.0.

Please take a look at my implementation, in case I'm missing something that will cause it to be fragile.

@KristofferC

This comment has been minimized.

Contributor

KristofferC commented Aug 21, 2018

Can't we have it as a non-default feature of the REPL in 1.1?

@JeffBezanson

This comment has been minimized.

Member

JeffBezanson commented Aug 21, 2018

Duplicate of #28523 and #28750. To those saying they don't want to teach people about global variables, I suggest teaching functions first, before for loops. Functions are more fundamental anyway, and this will help set the expectation that code should be written in functions. While I understand the inconvenience, this scoping behavior can be turned into a pedagogical advantage: "In fact, global variables are such a bad idea, particularly using them in loops, that the language makes you bend over backwards to use them."

Adding a non-default feature to the REPL for this seems ok to me though.

@stevengj

This comment has been minimized.

Member

stevengj commented Aug 21, 2018

@JeffBezanson, remember that many of us would like to use Julia as a substitute for Matlab etcetera in technical courses like linear algebra and statistics. These are not programming courses and the students often have no programming background. We never do structured programming — it's almost all interactive with short snippets and global variables.

Furthermore, the reason I'm using a dynamic language in the first place is to switch fluidly between interactive exploration and more disciplined programming. The inability to use the same code in a global and a function context is a hindrance to that end, even for someone who is used to scoping concepts, and it is much worse for students from non-CS backgrounds.

@ExpandingMan

This comment has been minimized.

Contributor

ExpandingMan commented Aug 21, 2018

remember that many of us would like to use Julia as a substitute for Matlab etcetera in technical courses like linear algebra and statistics. These are not programming courses and the students often have no programming background. We never do structured programming — it's almost all interactive with short snippets and global variables.

Many of us Julia users have absolutely 0 CS background (including myself), but it seems to me that the proper attitude (especially for students) is a willingness to learn rather than demanding things be changed for the worse to accommodate our naivete.

Now, I'm not necessarily implying that this particular change would be for the worse as I only have a limited understanding of what's going on here, but if it is the case that this is a significant complication or makes it excessively easy to write needlessly badly performing code it does not seem worth it to make a change in order to have a better lecture example. You can't change the laws of physics so that the electrostatics examples you show to freshman are more applicable to real life.

So my question as a non-CS user who also cares about performance is how would I be likely to screw up if this were made the default behavior. Is it literally just the sorts of examples we are seeing here that are a problem (which I was already aware of), or are we likely to often screw this up badly in more subtle ways?

For what it's worth, I do agree that having code behave differently depending on its enclosing scope is a generally undesirable feature.

@stevengj

This comment has been minimized.

Member

stevengj commented Aug 22, 2018

Making code harder to write interactively, forcing beginners writing their first loops to understand obscure scoping rules, and making code pasted from functions not work in global scopes does not help programmers write fast code in functions. It just makes it harder to use Julia interactively and harder for beginners.

@stevengj

This comment has been minimized.

Member

stevengj commented Aug 22, 2018

Can't we have it as a non-default feature of the REPL in 1.1?

Making an "unbreak me" option the default seems wiser, especially an option that is aimed squarely at beginning users. If it is a non-default option, then precisely those people who need it most will be those who don't have it enabled (and don't know it exists).

@mauro3

This comment has been minimized.

Contributor

mauro3 commented Aug 22, 2018

What would the proposed REPL-mode do to includeed scripts? Would the evaluation of global statements depend on whether the REPL mode is activated? If so, IMO this would be at odds with the 1.0 stability promise.

@StefanKarpinski

This comment has been minimized.

Member

StefanKarpinski commented Aug 22, 2018

If we did something like this it seems like it might make sense for the module to determine how it works. So Main would be a "soft scope" module while by default other modules would be "hard scope" modules.

@dawbarton

This comment has been minimized.

dawbarton commented Aug 22, 2018

I was interested to see if it was possible to monkey patch the REPL to use @stevengj's globalize function and it appears it is without too much effort (though quite hacky). See the gist. This doesn't work with Juno (or anything else that calls Core.eval directly).

I'm not going to be recommending this to people, but it's quite useful to me when doing quick-and-dirty data analysis. I would very much like to see a (better thought out) solution since it really is quite confusing for inexperienced and often reluctant coders (i.e., my students) when you can't copy and paste in code from a function into the REPL to see what it does and vice-versa.

julia> a = 0                                                                
0                                                                           
                                                                            
julia> for i = 1:10                                                         
         a += i                                                             
       end                                                                  
ERROR: UndefVarError: a not defined                                         
Stacktrace:                                                                 
 [1] top-level scope at .\REPL[2]:2 [inlined]                               
 [2] top-level scope at .\none:0                                            
                                                                            
julia> using SoftGlobalScope                                                
[ Info: Precompiling SoftGlobalScope [363c7d7e-a618-11e8-01c4-4f22c151e122] 
                                                                            
julia> for i = 1:10                                                         
         a += i                                                             
       end                                                                  
                                                                            
julia> a                                                                    
55                                                                          

(BTW: the above is about as much testing as it has had!)

@jlperla

This comment has been minimized.

Contributor

jlperla commented Sep 6, 2018

the questioner appears to have been confused by it:

Not to mention that this is someone who clearly knows enough about programming languages to understand the nuances of scope. What about all of the matlab type users that are completely ignorant of these topics..., and probably will never invest enough time to understand the nuances.

@crstnbr

This comment has been minimized.

Contributor

crstnbr commented Sep 6, 2018

Possibly, but at this stage this is hypothetical

I've already answered multiple questions related to this on stackoverflow, mostly by new users, and even more in real life (last one just yesterday, from a Matlab user, who saw this as a no go).

@rickhg12hs

This comment has been minimized.

Contributor

rickhg12hs commented Sep 6, 2018

There will be 100 stack exchange questions which come down to the same issue.

In my "spare time", I've been adding scope, scoping, and global-variables tags to the SE questions. I only stop because of lack of time, not because there aren't more.

@JeffBezanson JeffBezanson added the triage label Sep 6, 2018

@JeffBezanson JeffBezanson added this to the 1.1 milestone Sep 13, 2018

@JeffBezanson JeffBezanson removed the triage label Sep 13, 2018

@StefanKarpinski

This comment has been minimized.

Member

StefanKarpinski commented Sep 14, 2018

Conclusion after much discussion including triage: we're going to include something along the lines of SoftGlobalScope in Base and use it in the REPL and all other interactive evaluation contexts. @JeffBezanson has pointed out that the way this is implemented is actually essentially the same as how soft scope was previously implemented, so to some extent we're coming around full circle. The difference is that now there is no scope behavior in modules or scripts, only in REPL-like contexts. I also think that explaining soft scope as a source rewrite is clearer than trying to distinguish between hard and soft scopes (which we never how Jeff explained it, I might point out).

@mauro3

This comment has been minimized.

Contributor

mauro3 commented Sep 15, 2018

These two statements confuse me a bit as they seem a bit contradictory:

and use it in the REPL and all other interactive evaluation contexts

there is no scope behavior in [...] scripts, only in REPL-like contexts.

Does this mean that the module Main has sometimes a soft scope (say at the REPL prompt) and sometimes a hard scope (say when julia -L script.jl)? Would it not make sense to say that Main always has soft scope? And a module can opt-in to soft scope by using SoftGlobalScope?

@derijkp

This comment has been minimized.

derijkp commented Sep 15, 2018

(I guess) scoping rules cannot be changed in scripts because it would be backwards incompatible, i.e. would break the promise that any code written for 1.0 will run on any 1.* version. You are correct though that the same problem with scoping for the REPL also applies to scripts (naive user at a complete loss why his/her code does not work properly when run as a script). A way to solve/alleviate this problem without major incompatibilty would be to add an option to the julia cmdline to use softscope (or alternative) , e.g. julia -f programfile, and show this option in any description/tutorial that a beginner is likely to come across.
I also see a potential alternative for the softscope that may have some advantages (though i am probably overlooking disadvantages): What if a file (a called script) would always introduce its own local scope: scoping rules would be in complete consistency with those in functions, and with the expectations of a lot of users. It would also remove a lot of the performance liabilities with new users:
No more unneeded globals (globals would have to be explicitly defined), and code might be compiled
(How many times have you had to say to put everything in a function, and to avoid using globals?)

@richardreeve

This comment has been minimized.

richardreeve commented Sep 20, 2018

I've just hit this and was completely boggled to be honest, having never seen it before in any other language. I'm planning on introducing an optional Julia course for advanced R users in my uni later this year once things have settled down, and my students will hit this on day 0 when they start randomly typing things in the REPL. And the fact that for loops behave differently from if statements just rubs salt in the wound, however logical this may be in terms of scoping. Scope inside functions is sufficiently hard to get biology students to grasp, the idea of having to explain albeit perceived glaring inconsistencies in it in the REPL / in a script / in a for loop / in an if statement (because that's what we're talking about here) in a way that is different from every other language on earth makes me very sad.

I understand the backward compatibility promise that was made, but having this work as expected by every non-cs person on the planet (and most cs people I suspect) seems like a bugfix rather than a backward compatibility issue - we're not saying that every bug will be reproduced for ever are we? The REPL fix is obviously essential, so it's great that you're proposing this, but then having to explain you can't copy a script into the REPL and expect the same behaviour seems as bad as or worse than the original problem.

Please, please, please think about treating this as a bugfix and pushing it out with scripts as well as the REPL - even if there's an switch to go to the "old" behaviour - and doing it as soon as possible in 1.0.1.

@jebej

This comment has been minimized.

Contributor

jebej commented Sep 20, 2018

A colleague that I was trying to get to learn julia also just ran into this. Having to explain the whole global vs. local variable thing at the first steps is not ideal...

@stevengj

This comment has been minimized.

Member

stevengj commented Sep 20, 2018

I don't think treating this as a "bugfix" is in the cards, because it would break the 1.0 stability contract. However, it seems reasonable to me to use softscope for scripts run with julia -i (i.e. "interactive" mode).

(That is, there would be a flag --softscope={yes|no} and it would default to the value of isinteractive.)

@StefanKarpinski

This comment has been minimized.

Member

StefanKarpinski commented Sep 20, 2018

We'll have to consider the script mode choice.

@stevengj

This comment has been minimized.

Member

stevengj commented Sep 20, 2018

For that matter, it's not crazy to me to default to --softscope=yes for any "script", i.e. for julia foo.jl, and only turn on the "hard" scoping rules for modules and include (at which point you should really be putting most code into functions).

@jlperla

This comment has been minimized.

Contributor

jlperla commented Sep 20, 2018

For that matter, it's not crazy to me to default to --softscope=yes for any "script",

That. The other one to seriously consider is Juno. Remember that people will <shift-enter> through their code to do interactive development(especially when working with the regression tests) and then later expect to be able to run the same file. Should it matter if the code is in a @testset or not (which I think might introduce a scope)? It would be very confusing to the user if the same text changes when in a @testset vs. not when using Atom's integration, and is inconsistent with doing ] test as well.

It sure sounds to me like the best solution is that the hard-scope is simply an opt-in thing, where if every other usage (including include within scripts) uses softscope unless you say otherwise.

@JeffBezanson

This comment has been minimized.

Member

JeffBezanson commented Sep 20, 2018

different from every other language on earth

Do you want to write var x = 0 to introduce every variable? That would also "fix" this, and be more like other languages.

we're not saying that every bug will be reproduced for ever are we

That is not how this works. You can't get any change to the language you want just by calling the current behavior a bug.

I reeeally don't think there should be a command line option for this. Then every piece of julia code will have to come with a comment or something telling you which option to use. Some kind of parser directive in a source file would be a bit better, but even better still would be to have a fixed rule. For example, hard scope inside modules only might make sense.

Let me try again to provide an explanation of this that might be useful for avoiding the mania, hysteria, and carnage people are seeing in the classroom:

"
Julia has two kinds of variables: local and global. Variables you introduce in the REPL or at the top level, outside of anything else, are global. Variables introduced inside functions and loops are local. Updating global variables in a program is generally bad, so if you're inside a loop or function and want to update a global, you have to be explicit about it by writing the global declaration again.
"

Perhaps that can be improved; suggestions welcome. I know, you'd rather not need any sort of explanation at all. I get that. But it doesn't seem so bad to me.

@jlperla

This comment has been minimized.

Contributor

jlperla commented Sep 20, 2018

I reeeally don't think there should be a command line option for this. Then every piece of julia code will have to come with a comment or something telling you which option to use. Some kind of parser directive in a source file would be a bit better, but even better still would be to have a fixed rule

I agree. Sounds like a teaching and communication headache to me.

For example, hard scope inside modules only might make sense.

Just so I understand: if I had a short script (not in a module!) in a .jl file which I had copied from an IJulia notebook, then if I ran that code in either the REPL directly or shift-enter in Juno, then it would behave consistently as soft-scope... but if I copied it instead of a module block then it would yell at me about globals? But if I copied that code inside of functions inside of a module, then it should work.

If so, that makes complete sense,is very teachable and coherent. Top-level scripts are an interactive interface for exploration, etc. but you would never put that kind of code in a module. Modules are something that you should fill with functions are very carefully considered globals. It would be easy to tell people about those rules.

@richardreeve

This comment has been minimized.

richardreeve commented Sep 20, 2018

Do you want to write var x = 0 to introduce every variable? That would also "fix" this, and be more like other languages.

No, I'd rather not! But scripting languages that have a REPL rarely do that (e.g. ruby, python, R, ...), they behave like Julia v0.6 did.

Julia has two kinds of variables: local and global. Variables you introduce in the REPL or at the top level, outside of anything else, are global. Variables introduced inside functions and loops are local. Updating global variables in a program is generally bad, so if you're inside a loop or function and want to update a global, you have to be explicit about it by writing the global declaration again.

I completely understand what you're saying here, and I won't (touch wood!) make this mistake again. But the whole problem I'm worried about is not me. I've found it relatively easy to introduce scope (without mentioning it directly) when I explain that variables inside functions can't see ones outside and vice-versa (even though that's more an aspiration than a fact in R!), because functions themselves are already a relatively advanced concept. But this hits much earlier in the learning curve here where we don't want anything remotely as complicated as scope to be impinging on people...

Note also it's not just "variables you introduce in the REPL or at the top level, outside of anything else, are global" and "variables introduced inside functions and loops are local", it's also that variables in if statements in the REPL or at the top level are global but variables in a @testset are local. We end up down a rabbit-hole of "just try it and work out for yourself whether it's local or global, good luck".

However, I agree with @jlperla - the proposal that "hard scope inside modules only might make sense" seems completely fine to me! Modules are a sufficiently advanced concept again... if soft scope works for the REPL and scripts, that's absolutely fine.

@JeffBezanson

This comment has been minimized.

Member

JeffBezanson commented Sep 20, 2018

we don't want anything remotely as complicated as scope to be impinging on people...
at the top level are global but variables in a @testset are local

What I'm trying to get at is that I feel a simple description of global vs. local is sufficient for early-stage teaching --- you don't even need to say the word "scope" (it does not occur at all in my explanation above). When you're just showing some simple expressions and loops in the REPL, you're not teaching people about testsets and you don't need an exhaustive list of the scoping behavior of everything in the language.

My only point is, this change does not suddenly make it necessary to teach lots of details about the language up front. You can still ignore the vast majority of stuff about scopes, testsets, etc., and a simple line on global vs. local should suffice.

@jlperla

This comment has been minimized.

Contributor

jlperla commented Sep 20, 2018

and a simple line on global vs. local should suffice.

In a world where everyone started writing all of their code from scratch, I would agree completely.

The issue is that you need to teach students not just about scope, but also about understanding the scope of where they copy-pasted code they got from. You need to teach them that if they copy-paste code that is on stackexchange within a function or a let block that they need to scan through it and find where to add "global" if they are pasting it into the REPL or a .jl file. But if they are copying that code inside a function or into the Jupyter notebook. they shouldn't. And if they find code inside of a stackexchange or tutorial page that has global variables in it, but they want to copy and modify that code inside of their own function, then they need to strip out the global.

And then students start asking why does for create this scope they need to worry about but not other things....

@JeffBezanson

This comment has been minimized.

Member

JeffBezanson commented Sep 20, 2018

We end up down a rabbit-hole of "just try it and work out for yourself whether it's local or global, good luck".

Pop quiz: in julia 0.6, is x global or local:

for i = 1:10
    x = i
end

The answer is that there's no way to know, because it depends on whether a global x has been defined before. Now, you can say for sure that it is local.

@StefanKarpinski

This comment has been minimized.

Member

StefanKarpinski commented Sep 20, 2018

Folks, this discussion is verging on no longer being productive. Jeff knows very well that the old behavior was nice in the REPL. Who do you think designed and implemented it in the first place? We have already committed to changing the interactive behavior. A decision still needs to be made about whether a "script" is interactive or not. It sounds interactive when you call it "a script" but it sounds far less interactive when you call it "a program"—yet they are exactly the same thing. Please keep the replies short and constructive and focused on the things which still must be decided. If there's comments that deviate from this, they may be hidden and the thread may be locked.

@StefanKarpinski

This comment has been minimized.

Member

StefanKarpinski commented Sep 20, 2018

One thought that I had but we dismissed as being "too annoying" and "likely to cause the villagers to get out their pitchforks" was that in non-interactive contexts, we could require a local or global annotation in "soft scope". That would guarantee that code from a module would work the same if pasted into the REPL. If we applied that to "scripts"/"programs" then the same would be true of them.

@leandromartinez98

This comment has been minimized.

leandromartinez98 commented Sep 20, 2018

When I was first introduced to Julia (not a long time ago, and I come from a Fortran background mostly), I was taught that "Julia is compiled and fast at the function level, thus everything that must be efficient must be done inside functions. In the main 'program' it behaves like a scripting language". I found that fair enough, as I cannot imagine anyone doing anything too computationally demanding without understanding that statement. Therefore, if there is any sacrifice in performance at the main program for using the same notation and constructions than in the functions, I find that totally acceptable, much more acceptable than trying to understand and teach these scoping rules and not being able to copy and paste codes from one place to another.

By the way, I am a newbie in Julia yet, having chosen it to teach some high-school and undergraduate students some basics of simulations of physical systems. And I am already hopping this issue returns to the 'normal' behavior of previous versions, because it gives us quite a headache.

@JuliaLang JuliaLang locked as too heated and limited conversation to collaborators Sep 21, 2018

@StefanKarpinski

This comment has been minimized.

Member

StefanKarpinski commented Sep 21, 2018

This conversation is locked now and only Julia committers can comment.

@stevengj

This comment has been minimized.

Member

stevengj commented Oct 8, 2018

@JeffBezanson, what would be the plan to implement the semantics you suggested in this discourse thread, initially only in the REPL and opt-in elsewhere?

It sounds like you are planning to put that directly into the lowering code (julia-syntax.scm), rather than as a syntax rewriting ala SoftScope.jl? Or would you rather have it as syntax rewriting first (modifying SoftScope to the proposed rule and converting it to a stdlib), and defer putting it into the lowering code for a later Julia release?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.