Please introduce explicit shadowing

[a-square]

A (simplified) real life example of why it's important: https://gist.github.com/Kallikanzarid/4736201

I see no reason to take away a feature of JavaScript that's useful, not broken, and cannot be misused.

Edit: also, for some reason many people claim that the same problem exists in Scheme. It doesn't, as the code in my gist illustrates.

[vendethiel]

That's Ruby scoping for you. See #1121 etc. If you really need shadowing, use do (or backticks).

[a-square]

Ruby scoping is one bad thing about Ruby, why imitate it for the sake of imitating it, especially when you can introduce explicit shadowing as an option?

Edit: to counterweight Ruby, consider that fact that pretty much every successful language has support for explicit shadowing: C, Pascal, Fortran 90, Java (as much as it has lexical scope), C#, Scheme, ML, Haskell, Erlang, Prolog - in short, pretty much any language I know and pretty much any language people care about, except Ruby, non-strict Perl and CoffeeScript.

There are good reasons why explicit shadowing should be in every lexically scoped language, mainly because with minimal effort it allows us to make sure we don't need to know anything about preceding code to reason about a function's behavior (unless it has free variables, in which case we obviously can't). As a consequence, it ensures that most functions can be rearranged without breaking anything, thus the code is less brittle.

[mehcode]

I'm all for allowing the let keyword to prefix variable declarations so they are enforced to be block-local (including shadowing any preceding declarations).

x = 42
foo = ->
  let x = 12
  x

# x === 42
# foo === 12

[jashkenas]

Sorry -- avoiding of shadowing is a choice we want to pursue as a design decision ... as I'm sure you're well aware. Take a look at Coco or Livescript for alterna-versions that embrace shadowing.

[a-square]

But you do shadow variables, it's lexical scoping! You just for some reason decided to gut our ability to do explicit shadowing.

[jashkenas]

Lexical scoping is about which variables are possible to reference across scopes, not about whether you tend to shadow your variables or not. It's well possible to program without shadowing, by choice, in a lexically scoped language.

You just for some reason decided to gut our ability to do explicit shadowing.

I did indeed. For the reasons why, see previous tickets.

[a-square]

It's well possible to program without shadowing, by choice, in a lexically scoped language

In the land of magic elves, yeah.

[epidemian]

In the land of magic elves, yeah.

That's not a very compelling nor constructive argument 😕

Also, i fail to see what would the problem be in your example. If the content variable would be global, i.e. a property of window, then that assignment wouldn't touch it (you have to explicitly do window.content = ...). The only way in that that assignment would not be declaring a new variable in that scope is if the content would be declared in an enclosing scope; but that's not the global scope, it is, at most, the scope of the file (and it would be a very poor decision to use such a generic name as "content" for a variable with very little context IMHO).

[a-square]

Some files have more than one author over time.

[juliankrispel]

I'd love to learn more about the reasoning behind that design decision @jashkenas. Is that in writing anywhere?

[vendethiel]

Probably in one of the thousands issues about this topic :p.

[juliankrispel]

@Nami-Doc not really, couldn't find anything comprehensive in the issues. I'm not really that keen on reading thousands of emotional comments about why coffeescript should behave like this or that, I'm just interested in the decision-making, would appreciate any pointers.

[michaelficarra]

Explicit shadowing is available through CoffeeScript's version of let: do.

x = 'outer'
do (x = 'inner') =>
  x is 'inner' # true
  x = 'still inner'
x is 'outer' # true

[vendethiel]

@juliankrispel I'm always the one who get to go through the issues! Fine. #712 and #1121

PS : github issue autocompletion seriously sucks :/

[juliankrispel]

He. Seems like it worked :D thanks @Nami-Doc!

[jashkenas]

T'would be good to blog about, for the record. Although that raises the threat of starting another shitstorm.

[epidemian]

It would be very good if you blog about it, @jashkenas. It could trigger some shitstorm, but it would also serve as an easy reference to point to people when asking/complaining about why CoffeeScript does not embrace shadowing.

[lewisou]

Simple example in nodejs

test.coffee

function_one = ->
    n = 10
    (i for i in [1..n])

function_two = ->
    n = 20
    n

console.log function_one() # returns  [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ]
console.log function_two() # returns 20

Then see how weird things happen, by just adding a new function at top. which will never work.

n = 1
new_function = (up_to) ->
    (i for i in [n..up_to])

function_one = ->
    n = 10
    (i for i in [1..n])

function_two = ->
    n = 20
    n


console.log function_one()
console.log function_two() 
console.log new_function(5)
# expecting [1, 2, 3, 4, 5]
# but you will get [ 20, 19, 18, 17, 16, 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5 ]

In real project function_one and two may very complex, and no one knows n is used in them...

[lewisou]

Here's a real project example

redis = require 'redis'
db = redis.createClient()

# ...... a lot of code here.

class GetLinks extends nodeio.JobClass
   input : (_, num, callback) =>
       # in here i want to create a new connection to redis. so i
       db = redis.createClient()
       # actually i have destroyed the outer db. and i don't know... and its very hard to debug.

       db.scard queue_set, (err, n) =>
          .....

[michaelficarra]

@lewisou: If you introduce n in that outer scope, you do have to realise that it is now available to everything in that outer scope including any of your function's siblings. You run the same risk in JavaScript. Always declare a variable in the innermost scope possible.

[lewisou]

@michaelficarra You right. but in javascript, I always use var when it's in need. as a variable will be global without var keyword. so there's no problem in javascript when a var key word given. which can shadow the outer one.

Yes. an outer scope var should be available to everything. but if local var can shadow the outer one, then there will be no problem.

So the lexical scope seems not that good. uh.

[machadogj]

+1
var for explicit shadowing sounds elegant enough. don´t see a reason for not letting devs bullet-proof inner scopes.

[michaelficarra]

You can always use let-scoping: do (n = undefined) =>

[machadogj]

We know.

But do you think that do (n = undefined) => is ideal?

We already deal with enough callbacks IMO to have to be doing that. Don't you think?

[michaelficarra]

@machadogj: I think it's fine for trying to do something that's strongly advised against in the first place. You don't need to do that if you just choose a different name.

[machadogj]

@michaelficarra "strongly advised against", where? by who? and most important, why? (other than personal taste)

CoffeScript is a very small mind-shift from Javascript in favor of readability and elegance, except for this. From those coming from JS (almost everybody) this feels awkward.

[epidemian]

I understand where the need for it comes from; the Real World example @lewisou posted is very illustrative. And i think i have been bitten by this at least once.

That being said, i'm with @michaelficarra in that introducing a var keyword for explicit shadowing would do more harm than good. Having more than one recommended way of doing the simplest thing like assigning a variable would be madness IMO.

Another option would be to make the = operator to always introduce a local variable (i.e. it's a variable declaration + initialization; the common case). And then have another symbol to denote variable assignment without declaration, like :=. Of course this would be a very breaking change, but, do you think these semantics would be better?

It's funny because as i was writing the above paragraph i though "this is something that LiveScript might have", and indeed it does! @lewisou's second example would work correctly in LiveScript.

That being said, i'm not entirely sold in LiveScript's implementation of this. I'd prefer the = to be more strict and not be allowed to assign the same variable twice in the same scope:

n = 5
if foo 
  n = 6 # This compiles; but i'd prefer to have to use a := here.

[machadogj]

@epidemian

"Having more than one recommended way of doing the simplest thing like assigning a variable would be madness IMO."

var is not for assigning. is for declaring. whether devs will abuse from this, is a possibility.

[vendethiel]

@epidemian if doesn't introduce a new scope. And :='s from coco.

[epidemian]

@Nami-Doc, triple post!

Yeah, i understand what the semantics of = vs := are. I just don't think they are the most intuitive. Even in the simple case of a = 42; a = 33 i'd expect the compiler to complain in the second statement with something like "there's already a variable 'a' in this scope, maybe you meant to use :=?".

But that's a minor nitpick. LiveScript is consistent with JS semantics if you consider that a = b decalres the variable "a" and assigns it to "b". a = 42; a = 33 in LiveScript is the same as var a = 42; var a = 33 in JS; but i always thought that it's quite confusing that JS doesn't even raise a warning about that construct.

@machadogj, you're right, i should have said "Having more than one recommended way of doing the simplest thing like declaring a variable would be madness".

[vendethiel]

Whooooops. Deleted them.

We do have an "shadowing by mistake" error tho. (a = 5; do -> a := 6; a = 7 iircH

[lewisou]

@michaelficarra

You can always use let-scoping: do (n = undefined) =>

Yes, we can. that's just like i use 'var' everywhere in javascript to ensure the variable is a local variable.
So removing the var keyword is a good idea in terms of people don't need to write var everywhere. But the compiler does a little bit unsound while people always refer to a innermost scope varible instead of the outermost scope varible.

So when i write

# a lot of outer scope codes here

a_fun = () ->
      i = 1 
# here i always mean i want to create a local varible 'i'. 
# as programers dont want to know what happens in outer scope. and even don't want to look at outer scope codes.

n = 1 
# here i just want to declare a outer scope varible 'n', 
# and don't want to know who is using the same name in the rest of the file. 
# and for sure don't want to persuade other programers to change there local var name to a different name if they are using n in their local function.

b_fun = () ->
      a lot of codes here

c_fun = () ->
      a lot of codes here

I think it's fine for trying to do something that's strongly advised against in the first place. You don't need to do that if you just choose a different name.

I admit that changing to a different name can go around this issue. but

• in 90% chance, programmer don't know there's a same name variable in the outer scope. which is an accident.
• When adding an outer scope varible, onbody want to go through the whole file to make sure no body using this name already.

@epidemian

That being said, i'm with @michaelficarra in that introducing a var keyword for explicit shadowing would do more harm than good.

Could you please give a link that illustrates the harmness of adding the var keyword back. (although i don't think we do want the var keyword back. may be we just need the keyword 'global' (just like what python does) because a variable should be treated as a local variable first)

@epidemian

Another option would be to make the = operator to always introduce a local variable

+1 i think so too. that's what we need.

[erisdev]

I would be in favor of = introducing a new variable if CoffeeScript were brand new, but I think it's unfortunately a little late in the game to go changing that now.

[epidemian]

@erisdiscord,

I would be in favor of = introducing a new variable if CoffeeScript were brand new, but I think it's unfortunately a little late in the game to go changing that now.

Yeah, i agree. It does however bug me a little bit that the only two seemingly possible ways for languages to evolve are to either maintain retro-compatibility at all cost (and thus having to deal with bad (or good at the moment but bad now) past decisions forever) or break it once in a while and deal with horrible fragmentation in the community and a painful and slow migration (e.g. Python 2 -> 3).

Isn't there some way to break backward compatibility when it makes sense to do so (i'm not talking about this particular issue here) and have an easy migration path at the same time?

I'd like to imagine that good tooling could help. Using the proposed semantics for = and := as an example, i think it should be possible to make a program that transforms code with the current assignment semantics to the proposed ones while maintaining its meaning. It would be hard to make such program, as it would need to understand "two versions" of the language and translate from one into the other. But maybe it is a worthwhile effort. It could enable users to migrate from a major version of the language to another without having to review all their code.

@lewisou

Could you please give a link that illustrates the harmness of adding the var keyword back.

I think the biggest drawback is the added complexity. If it's hard to see why that complexity could be so bad, i'd recommend to think of languages have tons of ways of doing the most basic things, like C++, where every time you declare a variable you have to think about so many things (like, will it be a simple value, or a pointer, or a reference, or an auto_ptr or some other kind of pointer/container type? will it be "const"? will i need to delete the object before the variable goes out of scope? Can an exception be raised at any point that will make the value pointed by this variable leak? etc, etc, etc). Only after trying out simpler and more opinionated languages did i realize how liberating it is to not having to consider so many things all the time.

I think there's tremendous value in keeping things simple and orthogonal, but there's always this tension when some users want "just this little thing" added to the language. I can totally relate to it. And i think it's important to consider those requests, but not be too quick on the "let's add it!" decision.

[a-square]

I think the biggest drawback is the added complexity. If it's hard to see why that complexity could be so bad, i'd recommend to think of languages have tons of ways of doing the most basic things, like C++

I call BS on that one. There's nothing complex about explicit shadowing, I dare you to make an example where there one would be confused on whether or not to use it.

Trying to bring up C++ is ridiculous. C++ has evolved under two constraints:

1. It should have explicit memory management,
2. It should support OOP well while remaining as performant as possible.

Any language evolving under these constraints would be in all important aspects exactly like C++. And while C++ has a lot of bloat (coming mostly from operator overloading and C compatibility, but also from legacy features, like old pre-closure support STL design, and some misguided features like multiple inheritance and, yes, the const), it's certainly the best we have for many applications, and it's getting better (e.g. it now has better semantics with rvalue references so it's not as clumsy with stack-allocated variables anymore). And it's certainly not comparable to JavaScript, which is a completely different language made for completely different use cases. So to even bring C++ up in a conversation about variable shadowing is pure, undilluted bullshit.

[juliankrispel]

Previously for me this had been a reason not to use Coffeescript. I had been convinced that having control over where in your code a variable is defined is crucial. Since then, I've learned that Coffeescript is really about focus. It reduces as much boilerplate from Javascript as is possible so that you can spend less time writing code and more time thinking about what you're trying to express. If you can't find the value in that, then maybe Coffeescript isn't for you.

[a-square]

It reduces as much boilerplate from Javascript as is possible so that you can spend less time writing boilerplate and more time thinking about what you're trying to express.

Exactly how does it do that? By providing syntactic sugar for something that looks like class-based inheritance? By providing a query DSL? I agree on those counts. What does it have to do with variable shadowing? Let me guess, nothing.

[juliankrispel]

var is boilerplate and something that can be a compilers responsibility.

[vendethiel]

1. I doubt this is gonna change by now (you can use forks providing this feature if you wish) since this has been a design decision since the beginning.
2. I'd rather have the discussion, if it has to exist, stay on focus without being disrespectful with one another :) thanks

[a-square]

I can't see how it can be the compiler's responsibility if the compiler has no way of knowing the intent.

[epidemian]

@Kallikanzarid

I call BS on that one. There's nothing complex about explicit shadowing, I dare you to make an example where there one would be confused on whether or not to use it.

It seems my message wasn't clear. Introducing a new concept to a language will invariantly increase its overall complexity. Coming with an example where having an optional var for explicit shadowing would increase the complexity of the code is easy, just mix the two types of variable declarations:

# ... code
foo = ->
  var x = 42
  y = 33
  # ...

If someone comes across that code, they'll have to decipher why one variable uses explicit shadowing while the other one doesn't. They may ask themselves: is the "var" really needed here? Shouldn't "y" also use it, or is it declared in an outer scope? If i need to declare a new variable "z" here, should i use "var" or not? Etc.

Now imagine having to deal with those questions for all the variables in a big and complex source file. Just to have optional explicit shadowing? No, thanks. That's the idea of cognitive load that i was trying to express in my previous comment.

Also, it may have sounded like i was bashing C++. That was not the intention at all. In fact i enjoyed programming in C++ quite a lot; and would gladly do it again if the problem asks for it. But i think there's no point in denying that it's a very complex language, and that the ramp you have to climb to use learn how to use it correctly is pretty high.

Finally, try to be civil man. I understand that you were criticising my opinion and not making a personal attack, but sincerely there's no need for stuff like:

So to even bring C++ up in a conversation about variable shadowing is pure, undilluted bullshit.

[lewisou]

I think the lexical scope should only be used in strict functional programming, eg SML/NJ, HASKELL. Am i right? Because in FP, there's no real variables, everything is a constant and can not be changed. So programmers don't need to consider whether an outer scope variable will be changed by accident. (but FP do have shadowing)

In OOP... I don't want to illustrate how bad the lexical sope is here any more, i think it should be quite obvious in my previous posts.

If coffee just copying whatever in Ruby... then that's so sad. As ruby has a terrible variable scoping system.

1. yes, it's lexical scope
2. No file scope, Any variable defined out of a module/class will be globally accessible from the whole application after loaded (Horrible, but fortunately ruby programmers don't use that kind of variable at all, my friend told me he never use global variable in his ruby projects).

Fortunately ruby restricts OOP, people always use instance variables, so shadowing issue rarely happens. which means when using a local variable in a local function, there's almost always no global one with the same name as there's almost no global variables. And methods can not be defined in another method in ruby, so there's not many outer scopes. use instance variable everything will be fine.

Back to coffee, which is based on javascript. outer scopes everywhere. global/file scope variables everywhere...at least in nodejs.

BTW, I found livescript is amazing...(ironically i did't know livescript before i joined this conversation) anybody who hates the scope issue in coffeescript and loves FP can try livescript out. which is also support OOP.

[lewisou]

@epidemian I saw your point... yes, Adding the var keyword back is a real bad idea.
But

y = 1
# bla bla bla, other codes
foo = ->
  x = 42
  y = 33

Don't you think the y should be a local variable by default regardless what happens out of function foo? it's called shadowing.

I think this is a very serious issue. But if you still say that's the design decision at very first place. then that's ok as I just give advices actually.

[juliankrispel]

@Kallikanzarid it is the cs compilers responsibility and it has been doing this exact thing for ages. Hundreds of thousands of people have been using coffeescript without var and are pretty happy with it. If you have such strong feelings about it why don't you fork it and create your own flavour? Or create a coffeescript extension of something.

@lewisou are you saying that not having var is ok but variables should always be local? How are you going to access variables from an outer scope then? As you can see it's either or, you can't remove var and keep shadowing.

[epidemian]

Don't you think the y should be a local variable by default regardless what happens out of function foo? it's called shadowing.

Yes, but i don't think we'll see that change happen.

[vendethiel]

You access them the same way, and you can an outerscope variable's value with := (in ls).

[machadogj]

Here is another real-world example of this problem, when converting javascript files into coffeescript (not even sure anybody cares about this), it can become a problem.

I ran into this sample app https://github.com/reza-farhadian/Simple-Users-Management/blob/master/users.js which I converted one file to CS using http://js2coffee.org/ and I believe (haven't tested it though) that we have the global users variable being overriden:

https://gist.github.com/machadogj/6249933

[realyze]

+1 for shadowing.

It is a very common issue. Example (stealing a snippet from my colleague):

domain = require 'domain'`
...
constructor: (@_data, @_errorData, @_options) ->
  @_filters.BlockDomains = (for domain in blockDomains
        new RegExp ".*://(www\\.)?#{escape domain}.*", 'i'
    )
...
  generate: () ->
    d = domain.create() # BROKEN!

[connec]

You can fix that example by renaming either the domain variable at the top of the file or the domain variable in ...for domain in blockDomains....