Add new proposal for lexical scoping

[jamesls]

This is a new proposal for lexical scoping based on comments here.

This supersedes JEP-11.

[jamesls]

FWIW I took a shot at implementing this in python, and it turned out to be pretty straightforward in case anyone wants to experiment with let expressions: jmespath/jmespath.py#307 .

I'm taking it as a good sign that the implementation was really straightforward and fairly minimal.

[springcomp]

@jamesls thanks tons for the timely feedback and progress. This is really appreciated.

The JEP is really complete. The only thing I would suggest needing calling out for is how to access potential properties called let and in in a JSON object.

I think, within a let-expression it is mandatory to use a quoted-identifier for that. However, what about outside of a let-expression ?

Are let and in a legal identifier outside let-expression rules?

Given:

{ "let": "in" }

Which of the following expressions are legal, if any?

1. let $let = let in { let: let, in: $let }
2. let $let = "let" in { let: let, in: $let }
3. let $let = "let" in { let: "let", in: $let }
4. let $let = "let" in { "let": "let", "in": $let }

I think the last expression is definitely valid.
Maybe it would be nice to have expressions 2 and 3 legal as well.

What about the first?
If not legal, should we specify that a syntax error MUST be raised?

For instance:

search( foo.let.in, { "foo": {"let": "in"} } ) -> ERRs syntax

[springcomp]

May I kindly suggest we call this JEP-11a?

[jamesls]

Which of the following expressions are legal, if any?

All of those are legal. There's nothing in the proposed grammar changes that would result in a parse error. Using let as an identifier in an expression wouldn't match the let-expression rule, so it would then match the rules for identifiers.

Are let and in a legal identifier outside let-expression rules?

Yes, new additions must be backwards compatible. These are valid identifiers with the current specification, they will continue to be valid. This will need to hold true for any keywords that get added in future JEPs.

I can update the JEP later with word clarification on these keywords as well as additional testcases.

[springcomp]

Yes, new additions must be backwards compatible. These are valid identifiers with the current specification, they will continue to be valid. This will need to hold true for any keywords that get added in future JEPs.

Of course. I naïvely started to tokenize let and in keywords, but that does not make sense.

I can update the JEP later with word clarification on these keywords as well as additional testcases.

Yes, indeed, I think that is definitely worth mentioning.

[springcomp]

@jamesls what about making the in <expression> optional, and support simple assignment as well?

This would effectively set the last part of the let-expression an implicit in @ or in `null` suffix.

Maybe:

• let $root = @ ([ in @ ])
• let $const = `0` ([ in @ ])

[jamesls]

what about making the in optional, and support simple assignment as well?
This would effectively set the last part of the let-expression an implicit in @ or in null suffix.

What would be the motivation for doing this? Is there a specific issue/usability problem it's trying to address?

[springcomp]

What would be the motivation for doing this? Is there a specific issue/usability problem it's trying to address?

There are no immediate issue or usability problems.

I have been thinking about a potential future reduce() or fold() function for a long time and which could make use of bindings.

Something that may look like (provided there were support for simple arithmetic operations):

• Canonical sum() defined as reduce(array, let $acc = `0`, &@ + $acc)
• Canonical product() defined as reduce(array, let $acc = `1`, &@ × $acc)

In a reduce operation, the issue is to be able to both name and assign a value to the accumulator.

A potential proposal I had in line with JEP-11 was to use a "scope" object as the first argument:

• JEP-11-like reduce( {acc: `1`}, array, &@ × acc )

That way, reduce() knows what is the name of the accumulator as well as its initial value.
Here a binding assignment seems cleaner.

[jamesls]

There are no immediate issue or usability problems.
I have been thinking about a potential future reduce() or fold() function for a long time and which could make use of bindings.

Got it. I think it'd be better to explore that in a separate proposal, as one of the key things in a JEP is having sufficient motivation as to why something's added.

I briefly mentioned it in this JEP, and I believe elsewhere in one of the other let discussion threads, but functions like reduce need functions as args, and I would explore what it would look like to allow users to define their own functions. I don't think let is the right mechanism for that.

So I'd research either adding rust/ruby style closures or javascript arrow functions to the spec, and see 1) is it even possible and 2) if the additional complexity is worth the use cases it enables.

But at any rate, worth a separate discussion.

[mtdowling]

This is great, James! It addresses any concerns I had around JEP 11. The addition of $ makes the difference between variable access and current node access explicit. The introduction of a contextual keyword for let is great and makes this much more readable than a function.

[mtdowling]

The fact that these are meant to be considered separately is a little hard to sus out. Maybe separate code blocks or at least add a newline between them.

[mtdowling]

Cool, makes sense. I'd probably add some kind of predefined-params option to the linters in https://github.com/awslabs/smithy/tree/main/smithy-jmespath/src/main/java/software/amazon/smithy/jmespath.

[eddycharly]

It looks like expressions like let $a = 'top-a' in let $a = 'in-a', $b = $a in $b are not allowed.

To me, it looks like if variables are set up in the same order they are declared it would make sense to allow that.
I mean in the example above, $a is assigned first, then $b comes second and uses $a, etc...

Allowing this could be useful and help simplifying expressions.
Use case I have in mind:
let $current = foo.bar.something, $priority = $current.priority in ....

Now $priority just references a field from $current.

[jamesls]

It looks like expressions like let $a = 'top-a' in let $a = 'in-a', $b = $a in $b are not allowed.

This is allowed and is a valid expression. It means that $b will refer to top-a here.

Allowing this could be useful and help simplifying expressions.
Use case I have in mind:
let $current = foo.bar.something, $priority = $current.priority in ....

I mentioned in the rationale that I want to stick to existing precedent here unless there's a good reason to deviate. Give an expression let x = e1 in e2, you can either have x be valid only in e2, or you can allow x to be visible in e1, which is closer to let rec in several languages. The latter enables mutual recursion, which I purposefully did not want to be possible with this feature.

Rather, your suggestion is somewhere in between where given N binding clauses, the evaluation of binding x_i in e_i gives visibility to any bindings defined form x_0 to x_i-1. That seems unnecessarily complex, when you already get similar readability with the existing proposal:

let $current = foo.bar.something in
let $priority = $current.priority in ...

A few extra characters while sticking to established precedent seems like the right tradeoff for me.

[eddycharly]

Rather, your suggestion is somewhere in between where given N binding clauses, the evaluation of binding x_i in e_i gives visibility to any bindings defined form x_0 to x_i-1. That seems unnecessarily complex, when you already get similar readability with the existing proposal:

let $current = foo.bar.something in
let $priority = $current.priority in ...

A few extra characters while sticking to established precedent seems like the right tradeoff for me.

I'm not a functional programming language expert at all and that's probably why i find it counter intuitive.

Most (if not all) languages allow referencing previously declared local variables:

func f() {
  var a = 10
  var b = a // `b` depends on `a`
  // ...
}

Anyway, I'm fine with both.

[jamesls]

Most (if not all) languages allow referencing previously declared local variables:
func f() {
var a = 10
var b = a // b depends on a
// ...
}

Right, and I want to be clear that I think that's a reasonable thing you might want to do. My previous comment was saying that it's possible to do that with this proposal in a way that adheres to existing precedent and has syntax almost identical to what you've quoted:

let $a = `10` in
let $b = $a in
<your expression>

Swap out var with let, and instead of a , replace it with in and it's the same thing as your code example, while still preserving simple evaluation semantics. IMO best of both worlds.

[jamesls]

Ok, looks like there's no more comments to address and we've had a couple weeks for feedback. At this point, I'm going to go head and accept this JEP. Thanks to everyone that helped with the ideas behind this JEP over the years, excited to add this to the language!