Add representation for meta properties #32

Merged
merged 1 commit into from Mar 26, 2015

Projects

None yet
@RReverser
Member

ECMAScript gets new syntax thing call "meta properties", which is already used as new.target and, according to meeting notes, can be used in future in other places as well.

For those cases, we need a new node that (up to discussion) derives from MemberExpression but is restricted to having Identifier in both object (as it's actually a keyword used as Identifier) and property and can't be computed (i.e. new['target'] is syntax error).

@michaelficarra
Contributor

The new.target production has little to do with MemberExpression, other than being at the same precedence level. In the Shift AST, we simply introduced a NewTarget node. If you're concerned about yield.input and other future meta properties, I suggest this:

interface MetaProperty <: Expression {
    type: "MetaProperty";
    object: Identifier;
    property: Identifier;
}
@RReverser
Member
  1. Yeah, I'm concerned about future meta-properties and willing to design node that would be flexible enough to cover those with minimal changes.
  2. I thought about it, but I don't think that MetaProperty would be correct type here since Property in AST represents completely different syntactic construction (object property) and having name that is so similar and differs only with prefix might be super confusing.
  3. On other hand, syntactic construct like obj.prop is the closest one to meta properties from both parser and "user familiarity" perspective, and so, while I'm not saying that we must inherit from MemberExpression, I do think that MetaMemberExpression or, if you wish, MetaMember will be less confusing for consumers than MetaProperty is.
@michaelficarra
Contributor
  1. That's a reasonable position.
  2. I don't think there's any reason to be concerned about people confusing MetaProperty with object properties.
  3. I can't agree with this. They really have nothing to do with each other. They just look similar in the concrete syntax. That shouldn't affect the design of the AST, as it is only concerned about the underlying structure of the program.
@kittens
Contributor
kittens commented Feb 19, 2015

Inheriting from MemberExpression and having computed: false doesn't make much sense because it'll never be true as it has no concept of computed even though it just looks like.

I agree with @michaelficarra here. I don't really think an ambiguous name (MetaProperty isn't ambiguous anyway IMO) is a problem since if a consumer ever wants to deal with those nodes they'll already be familiar with them and if they aren't then this is where the spec comes in.

@mikesherov
Contributor

I agree with @michaelficarra on all points here.

@getify
Contributor
getify commented Mar 4, 2015

They just look similar in the concrete syntax.

Building off that mindset, I don't think that object (and quite possibly property) are necessarily the right names, as they imply something like the new part being an "object", which is only an accident of the appearance in the concrete syntax, not how JS is actually treating it.

What about context instead of object? And perhaps what about meta or flag instead of property?

interface MetaProperty <: Expression {
    type: "MetaProperty";
    context: Identifier;
    meta: Identifier;
}
@michaelficarra
Contributor
interface MetaProperty <: Expression {
    type: "MetaProperty";
    kind: "new.target";
}

kind can be extended in the future to allow "yield.input" or other indicators. I don't see a reason to split them on . since the components of a meta property aren't individually meaningful.

@getify
Contributor
getify commented Mar 4, 2015

since the components of a meta property aren't individually meaningful.

Is that entirely true? The grammar says:

NewTarget :
    new . target

I'm unable to validate at the moment if that means that new.target must appear directly together as a single identifier, or if new . target or new \n .target or other variations are going to be allowed. The same grammar for super . identifier does appear to allow whitespace in/around the ., so I'm guessing the same will be true of new.target. If that's true, then we of course may have concrete syntax associated with each part individually.


If it turns out that only new.target altogether is allowed, no whitespace, then I think it should be:

kind: "newTarget";

To match the terminology used in the spec, rather than the new.target JS syntax, which we're basically then asserting is nothing more than an identifier that can have a . in it.

@michaelficarra
Contributor

They are separate tokens, so whitespace is indeed allowed. But that has no bearing on whether we would represent the MetaProperty using two values or one. If we want to start storing concrete syntax, we could simply store each concrete syntax element in separate named fields for the four possible positions whitespace is allowed. We would also need to store the escape sequences used to represent the MetaProperty. Anyway, I don't think we should be concerning ourselves with those things right now, since we currently have no plan to add concrete syntax info to the AST.

@getify
Contributor
getify commented Mar 4, 2015

since we currently have no plan to add concrete syntax info to the AST.

I object to making any "final decisions" on any of the stuff here that then paints us into an awkward corner when (later) trying to even decide if and how we might approach concrete syntax. No, we haven't officially decided to doing so or not, but we also shouldn't design as if there's no knowledge that such a thing might be a concern in the near future.

In the same way that we can say "you know, yield.input (or whatever) may happen, let's not unnecessarily restrict ourselves", I think we can also reasonably say "let's not rush to assume that new.target couldn't possibly benefit from individual addressability."

Does it really gain us that much to collapse to kind: "newTarget"?

@isiahmeadows
Contributor

Since the spec appears to simply whitelist it, and further discussion in ES
Discuss (there's a thread on other proposed meta properties, but there has
also always been consensus on whitelisted properties only), I don't quite
see the problem of simply collapsing it to kind.

Now, as for "newTarget" being the collapsed form, I find it easier to
reason "new.target" instead. It is easier to connect to the meta property
itself than the camel cased version, and if you ever need the keyword
itself, it's a little simpler to figure out, especially if camel cased
identifiers are eventually (inevitably) introduced as the "property" name:

// my proposal
// "new.target"
// -> keyword = "new"
// -> property = "target"
let [keyword, property] = node.kind.split(".");

// current idea
// "newTarget"
// -> keyword = "new"
// -> property = "target"
let keyword, property;
{
  let {kind} = node;
  let re = /[A-Z]/;
  re.test(kind);
  let index = re.lastIndex;
  keyword = kind.slice(0, index);
  property = kind[index].toLowerCase() +
    kind.slice(index + 1);
}

I don't see much justification for not collapsing the properties,
especially if my proposal is accepted for how it's collapsed. Even if not,
I don't see much use for not collapsing the property to a simple kind,
since there does seem to be long term consensus on the meta properties
being exclusively on a whitelist (with all unknown ones being (sadly) a
compile time error (have to feature test and then send a conditional Ajax
request or a conditional module substitution with the future module
loader...sorry for the rambling)).
On Mar 4, 2015 2:31 AM, "Kyle Simpson" notifications@github.com wrote:

since we currently have no plan to add concrete syntax info to the AST.

I object to making any "final decisions" on any of the stuff here that
then paints us into an awkward corner when (later) trying to even decide if
and how we might approach concrete syntax. No, we haven't officially
decided to doing so or not, but we also shouldn't design as if there's no
knowledge that such a thing might be a concern in the near future.

In the same way that we can say "you know, yield.input (or whatever) may
happen, let's not unnecessarily restrict ourselves", I think we can also
reasonably say "let's not rush to assume that new.target couldn't
possibly benefit from individual addressability."

Does it really gain us that much to collapse to kind: "newTarget"?


Reply to this email directly or view it on GitHub
#32 (comment).

@kittens
Contributor
kittens commented Mar 9, 2015

@impinball You shouldn't have to resort to hacks like that to get the keyword and property.

@isiahmeadows
Contributor

Actually, I take back my entire support for collapsing the node, since it would better support position information interesting for refactoring tools. Also, the spec appears to allow syntactically insignificant whitespace on either side of the dot, as in traditional member accessors.

It would still be easy to check if left as explicitly extending MemberExpression with no extra properties and some prose to explain its purpose separate from MemberExpression.

function isMetaProperty(node, type) {
  if (node.type !== "MetaProperty") {
    return false;
  }
  if (type == null) {
    return true;
  }
  let [keyword, property] = type.split(".");
  return node.object.name === keyword &&
    node.property.name === property;
}

// ...
if (isMetaProperty(node)) {
  // it's a meta property
}
if (isMetaProperty(node, "new.target")) {
  // it's new.target specifically
}
@RReverser
Member

Actually, I take back my entire support for collapsing the node, since it would better support position information interesting for refactoring tools.

👍

@isiahmeadows
Contributor

Clarification (I forgot): all of these are equivalent according to the
spec, and this is why I said that.

new.target
////////////////////////
new.

    target
////////////////////////
    new

.target

Sorry about it.
On Mar 9, 2015 5:47 PM, "Ingvar Stepanyan" notifications@github.com wrote:

Actually, I take back my entire support for collapsing the node, since it
would better support position information interesting for refactoring tools.

[image: 👍]


Reply to this email directly or view it on GitHub
#32 (comment).

@RReverser
Member

So, to get conclusion on this... I agree with @impinball and understand concerns about inheritance from MemberExpression. I actually like keyword and property names for this, as they are pretty clear when used in context of meta properties.

How about following?

interface MetaProperty <: Expression {
    type: "MetaProperty";
    keyword: Identifier;
    property: Identifier;
}
@isiahmeadows
Contributor

LGTM
On Mar 12, 2015 12:42 PM, "Ingvar Stepanyan" notifications@github.com
wrote:

So, to get conclusion on this... I agree with @impinball
https://github.com/impinball and understand concerns about inheritance
from MemberExpression. I actually like keyword and property names for
this, as they are pretty clear when used in context of meta properties.

How about following?

interface MetaProperty <: Expression {
type: "MetaProperty";
keyword: Identifier;
property: Identifier;
}


Reply to this email directly or view it on GitHub
#32 (comment).

@getify
Contributor
getify commented Mar 13, 2015

Are we (reasonably) certain that all future meta properties are going to be keyword.property? So far, new.target and a possible future yield.input have been thrown around. new and yield are definitely "keywords", though technically the spec says that yield sometimes has the semantics of "Identifier".

Is it reasonably possible that in the future there's a metaproperty which is not based on a keyword? For example, words like constructor, static, let, get, set, from, module, and as are not called "keywords" in the spec. Could there be things like static.method or let.property? Also, there's several literals like true, false, null, etc, which could theoretically be used, like null.empty, right?

Also, the module export mechanism apparently supports a special token of this module (note the space, not period). It's unclear to me if that one should be considered a "metaproperty" or not, but if other such things are conceived in the future, might they need a way to be represented in this node type?

Just don't want us to regret a name like "keyword" sometime down the road.

@getify
Contributor
getify commented Mar 15, 2015

Further info on potential future metaproperties:

https://github.com/allenwb/ESideas/blob/master/ES7MetaProps.md

Looks like all those so far hang off function. But it clearly lets us know there's a good chance there may in fact be quite a few of these. We should design as flexibly as possible.

@getify
Contributor
getify commented Mar 15, 2015

More info:

https://esdiscuss.org/topic/function-name-property

That thread starts out talking about function.name but morphs into a general discussion about meta properties, their original design and intent, etc. Good back-story context for them, if you haven't read it yet.

Some observations:

  1. Several messages tossed around ideas like in.function.
  2. Some people floated ideas like =>.self.
@isiahmeadows
Contributor

@getify Good catch. Maybe we should come up with some better representation/node name instead of keyword for the first. Maybe instead using host and property, so it's a little more precise?

Also, am I the only one wondering about the merge conflicts with this PR?

@getify
Contributor
getify commented Mar 16, 2015

i'd be ok with "host". I suggested "context" earlier in the thread, too.

@jeffmo
jeffmo commented Mar 18, 2015

After our weekly esprima meeting, it seems most of us are onboard with the structure that @RReverser suggested above, except that Identifer probably can't be used since the tokens on either side of the . can (and already are) keywords and not identifiers.

So an example of the structure that at least everyone in that meeting was ok with was:

{
  type: 'MetaProperty',
  meta: {
    type: 'Meta',
    value: 'new'
  },
  property: {
    type: 'Property',
    value: 'target'
  }
}

The benefits of this structure are (a) it allows things like esprima/acorn/etc to extend the structure and include location info alongside the 'new' and 'target' tokens (since it's possible to have whitespace between them) in a consistent way with how it includes location data for other nodes (b) it makes the overall structure immediately distinct from MemberExpression.

@RReverser
Member

Well, we already have a precedent of using e.g. property: Identifier in representation of obj.new - new is also generally a keyword but is used as Identifier in current context, which is similar to MetaProperty case (keyword-as-an-Identifier-when-used-in-certain-type).

Also, in:

property: {
  type: 'Property',
  value: 'target'
}

using Property type is rather incorrect since this type has different meaning and structure, see https://github.com/estree/estree/blob/master/spec.md#property for details.

@jeffmo
jeffmo commented Mar 18, 2015

using Property type is rather incorrect since this type has different meaning and structure

Works for me. I'm not personally tied to any of those names, but I think @michaelficarra found them appealing. I'm mostly interested in the structure (I'd really like for it to be possible to extract location info for new separately from target)

@RReverser
Member

@michaelficarra sometimes tries to patch ESTree to look it more like Shift :P just saying

@RReverser
Member

But yeah, as for structure I'm totally with you and I actually like meta and property pair more than any of previous proposals.

@mikesherov
Contributor
{
  type: 'MetaProperty',
  meta: {
    type: 'MetaPropertyMeta',
    value: 'new'
  },
  property: {
    type: 'MetaPropertyProperty',
    value: 'target'
  }
}

@RReverser @sebmck @jeffmo ?

@michaelficarra
Contributor

I had actually forgotten that Property was a node in SpiderMonkey during today's meeting. Whoops.

👍 @mikesherov.

@nzakas
Contributor
nzakas commented Mar 18, 2015

👍

@gibson042

The spec defines MetaProperty as an expansion of MemberExpression, and comments imply that it is fixed as ReservedWord . IdentifierName (with the first token potentially being expanded later to include Punctuator). I see no reason to trump that here—despite the suboptimal "object" property name and slight weirdness of making its value an Identifier, both remain comprehensible.

This PR looks good to me as it is right now, deriving from MemberExpression and thereby exposing the inherent substructure of MetaProperty without introducing new concepts that slightly differ from already-existing types just to get better property names.

@kittens
Contributor
kittens commented Mar 19, 2015

MetaPropertyMeta
MetaPropertyProperty

This seems crazy and looks like code smell. It's similar to functionNode.body.body.

@mikesherov
Contributor

Yes, it's unfortunate. Although I need some time to formulate response to @gibson042 that explains the reasons MetaProperty can not extend MemberExpression despite it being part of MemberExpression's productions.

@mikesherov
Contributor

So some things questionable about this PR:

  1. it has useless computed: false.
  2. Extending from MemberExpression implies Liskov substitution... Is MetaProperty allowed anywhere a MemberExpression is allowed? If not, then we need to change other parts of spec to allow/disallow MetaProperty.
  3. It extends from MemberExpression seemingly for the property names.
  4. It lists identifier as the type of the object when they are not objects nor are they identifiers (especially in the case of => which is not even a valid identifier).
  5. It ignores the fact that the object property needs to be something other than identifier. We don't currently represent reserved words in the AST because they're embedded in the structure. This is why meta programming is weird. We are tasked to describe the structure in the structure.
  6. We must introduce a new type to capture the object here. I say, call it Meta. We can still have the property be an identifier, but Im certain we need a meta type.
@kittens
Contributor
kittens commented Mar 19, 2015

Also I fail to see why the term "keyword" can't be used in a different context here to just imply the left side rather than an ECMAScript keyword. It seems perfectly fine and unambiguous IMO.

@gibson042

So some things questionable about this PR:

  1. it has useless computed: false.
  2. Extending from MemberExpression implies Liskov substitution... Is MetaProperty allowed anywhere a MemberExpression is allowed?

Yes: https://people.mozilla.org/~jorendorff/es6-draft.html#sec-left-hand-side-expressions
Also, computed isn't useless in the analogous SuperProperty (cf. #54 (comment) ).

3. It extends from MemberExpression seemingly for the property names.

It's stronger than that... I would say that it extends from MemberExpression because both the spec and spec authors treat it like one, just with the token preceding . being something other than an IdentifierName (e.g., a Keyword).

4. It lists identifier as the type of the object when they are not objects nor are they identifiers (especially in the case of => which is not even a valid identifier).
5. It ignores the fact that the object property needs to be something other than identifier. We don't currently represent reserved words in the AST because they're embedded in the structure. This is why meta programming is weird. We are tasked to describe the structure in the structure.
6. We must introduce a new type to capture the object here. I say, call it Meta. We can still have the property be an identifier, but Im certain we need a meta type.

{ type: "Keyword", name: string }, extending Node? And adding type: "Operator" (or type: "Punctuator") if/when => becomes a valid meta-object?

@RReverser
Member

Representation of such a simple thing becomes crazy and definitely shouldn't take pages of comments and more than one type.

MetaPropertyMeta
MetaPropertyProperty

This seems crazy and looks like code smell. It's similar to functionNode.body.body.

Agree with @sebmck. Please all remember that 1) we need to have very good reasoning behind introducing every new type, and 2) contextual differences don't count as reasoning for it, since ESTree in opposite to, for example, Shift is non-contextual by design. @mikesherov, I think this is the place where PR with principles you mentioned in #41 (comment) would help a lot with bikeshedding.

It lists identifier as the type of the object when they are not objects nor are they identifiers

Why not? Let's take a look at following examples:

  1. obj.new
  2. obj = {new: 1}
  3. new.target

Now, what is the difference between all of these?

There is none - it's purely contextual. In all these cases new is not treated as keyword (like in new Ctor()) but rather as name, and all the names in ESTree were always represented as Identifier, no matter whether they would be keywords with special meaning in other contexts or not (see examples 1-2).

And there's nothing really special about this particular case to make it different. If things like => will be ever allowed as MetaProperty.meta, it's obviously a different thing and we can easily introduce Punctuator node that wouldn't be bound to this particular case and could be even reused later in CST. So again, contextless types.

We can still have the property be an identifier

I'm glad we at least already agree here, so I won't elaborate on this much, especially given that property isn't even a keyword, just a regular Identifier, so there should be no controversion and need for special MetaPropertyProperty 🐙 .

And I kinda agree with @gibson042, as that's what my initial arguments were, too, but I'm also fine with having different node type with custom property names - whatever works better.

@mikesherov
Contributor

CMIIW, but isn't new.target an error outside of function context? Doesn't that make it not substitutible for any old MemberExpression?

@RReverser
Member

@mikesherov As I said, this particular aspect is not bothering me too much, but if we talk about it - it's again purely contextual difference.

Another example - YieldExpression is also not allowed outside of generator function context, but it's perfectly valid to make it derive from Expression.

It's up to tools not to generate invalid code - there's a lot of ways to do that if that's what you really want 😄

@michaelficarra
Contributor

@gibson042: That is false. See sections 15.1.1 and 15.2.1.1 (in combination with 14.1.4 etc). It is syntactically disallowed outside functions. But that's besides the point.

I'm amazed nobody pointed what I feel is the most compelling reason we cannot use MemberExpression: NewTarget is not a valid SimpleAssignmentTarget. See 12.3.1.5. a.b = 0 and [a.b] = 0 are valid, but new.target = 0 and [new.target] = 0 are not valid, which makes this a terrible representation, since users will always have to check that the MemberExpression is not new.target before putting it in one of these positions.

@mikesherov
Contributor

since users will always have to check that the MemberExpression is not new.target before putting it in one of these positions.

Simply because it extends from MemberExpression doesn't mean it's type in ESTree is MemberExpression. Users won't have to check anything because they'll see the type as MetaMemberExpression. But you're right in the general sense.

@gibson042 , @RReverser , @sebmck The biggest problem is that we should be striving the use extension not simply because of structure or grammar production similarity, but to enforce liskov, and to not have to produce long chains of accepted subtypes.

Remember that we are using types to list what are valid as properties of other types. MemberExpression extends both Expression and Pattern. Do we need to change all types that accept a Pattern in it's substructure to explicitly call out they don't accept MetaMemberExpression? There is prior art here with ImportDeclaration not extending Declaration for exactly this reason.

For example, considering a MetaMemberExpression can't be the child of a Program node, and Program currently has body set to [ statement ], do we need to now list out all of the subtypes of Program or create a new syntax to disallow certain subtypes of Statement?

Another example - YieldExpression is also not allowed outside of generator function context, but it's perfectly valid to make it derive from Expression.

It's up to tools not to generate invalid code - there's a lot of ways to do that if that's what you really want 😄

It's up to the spec to help define what is a valid program. If it weren't, then we could just make everything extend from Node and not worry about typing at all. Our type system is there to enforce program validity. To suggest otherwise is against the spirit of a spec to begin with. ESTree takes a more lax approach than Shift on principle, but this is one case where the advantages of making it extend MemberExpression are outweighed by the downsides.

@RReverser
Member

@mikesherov As for Pattern we already have a lot of problems there - remember #37 (comment)? + the fact that MemberExpression derives from Pattern, but can't be used in VariableDeclarator.

In any case, it's not derivation from MemberExpression what we've been discussing, let's get to the point about necessity for new types vs Identifiers.

@mikesherov
Contributor

I am fine on adding a new Keyword type, and a Punctuator type as needed for the object, and am fine on Identifier as the property.

@mikesherov
Contributor

The reason why we should use a new keyword type is that it isn't contextual, it's gauranteed. Literally, meta properties have to deal with properties on things that are language constructs and not actual objects that have identifiers.

@RReverser
Member

In ESTree, any real use is defined by it's type + (when needed) parent. This is just like with Function.method vs Function+Property.method we were discussing. What does Keyword type give you here what combination of Identifier+looking at parent's type and .object doesn't?

@mikesherov
Contributor

Use case 1: Perhaps a linter is implementing requireUpperCaseIdentifiers. They iterate over all identifier nodes and see => as the value.

Use case 2: scope analysis. I want to make sure all identifiers are declared before use. It sees new as an identifier without it be declared somewhere. Explosion.

Tools are relying on us to report real identifiers as identifiers and nothing else, which is why we have thisExpression and Super and others. Already, all downstream projects have to account for mistakes like not treating arguments specially.

We shouldn't just call something an identifier that isn't. I'm not sure what the advantage to calling it an indentifier is?

@RReverser
Member

They iterate over all identifier nodes and see => as the value.

As I said above - => is not the case for Identifier.

It sees new as an identifier without it be declared somewhere.

You define traversal rules, in any library. So you just don't go inside of MetaProperty (just like you don't go into MemberExpression.property or Property.key when computed: false - those are not identifiers meant to be declared at all).

@RReverser
Member

We shouldn't just call something an identifier that isn't. I'm not sure what the advantage to calling it an indentifier is?

It is identifier - just like in MemberExpression.property and Property.key. In ESTree, Identifier is used for any identifier-like set of chars, not just for names meant to be declared.

@mikesherov
Contributor

So you just don't go inside of MetaProperty (just like you don't go into MemberExpression.property or Property.key when computed: false - those are not identifiers meant to be declared at all).

I suppose that's true. So we're left with creating a Punctuator type should => ever come into existence?

@RReverser
Member

Yup, that's how it looks to me.

@mikesherov
Contributor

So....

interface MetaProperty <: Expression {
    type: "MetaProperty";
    meta: Identifier;
    property: Identifier;
}

with potential for the future:

interface Punctuator <: Node {
    value: string
}

interface MetaProperty <: Expression {
    type: "MetaProperty";
    meta: Identifier | Punctuator;
    property: Identifier;
}

this is rehash of best parts of @all suggestions. With the key points being:

  1. incorrect to have object as the property name, computed was useless, I had concerns about MemberExpression extension, so inherit from Expression instead.
  2. meta as name of property because it is a hit tip to the term "MetaProperty".
  3. Extensible into the future by adding Punctuator Node.
  4. Identifier is fine as traversals will need to examine parent anyway.
@RReverser
Member

Exactly, that's what I suggested. (Though I like meta and property proposed from Esprima's meeting more than context and property, but whatever works)

@mikesherov
Contributor

@RReverser updated.

@mikesherov
Contributor

@RReverser can you do a new PR so we can discuss the new proposal and lose the baggage of this one?

@RReverser
Member

@mikesherov Are you sure we want to lose context and previous discussions?

@mikesherov
Contributor

Nah, you're right. I suppose just update the PR.

@RReverser
Member

(done)

@mikesherov mikesherov and 1 other commented on an outdated diff Mar 19, 2015
@@ -148,3 +148,13 @@ interface ClassExpression <: Class, Expression {
type: "ClassExpression";
}
```
+
+## MetaMemberExpression
@mikesherov
mikesherov Mar 19, 2015 Contributor

## MetaProperty

@RReverser
RReverser Mar 19, 2015 Member

Thanks, will fix.

@RReverser RReverser Add representation for meta properties
ECMAScript gets new syntax thing call "meta properties", which is already used as `new.target` and, according to meeting notes, can be used in future in other places as well (i.e. `yield.input`).
7d356d1
@RReverser
Member

Fixed & rebased.

@gibson042

(by request, long-winded argument for subtyping moved in part to #67 (comment) )

@gibson042 gibson042 referenced this pull request Mar 19, 2015
Merged

Add philosophy to README #67

@RReverser
Member

@mikesherov So... can we merge it?

@mikesherov
Contributor

@RReverser no, we can not yet. It has changed a lot since we discussed it at the esprima team meeting yesterday.

@RReverser
Member

@mikesherov Hm, ok. Let's discuss what's left here so that it's transparent for all participating members. Otherwise we might go a long way with such a simple things (also related to #68).

@mikesherov
Contributor

Basically, just need @ariya to concede that this is an OK direction to take for Esprima. We had a lengthy discussion of this at the team meeting, so want to be sure the discussion here is consumed and understood by esprima team before merging.

@RReverser
Member

Sounds good. Fingers crossed and thanks for staying in touch and constructive discussions - really appreciate it (and sorry for alarmist tone in some messages).

@mikesherov
Contributor

That's what we're all here for.

@ariya
Contributor
ariya commented Mar 20, 2015

LGTM 👍

@RReverser
Member

IIUC, we also need confirmation from @dherman and good to go.

@RReverser RReverser added a commit to ternjs/acorn that referenced this pull request Mar 20, 2015
@RReverser RReverser Add support for meta-properties. 7959d69
@gibson042 gibson042 referenced this pull request Mar 20, 2015
Open

Versioning #69

@RReverser RReverser added a commit to ternjs/acorn that referenced this pull request Mar 20, 2015
@RReverser @marijnh RReverser + marijnh Add support for meta-properties. 864268a
@gibson042

#70 just landed interface Super <: Node as a valid value for CallExpression#callee and MemberExpression#object, so internal consistency would have us represent new (and any future meta-object) similarly here instead of as a generic Identifier.

@mikesherov
Contributor

@gibson042 to be clear, you're suggesting:

interface MetaProperty <: Expression {
    type: "MetaProperty";
    meta: Keyword;
    property: Identifier;
}

interface Keyword <: Node {
    value: string
}

?

Right, this is the strongest argument for a new keyword node type: this.hello has thisExpression as it's object, super.hello has Super as its object, new.target has keyword as its object.

@gibson042

That's a weak form of what I'm suggesting, although Keyword should have a defined type and name: string (instead of value: string) in analogy with Identifier.

It's also worth mentioning that the spec partitions IdentifierName (ESTree's unfortunately-named Identifier) into Identifier and ReservedWord, and ReservedWord further into Keyword (which includes both new and super), FutureReservedWord, null, true, and false, so a ReservedWord interface is also possible. On the other hand, I seriously doubt that any of the other subsets will ever participate in meta properties, since the last three are Literals and therefore already valid in MemberExpression, and future reserved words would become actual reserved words in the definition process.

Honestly, we'd be more in line with the spec (but less with ourselves) if we just added a keyword: string property to Identifier.

@RReverser
Member

Honestly, we'd be more in line with the spec

That's not the goal, see #62 (comment).

(but less with ourselves) if we just added a keyword: string property to Identifier

Explains my thought even further - if we would just have a keyword, dedicated type for Super wouldn't happen and it would be still harder for tools to properly distinguish super from Identifiers without checking value-based property. Same for this and other cases, where having a keyword just as string doesn't help much.

What is good for description of semantics, isn't same good for syntactic manipulations, that's why those deserve separate specs and rules.

@gibson042

I don't see how it's harder to check node.keyword === "super" (which is actually in direct analogy with your linked discussion about VariableDeclaration#kind) than isInstanceOf( node, "Super" ), but then again I have less skin in that game. I just generally see introducing types, especially when it broadens rather than deepens the inheritance tree, as detrimental (if only slightly) to both the comprehension and future flexibility of this specification. I guess #67 will reveal whether or not that view is shared.

I really like this statement, though:

What is good for description of semantics, isn't same good for syntactic manipulations, that's why those deserve separate specs and rules.

Could you document some manipulations that you think are important to make convenient? Assuming agreement, it would provide valuable input in selecting between alternative options for representing syntax.

@RReverser
Member

I don't see how it's harder to check node.keyword === "super" (which is actually in direct analogy with your linked discussion about VariableDeclaration#kind)

The thing is, for traversals the usual and easy way is to have handlers attached to types, so with most of AST traversal tools you can do smth. like following (transformations are dumb and shown just for example purposes):

traverse(ast, {
  VariableDeclaration(node) {
    node.kind = 'var'; // this is very stupid, just to show the point
  },
  Super(node, parent) {
    switch (parent.type) {
      case 'MemberExpression': return memberExpression(this._superNode, identifier('prototype'));
      case 'CallExpression': return this._superNode;
    }
  }
});

That way, it's 1) easy to split handlers by types and 2) you still have groups by type for stuff that can be handled in pretty much the same way (so internal values are details of implementation, but not the main source of how transformation should happen).

So, as you can see, examples that you mentioned are not a real analogy.

For VariableDeclaration, analysis and transformations are really very similar (as they are all declarations for variables) and kind only introduces own differences on which scope stuff belongs to + whether it's mutable or not.

But, i.e., super and this and so on, differ drastically, as they are completely different things from the analysis/transformation perspective, and combination them into one type and forcing everyone to check internal values like keyword: string for all the possible options would make processing much harder and inconvenient.

@gibson042

Thanks; that's very insightful. And if I'm not mistaken, it highlights a gray area around undefined/Infinity/NaN/etc. and arguments—whether they identify their intrinsic values matters, which supports a distinct node type, but they are non-reserved and therefore position-dependent, which supports Identifier. Right? (assume a hypothetical green field if that affects your answer). And for this issue, such a perspective supports a distinct node type for each meta property (e.g., interface NewTarget like in Shift), since they'll all be semantically unique.

And one more question: noting the literal node type values in your example, are we not expecting downstream tools to be aware of and use the type inheritance hierarchy (e.g., ForOfStatement <: ForInStatement <: Statement <: Node)? Because I've been including it in my reasoning, and it's probably a big deal if that's wrong.

@mikesherov
Contributor

@gibson042 downstream tools are not typically aware of the hierarchy.

@michaelficarra
Contributor

@gibson042: There is simply no question about the representation of those identifiers. If parseInt and Date are identifiers, so are each of those. Please don't bring this up again; it serves no value.

I agree with @mikesherov about the hierarchy. I can't think of a single tool that programmatically depends on the hierarchy. It's only used in discussion.

@RReverser
Member

I agree with @mikesherov about the hierarchy. I can't think of a single tool that programmatically depends on the hierarchy. It's only used in discussion.

Well, to be honest, there are ast-types and internal tools in Babel that do have & use stuff like isStatement(node) / isExpression(node) which kind of depends on hierarchy.

@gibson042

I think you missed my point, @michaelficarra. This statement:

But, i.e., super and this and so on, differ drastically, as they are completely different things from the analysis/transformation perspective, and combination them into one type and forcing everyone to check internal values like keyword: string for all the possible options would make processing much harder and inconvenient.

(which I was exploring via thought experiment) argues in favor of a NewTarget type instead of instantiating MetaProperty directly, since tools that care will always have to figure out what the expression is anyway, and handlers usually associate only with leaf types. Though for that matter, it also seems to argue in favor of introducing SuperProperty and SuperCall.

there are ast-types and internal tools in Babel that do have & use stuff like isStatement(node) / isExpression(node) which kind of depends on hierarchy.

Thanks @mikesherov @RReverser, those are exactly the kind of operations I was imagining.

@mikesherov
Contributor

I'm almost certain it won't matter what type the MetaProperty#object is. Because it can't exist on its own, you're always going to be looking at all MetaProperty nodes, and checking the value of the object.

If by "leaf types" you mean types that aren't extended by another, then no, I don't think its safe to assume that leaf types are what get bound to by handlers.

The only safe thing to say is that composites (objects that don't represent actual leafs like identifier) are what usually get bound to.

The only argument that makes pragmatic sense to me about whether or not a distinct type gets created is whether or not is can be traversed independently.

That is, everyone who implements visitor pattern identifies nodes by their type property. If we want to make it easy to traverse the different metaproperties individually, we make a newTarget type (and the fact that it would extend from an abstract MetaProperty would be just for the field definitions). Otherwise, if we want to enable easy traversal of ALL metaProperties, we stick with the proposed PR and have consumers filter out the nodes upon visit.

Note that if we go with the latter, it actually doesn't matter what type we make new (either keyword or identifier) because rules will be need to filter one way or another and the type doesn't matter much.

My preference is the latter because since JS has no type system (nor can one be inferred from a JSON representation of the tree), rather than forcing ast-types and babel to compile a list of which types are children of MetaProperty, they identify which metaProperties to modify through inspection of values.

@RReverser
Member

Thanks @mikesherov, those are exactly the kind of operations I was imagining.

It wasn't @mikesherov's quote :P

As for the topic, I see pros of introducing stuff like interface NewTarget extends MetaProperty {}. This way, we get all of positions of particular identifiers, ability of traversing through any MetaProperty and ability to attach custom handlers to specific types of them.

@RReverser
Member

@mikesherov Is it the same as your preference?

@mikesherov
Contributor

What will the value of the type property of NewTarget be? MetaProperty or NewTarget?

I prefer MetaProperty because my bias is to enable traversal over all MetaProperties rather than having downstream maintain those lists.

This is just my preference.

@gibson042

What will the value of the type property of NewTarget be? MetaProperty or NewTarget?

The only purpose of a NewTarget type would be exposing type: "NewTarget".

But this question really cuts to why I asked about tools using the type hierarchy... knowing that all NewExpressions are also CallExpression (or that all NewTargets are also MetaProperty) can really lead to a better experience, but it sounds like many or most of them don't do it. And if the AST spec is optimized for dumb tools, then the smart ones will sometimes suffer.

@mikesherov
Contributor

I'll say again that I have no real preference either way here.

If @sebmck and other downstream tools want to have to keep a list of what constitutes a MetaProperty rather than being able to iterate over MetaProperties, that's fine too.

@kittens kittens referenced this pull request in babel/babel Mar 26, 2015
Open

Class support should include `new.target` #1088

@dherman
Contributor
dherman commented Mar 26, 2015

👍 to the PR as is.

@kittens kittens merged commit 3560243 into master Mar 26, 2015
@kittens kittens deleted the es6-meta-property branch Mar 26, 2015
@michaelficarra
Contributor

Why was this merged? I thought we were still considering a NewTarget interface, a la Shift.

@kittens
Contributor
kittens commented Mar 26, 2015

@michaelficarra There had been approval from Esprima (@ariya), Acorn (@RReverser) and SpiderMonkey (@dherman).

@mikesherov
Contributor

I think it's worthwhile to continue discussion further. @sebmck I'm interested to here your thoughts on MetaProperty as a concrete node vs. an abstract node, and it's effects on visitor pattern.

@kittens
Contributor
kittens commented Mar 26, 2015

@mikesherov What do you mean by concrete node and abstract node?

@RReverser
Member

@sebmck It's about MetaProperty as just a base node and things like NewTarget as actual types vs type: MetaProperty for all of them (current impl.)

@mikesherov
Contributor

@sebmck the discussion was whether:

Choice 1. NewTarget should be a node type that inherits for MetaExpression, e.g.:

interface MetaProperty <: Expression {
    meta: Identifier;
    property: Identifier;
}

interface NewTarget <: MetaProperty {
    type: "NewTarget";
}

such that all MetaProperties, now and in the future have their own type, which means distinct nodes for visitor pattern, or...

Choice 2. if we do (as merged):

interface MetaProperty <: Expression {
    type: 'MetaProperty'
    meta: Identifier;
    property: Identifier;
}

such that there is one node type for all MetaProperties with respect to visitor pattern, and if you'd like to target a specific meta expression, you'd filter based on the meta and property properties.

Choice 1:
Pro: ability to distinctly visit different MetaProperties.
Con: means binding the same callback to many different node types (and maintaining a list of what node types inherit from MetaProperty)

Choice 2:
Pro: no need to maintain list of types that inherit from MetaProperties, easy to traverse all metaProperties
Con: must filter MetaProperty nodes to find distinct Metaproperties if needed.

@kittens
Contributor
kittens commented Mar 26, 2015

It seems more future proof and easier to just have a single MetaProperty node since this was proposed at the latest TC39 which includes function.callee, function.count, function.arguments, and function.next. Having all of those as separate node types just seems bizzare.

@mikesherov
Contributor

I agree.

@kittens
Contributor
kittens commented Mar 26, 2015

I understand the perspective of separating them into distinct node types since they do fundamentally different operations, but I don't think runtime semantics should be taken into consideration that much when designing the AST structure.

@RReverser
Member

@sebmck On the other hand, checking node.meta.name === 'new' && node.property.name === 'target' is not fun neither.

@kittens
Contributor
kittens commented Mar 26, 2015

@RReverser Abstract it away into helpers then, it's painful to do any AST introspection without them. The Babel equivalent would be t.isMetaProperty(node, { meta: { name: "new" }, property: { name: "target" } }). (Although that's just as long)

@RReverser
Member

Maybe kind? But it would break "uniqueness" principle (btw when are we going to merge those?)

@RReverser
Member

Well yeah, utils help a lot :)

@gibson042

I don't think runtime semantics should be taken into consideration that much when designing the AST structure

@sebmck: That is in precise contradiction to @RReverser's #32 (comment) :

What is good for description of semantics, isn't same good for syntactic manipulations, that's why those deserve separate specs and rules.

And that seems to be the real root of disagreement here.

@RReverser
Member

That is in precise contradiction

Why do you think it's a contradiction?

@gibson042

Because one view considers only syntax in designing the AST structure, while the other also factors in semantics.

@kittens
Contributor
kittens commented Mar 26, 2015

@gibson042 I never said that semantics don't matter.

@mikesherov
Contributor

Guys, let's not devolve here. The theoretical arguments about syntax vs. semantics can go on for days if we don't curb it, and are completely trumped by how people are actually using the AST. The only argument that is currently contentious at the moment (because we've exhausted the rest), is the one about visitor pattern / inspection.

@kittens
Contributor
kittens commented Mar 26, 2015

Here's a real life scenario where splitting up all the meta properties into separate nodes is a terrible idea, at least for this specific scenario.

I have a loop with a let variable:

function foo() {
  for (let num of nums) {}
}

It has a let variable. My block scoping transformer just changes that let to a var since that's the only changes that need to be done for this scenario, producing:

function foo() {
  for (var num of nums) {}
}

But now I add a reference inside a closure to my loop body:

function foo() {
  for (let num of nums) {
    bar(function () { num; });
  }
}

Oh no. Now I need to ensure that num is retained and correct to each loop iteration. I do this by wrapping it in a closure like so:

function foo() {
  for (var num of nums) {
    (function (num) {
      bar(function () { num; });
    })(num);
  }
}

Oh wait, fuck. That means that if someone does:

function foo() {
  for (let num of nums) {
    bar(function () { num; });
    function.count;
  }
}

It'll produce:

function foo() {
  for (var num of nums) {
    (function (num) {
      bar(function () { num; });
      function.count;
    })(num);
  }
}

Shit. It references the wrong function. Looks like I'll have to memoise those references to before the closure to something like:

function foo() {
  for (var num of nums) {
    var _function$count = function.count;;
    (function (num) {
      bar(function () { num; });
      _function$count;
    })(num);
  }
}

Perfect!

Now how would I catch all the function.*?

With node types for each meta property I need to go in and special case every single one.

traverse(node, {
  enter(node) {
    if (node.type === "FunctionCountMeta" || node.type === ...) {}
  }
});

With a single node:

traverse(node, {
  MetaProperty(node) {
    if (node.meta.name === "function") {}
  }
});

Not only does that simplify the visitor logic but it's future proof against new function meta properties.

@kittens
Contributor
kittens commented Mar 26, 2015

Also having separate nodes for each meta property is crap for code generation:

export function FunctionCountMeta() {
  this.push("function.count");
}

export function FunctionCalleeMeta() {
  this.push("function.callee");
}

...

vs:

export function MetaProperty(node, print) {
  print(node.meta);
  this.push(".");
  print(node.property);
}

Hell, you could just share the same code generation method as MemberExpression.

@kittens
Contributor
kittens commented Mar 26, 2015

Also, important to note that ESTree needs to balance the needs of all consumers. A linter doesn't give a shit about the difference between function.callee and function.count so it just adds unnecessary complexity to operations on those nodes.

@kittens
Contributor
kittens commented Mar 26, 2015

Also another important thing to note is that you can do:

new.
target

Combing both of those distinct properties new and target into the same node means you're losing tracking information about where they start and end so a linter can't find the different tokens very easily to verify whitespace etc.

@kittens
Contributor
kittens commented Mar 26, 2015

Apologies for the long narratives but I'm pretty sure all the issues I raised put the nail in the coffin about distinct node types, happy to be proven wrong.

@RReverser
Member

👍 sounds reasonable to me, let's just leave it as is and stop bike-shedding

@gibson042

Tools aware of the type hierarchy can still do all that logic you just described for MetaProperty, and introducing NewTarget <: MetaProperty changes only the value of "type" (i.e., it doesn't eliminate the Identifier properties).

Lack of subtypes, on the other hand, could complicate visitor code if meta properties start being used for very different purposes—new.target and function.callee may have similar semantics, but there's no guarantee that such a basic language production will always be limited to scope introspection.

P.S. Thank you very much for the code examples. They help in clarifying issues more than anything else.

P.P.S.

Hell, you could just share the same code generation method as MemberExpression

I also like such reuse and want to enable more of it, but note that MetaProperty as committed describes the syntax like <meta> . <property> while MemberExpression is <object> . <property>.

@RReverser
Member

Tools aware of the type hierarchy

That's the key. Hierarchy is needed for very few tools, and we shouldn't force all the other tools to complicate their codebase by either checking even more different types or introducing hierarchy.

@kittens
Contributor
kittens commented Mar 26, 2015

(i.e., it doesn't eliminate the Identifier properties).

Right, in the case I must have misinterpreted what was being proposed. If anything that makes it worse since then you have two pieces of linking information that can go out of sync. It's been said before that changes/additions that duplicate metadata and information wont be accepted due to how dangerous they are.

@gibson042

two pieces of linking information that can go out of sync

An excellent point. I'm sold.

@mikesherov
Contributor

I just wanted to say, now, that this thread is concluded, that having discussion until everyone is satisfied is important, and a different activity then getting a thread to the point of mergeability.

While this was a long and arduous thread, the fact that we stuck around and talked it through despite getting heated trumps any frustration any of us experienced.

All members should keep this in mind in further discussions: aside from suggestions for throwing out the AST completely, or seriously breaking es5 BC, we must be willing to fully discuss the points here.

@monolithed

So, what the current status?

@kittens
Contributor
kittens commented Aug 16, 2015

@monolithed Well it was merged?

@monolithed

@sebmck, I don't think so :)

@kittens
Contributor
kittens commented Aug 16, 2015

@monolithed No I think so. I was the one who merged it :) The merge commit is 3560243

screen shot 2015-08-15 at 9 13 54 pm

@monolithed

@sebmck, I got it, could you please explain what are we waiting for babel/babel#1088?

@kittens
Contributor
kittens commented Aug 16, 2015

@monolithed new.target semantics are intricate and I haven't had enough time to determine how much I can replicate them with just static analysis and some trickery. It's not a super high priority.

@monolithed

@sebmck, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment