proposals: Add new BIP-003 proposal, Agglutinative Roles language

[abeaumont]

Part of bblfsh/sdk#167

[juanjux]

Missing comma after case.

[juanjux]

The current proposal.

[juanjux]

Nitpicking: at first I tough about logging, maybe change for natural logarithm operator?

[juanjux]

lacks a specific

[abeaumont]

Changed to ** (pow), to avoid potential confusion.

[juanjux]

Comparison

[juanjux]

Incompatible

[juanjux]

Python and Go (uppercase).

[juanjux]

Visibility

[juanjux]

Good job!

About possible improvements, I miss a note in the Impact section about the current users of the UAST having to update their code and the way they interpret the roles.

Also, a third alternative could be the "two-level UAST" I proposed some time ago, where the first level roles are the more generic (loop, branch, procedure) and a second level would have a more concrete semantic meaning (foreach, if, coroutine). But it is probably tangential for this proposal (both levels could use agglutinative roles) so it doesn't need to be added.

Other than that, this BIP has my ACK for the main proposal.

[abeaumont]

Updated with typo and grammar fixes from the review.

[mcuadros]

we don't have a Class role?

[juanjux]

That would be TypeDeclaration or Type+Declaration with the new roles.

[ajnavarro]

In this or other cases, can be a possibility that a role with specific properties can match one by one properties from another role and be nodes slightly different?

[abeaumont]

For nodes with incomplete roles, that can certainly be the case. You could have unsupported operators (let's say pow and log), that you know they're arithmetic operators, but cannot distinguish between them. You'd have to go and check the token, if available, in that case.

[eiso]

First of all, thank you for the work on bip-3. It's a really interesting and very well written proposal. Having been giving it some thought today it makes a lot of sense to me (de)composing the roles, for the reasons that you state.

The one concern that I have, is around higher level abstractions on the roles for easy usability (what @juanjux I believe calls 2 level roles), in particular in the spark-api project. Since that project will be the way a large # of our intended audience will use babelfish, with the composed roles I can imagine some difficulties.

Here is some pseudo code from the Spark API design document.

//DS - PySpark
//for 1000 Python repositories (repos.txt or LanguageDataset demo)
src-d.select()

//clone to local FS (or use .siva files files in hdfs://)
src-d.clone(repos.txt, “/path/to/cloned/repos”)

//get UASTs for HEAD
uasts = src-d.read.gitLocal(“/path/to/cloned/repos”)
  .getReferences().filter($“reference_name” === “HEAD”)
  .getFiles()
  .applyEnry()
  .applyBblfsh()

//extract SimpleIdentifier roles (ids)
uasts.filter("uast uast_lib('//*[@class=SimpleIdentifier]')")

uast_lib is essentially libuast as you can see here. If instead of being able to extract all simple identifiers, I know need a rule based system for roles (e.g. include x, but excludes y, z). I can imagine the syntax becoming difficult, less easy to get started with and also more intense to process on the Spark side. That's why I am tagging @ajnavarro @bzz and @erizocosmico here as well.

The above could be partially solved with the two-level UAST suggestion of @juanjux .

[abeaumont]

@eiso that is a very valid concern, and I see now that I forgot to give a proper answer to @juanjux suggestion, sorry.
I've two comments on this topic:

1. I think what both @juanjux and @eiso are suggesting is just a particular case of the alternative approach I included at https://github.com/bblfsh/documentation/pull/82/files#diff-b3182c12132bb09dddfe38e712ae248bR289, which just talks about a categorization in general. The current proposal doesn't deepen in that approach for the reasons commented there, but if you consider otherwise, we can go deeper into that.
2. It's true that with this proposal search may become more complex but apart from categorization there are two additional tools we can use to solve them:

• Use appropiate roles to facilitate code analysis. It may be true that a code analyst may want to look for simple identifiers and doesn't want to get the qualified identifiers. So let's say that's the case for discussion's sake. Then we could add a Simple role to handle this properly and make analyst's life easier. I think the new approach makes this easier. Look for example at the arithmetic operator example I used, it would make it easier to look for arithmetic operators instead of all of them. The same way we can add roles for say, relational operators, control flow constructs, looping constructs, etc.
• Use the power of xpath syntax to do the search. Even if we missed some valid searches needed by code analysts with our roleset, I think it would still be a reasonable approach (at the usability level) to use something like: //*[@roleIdentifier and not(@roleQualified)]. Not sure about performance, we may need to check that.

Note that these points are presented to have a wider view of the possibilities, not to discourage a categorization (of two levels or otherwise), which I consider a valid approach.

[eiso]

@abeaumont regarding topic 1, could you take your suggested approach and have a 'category type' role that gets added in the same manner. Or would this be mixing concepts?

[abeaumont]

@eiso I'm not sure I understand what you mean, do you mean having a special field/attribute in a node named type which would contain the main category of a node? Something like:

internalType: SimpleIdentifier
type: Identifier
roles: Simple, OtherRole, ...

If that's what you mean, yes, that could be a way to do it. If not, please elaborate a bit on your appoach.

[eiso]

@abeaumont the options I meant were:

Option 1. When ForEach becomes For, Iterator, adding Loop as a sub-role. So having (For, Iterator, Loop). Where Loop is defined in the spec as a higher level role.

Option 2. ...or having (For, Iterator, Loop) but typed as:

role: For
type: level2

role: Iterator
type: level2

role: Loop
type: level1

Please ignore the terrible naming.

[abeaumont]

@eiso ok, I understand now. I think both options would be similar from a Babelfish point of view. I think we could add automatic support for option 2, to make role 'level' explicit, without much work, it would just be more verbose.

So the main question would be if this approach would be of any use for code analysis. I think an analyst would need to know the roles and their categories beforehand anyway, but you surely have a better code analysis perspective and I guess you may have some use case in mind where this kind of annotation would be of help?

[juanjux]

, at the end.