SIMPLE_IDENTIFIER is not assigned to import symbols and aliases

[EgorBu]

Hi,
I was playing with UASTs and found some bug (as I think):
minimal reproducible example:

from os import path
import sys

import numpy as np

All SIMPLE_IDENTIFIERS: {'path': 1, 'numpy': 1, 'sys': 1} - as we can see - np and os are missed.

Some helper code to debug:

from collections import Counter

from ast2vec.bblfsh_roles import SIMPLE_IDENTIFIER
from ast2vec.repo2.base import Repo2Base


class Repo2IdModel:
    NAME = "Repo2IdModel"


class Repo2IdCounter(Repo2Base):
    """
    Print all SIMPLE_IDENTIFIERs (and counters) from repository
    """
    MODEL_CLASS = Repo2IdModel

    def collect_id_cnt(self, root, id_cnt):
        for ch in root.children:
            if SIMPLE_IDENTIFIER in ch.roles:
                id_cnt[ch.token] += 1
            self.collect_id_cnt(ch, id_cnt)

    def convert_uasts(self, file_uast_generator):
        for file_uast in file_uast_generator:
            print("-" * 20 + " " + str(file_uast.filepath))
            id_cnt = Counter()
            self.collect_id_cnt(file_uast.response.uast, id_cnt)
            print(id_cnt)


if __name__ == "__main__":
    repo = "test/imports/"
    c2v = Repo2IdCounter(linguist="path/to/enry", bblfsh_endpoint="0.0.0.0:9432")
    c2v.convert_repository(repo)

[juanjux]

Both should be easy to fix. I'll take a look tomorrow.

[EgorBu]

role vis - hope that it may help you to debug roles in future

[EgorBu]

Command to reproduce:
python3 -m uast_playground repo2id_str -r test_data/exp2
Additional experiments:

1. wrong start and end positions of tokens (token has more than 1 line, overlaps, wrong length, etc)
2. empty SIMPLE_IDENTIFIER
   !Wrong length of token '' (expected 0 - 0 + 1 == 0)!!Wrong length of token '' (expected 0 - 0 + 1 == 0)!
3. decorators and function definitions have the same start line
4. arguments to functions are not SIMPLE_IDENTIFIERS (but type for argument is SIMPLE_IDENTIFIER)

[juanjux]

Please open other issues for the other problems.

[juanjux]

About 1: the token of a function is the name, but the start and ending positions are the ones of the function node itself with body and all, not the token. This is the same for all nodes. The current UAST specification doesn't give a position for the tokens except when the token match the length of the node (variable names, etc.)

[juanjux]

About 4:

Function arguments have the role FunctionDeclarationArgumentName. I'm not sure if SimpleIdentifier is needed here since it would be redundant.

[juanjux]

Back to the original issue, I've already fixed the os not having SimpleIdentifier, but the np problem is going to be harder since in the native AST both the numpy and np are inside the same node, with the name (that we take in this case as token) being numpy and np being a property called asname. I'll try to find some workaround but I'm not sure it can be done with the current version of the SDK (we can promote lists of dictionaries to their own nodes, but not strings, I think).

[juanjux]

I'm implementing a feature in the SDK that should fix the second problem; we already have one to optionally promote object properties with list values to child nodes, this one will promote string properties to nodes with the key as node type and the value as token.

[EgorBu]

what about empty SIMPLE_IDENTIFIERs - do you have an idea what it is?

[juanjux]

Write a as small as possible code example that produces that, please.

[juanjux]

Should be fixed by bblfsh/sdk/pull/161 and #64 when merged.