Support for Python

[macsj200]

Python has a rich AST module. We can most likely render this into an abstract representation pipe it into the existing infrastructure in this library. How might one go about adapting your code to do this?

Edit:
I saw your emailed suggestion to use a serialization format to read it into Java. I think I can render the tree into JSON, could I hand that off to your library?

I found ast2json, but it looks a little outdated. There's also this python-astexport github project, I'll look into these.

[jrfaller]

Yes I think this would be a good idea. GumTree has an internal XML format to load ASTs so the only thing required is to write a script that outputs a XML in this format. Then you need to write a tree generator that reads that standard output. You can look how it is done in the gen.c module that works exactly like that.

[macsj200]

Would it be possible to just read in Json directly using something like google's gson library for example? Alternatively, maybe we can use something like jython or pyjamas to traverse the ast using python native APIs from Java, similar to the JavaScript rhino approach.

[jrfaller]

I am not sure that there is already a json internal format for GumTree, but it would not be hard to implement. Why it is easier to generate Json?

[HalleyYoung]

Hi,
I'm very interested in using GumTree for my research involving comparing python programs. I already have a representation of Python code in XML. How difficult would it be for me to try to add Python functionality? Or if someone is working on this already, can I be of any use? :-)

[jrfaller]

Hi! This is very cool since python support is top wanted feature. If you have an XML representation of a python program you are halfway through, there is only the step to convert the XML to GumTree's Tree format remaining. Maybe you can post a sample of an XML representation, I will check if there are is all the required information. BTW how do you obtain this representation? Via a Python program?

[HalleyYoung]

I've attached an example file, saved as a text file to fit with Github's requirements. Basically, the syntactic parts are the XML tags, and the individual values (such as the number "1" in a Num element), are stored under the attribute "value." The program to generate the XML from a python file is a python program.
example_python.txt

[jrfaller]

OK that's cool there is almost everything we need, it only lacks the position of the nodes (absolute or line/character) in the original source code text file, do you think it's possible to add that?

[HalleyYoung]

How is this?
example_python.txt

[jrfaller]

OK this is really nice

The only thing lacking is the end position but if this information is not present in python's parser, it can be deduced from the position of the children and sibilings. Once you have all this information, you can take a look at then gen.srcml that basically do the same thing as we want to do except that it is based upon the XML produced by srcml.

[HalleyYoung]

Is there an example of the format of the XML that needs to be used (for example, a sample Java program in XML format)?

[jrfaller]

You do not have to produce one particular format, as each tree generator will furnish the "decoder" to produce a GumTree AST