Using pyastgrep as a library

[spookylukey]

Currently we're not documenting any APIs for use as a library, but can do so if there is demand. Please share ideas and use cases you'd like to cover.

At the moment, the printing functionality is seeing a lot of flux, but the search is fairly stable.

[gkapfham]

Hello @spookylukey, thanks for starting this discussion about using pyastgrep as a library! In fact, this is something that I'm very interested in doing. Here is one task that I would like to do with pyastgrep:

• Read in a Python program's source code
• Convert the source code's AST to the XML format
• Run XPATH queries on the XML format to confirm the existence and/or counts of certain code "components" (i.e., functions, if statements in functions, or if statements in while loops in a function)

I would also be interested in building a tool that completes the first two aforementioned steps and then also takes these steps:

• Run XPATH queries to collect certain "components" of a Python program (i.e., all function bodies in a specified module or all if statements in function bodies)
• Perform statement deletion mutation testing on the selected components (i.e., remove a function body from a specified function in a Python module)
• Transform the Python module represented as an XML document back to a Python AST

I should additionally point out that I quickly scanned the documentation for pyastgrep and it was not yet clear to me whether "round trip engineering" was possible. That is, does this tool make it possible to modify the XML-based representation of the AST and then create a new Python program?

So, the main questions I have are:

• Can you offer pointers for using the API of pyastgrep to search and count code constructs?
• Is it possible to use and/or extend pyastgrep to modify Python source code?

Finally, as a reference, I thought it would be useful to point out this tool suite:

https://www.srcml.org/

srcML allows for analysis and manipulation of a program's AST through XML. However, it does not currently support the Python programming language. As such, I would be interested in seeing if it would be possible to use pyastgrep for a similar purpose. Let me know what you think!

Finally, thanks for all of the time and effort that you have invested in the creation of pyastgrep! It is an impressive tool and it seems like I should be able to use it as a building block for several programs that I would like to be able to implement.

[spookylukey]

@gkapfham Thanks for your input, that's all really helpful for the discussion. Regarding your questions:

Can you offer pointers for using the API of pyastgrep to search and count code constructs?

At the moment, it's "Use The Source Luke" - https://github.com/spookylukey/pyastgrep/tree/master/src/pyastgrep . I think you most likely want to do something similar to the search_python_files function. It won't work as it is, however, because if your query expression returns counts instead of XML nodes, it throws an exception (see the test ).

So, I think you can use all the lower level functions (get_files_to_search, parse_python_file, ast_to_xml), but not the search_python_files function itself.

Is it possible to use and/or extend pyastgrep to modify Python source code?

Going from XML to AST should certainly be possible, though not implemented. Going from AST back to Python source is also possible using tools like ast_decompiler . However this has some significantly limitations - AST does not preserve comments or formatting, which means those would be lost on a roundtrip. To avoid that, you'd have to get clever e.g. doing some kind of diff against the original and transformed AST. Tools like LibCST are better suited for code modification tools.

For the code modification needs, I'm not sure how to bridge the gap from querying. Being able to go from an AST node to a LibCST node? Being able to do an XPath query against the structures that LibCST produces?

[gkapfham]

Hi @spookylukey! Thanks for all of these pointers, this is really helpful. 😄

I will start with the functions that you have described and use them as an inspiration for the tool that I plan to implement. At the outset, my tool will work in the same way as pyastgrep does except for the fact that there will be tool-customized XPATHs (along withe the option for a user-provided XPATH) and then counts (with cutoffs, if appropriate) for the code constructs that match the XPATHs.

I'm considering the implementation of a tool like this for two purposes:

1. As a computer science professor I need a tool that can ensure that a Python program that a student submits has a certain number of specified code constructs. The idea is that if the student's program has the specified code constructs, perhaps within a range or a minimum or a maximum, then it would pass the checks provided by my tool. Right now, I do this with regular expressions, which, as you know, are too brittle and not always accurate!

2. As a software engineering researcher, I want to replicate (specific parts of) the empirical study in this journal paper: https://www.sciencedirect.com/science/article/pii/S0164121220302545. Essentially, I want to investigate what is the correlation between the existence/count of certain code constructs, the observability/testability of a Python program, and the mutation score for the program's test suite. While this study has been done for Java programs, to my knowledge there is not an equivalent study for programs implemented in Python.

If I can get a basic prototype working I'll make sure to release it to GitHub and credit you for your assistance and pyastgrep as one of the key building blocks.

Yeah, I was expecting that the "round trip modifications" would be harder with pyastgrep. I'm familiar with libCST and parso, which both address some of the limitations that you mentioned, like preserving comments and formatting. Currently, I am also investigating the refactor framework that has a number of the features I need, even though it works on an AST and not a CST. (Reference: https://github.com/isidentical/refactor).

I should note that, I was investigating pyastgrep (and libCST and parso and refactor and the ast package) so that I could implement a new type of mutation testing tool for the Python programming language. I'm aware of a bunch of existing mutation testing tools for Python but they don't provide the types of operators that I plan to build and, in fact, some of them don't always work as I would expect for certain programs that I've tried.

It is also worth point out that, some of the features that a CST provides (like preservation of comments) are not something that is always important from a mutation testing perspective. As long as I can create the syntactically correct mutant and then run the test suite against it and check the outcomes from testing, then I currently have less interest (at least right now) in whether or not the comments are precisely the same.

Okay, thanks for your insights, I really appreciate them! If I get stuck when it comes to using get_files_to_search, parse_python_file, and ast_to_xml, can I ask questions here?

Finally, from your perspective, what are the benefits associated with using pyastgrep instead of implementing the counting functions directly using ast or libcst or parso or refactor? From my vantage point the big benefit is that I can express the queries in XPATH that my tools can take as an input. This makes my tools more configurable by the user, which is beneficial both for the teaching and research applications that I have in mind. Are there other considerations as well?

[spookylukey]

Okay, thanks for your insights, I really appreciate them! If I get stuck when it comes to using get_files_to_search, parse_python_file, and ast_to_xml, can I ask questions here?

Sure no problem

Finally, from your perspective, what are the benefits associated with using pyastgrep instead of implementing the counting functions directly using ast or libcst or parso or refactor? From my vantage point the big benefit is that I can express the queries in XPATH that my tools can take as an input.

That's the big one for me - if you don't use XPath as a query language, you'll need some other kind of query language, or have to write code that manually traverses a tree of AST nodes to find what you want. Depending on how many queries you need to write, that could be a lot of repetitive code.

I guess if you are going for transformations, then having the code in XML format also brings the advantage of being able to use XSLT.

[spookylukey]

Thinking about this a bit more - I think XSLT would be a pretty interesting option for transforming Python code. It's ages since I've used it, and it's not the easiest to get into, but it is a powerful language designed exactly for the use case of transforming documents, so it might work pretty well for code.

I'd definitely be interested in extending pyastgrep with something like an --apply-xslt option that would do transforms on every match and write the document back as Python code. This would still have the issue that it would lose comments and whitespace info since they are not part of the AST, but as you say it might not matter for your use case.

[spookylukey]

asttokens looks like another options for going from AST back to text. The astroid library also looks like it might be useful for refactoring tasks.