A Concrete Syntax Tree (CST) parser and serializer library for Python
LibCST parses Python 3.0, 3.1, 3.3, 3.5, 3.6, 3.7 or 3.8 source code as a CST tree that keeps all formatting details (comments, whitespaces, parentheses, etc). It's useful for building automated refactoring (codemod) applications and linters.
LibCST creates a compromise between an Abstract Syntax Tree (AST) and a traditional Concrete Syntax Tree (CST). By carefully reorganizing and naming node types and fields, we've created a lossless CST that looks and feels like an AST.
1 + 2
BinaryOperation( left=Integer( value='1', lpar=, rpar=, ), operator=Add( whitespace_before=SimpleWhitespace( value=' ', ), whitespace_after=SimpleWhitespace( value=' ', ), ), right=Integer( value='2', lpar=, rpar=, ), lpar=, rpar=, )
Examining a sample tree
To examine the tree that is parsed from a particular file, do the following:
python -m libcst.tool print <some_py_file.py>
Alternatively, you can import LibCST into a Python REPL and use the included parser and pretty printing functions:
>>> import libcst as cst >>> from libcst.tool import dump >>> print(dump(cst.parse_expression("(1 + 2)"))) BinaryOperation( left=Integer( value='1', ), operator=Add(), right=Integer( value='2', ), lpar=[ LeftParen(), ], rpar=[ RightParen(), ], )
For a more detailed usage example, see our documentation.
LibCST requires Python 3.6+ and can be easily installed using most common Python packaging tools. We recommend installing the latest stable release from PyPI with pip:
pip install libcst
Start by setting up and activating a virtualenv:
git clone firstname.lastname@example.org:Instagram/LibCST.git libcst cd libcst python3 -m venv ../libcst-env/ # just an example, put this wherever you want source ../libcst-env/bin/activate pip install --upgrade pip # optional, if you have an old system version of pip pip install -r requirements.txt -r requirements-dev.txt # If you're done with the virtualenv, you can leave it by running: deactivate
tox -e autofix
To run all tests, you'll need to install tox and do the following in the root:
tox -e py37
You can also run individual tests by using unittest and specifying a module like this:
python -m unittest libcst.tests.test_batched_visitor
See the unittest documentation for more examples of how to run tests.
We use Pyre for type-checking.
To set up pyre check environment:
- Copy the example Pyre config:
cp .pyre_configuration.example .pyre_configuration.
- In the config file, add your venv site-packages dir to "search_path". (e.g. add "/workspace/libcst-env/lib/python3.7/site-packages")
- Remove installed LibCST and install from the source code:
pip uninstall -y libcst pip install -e .
To verify types for the library, do the following in the root:
To generate documents, do the following in the root:
tox -e docs
- Advanced full repository facts providers like fully qualified name and call graph.
LibCST is MIT licensed, as found in the LICENSE file.
- Guido van Rossum for creating the parser generator pgen2 (originally used in lib2to3 and forked into parso).
- David Halter for parso which provides the parser and tokenizer that LibCST sits on top of.
- Zac Hatfield-Dodds for hypothesis integration which continues to help us find bugs.
- Zach Hammer improved type annotation for Mypy compatibility.