GCC plugin that embeds CPython inside the compiler
Python C HTML C++ CSS Makefile Other
Clone or download
davidmalcolm Filter out -flto from sysconfig's CFLAGS
An rpm build was failing during %test with Python 3.7, built with LTO, due
to "-flto" coming in from sysconfig's CFLAGS in
testcpybuilder.BuiltModule.compile_src, and thus the plugin
unexpectedly running from within lto1 (and failing).

Filter out any "-flto" in testcpybuilder.BuiltModule

+ python3 testcpychecker.py -v

======================================================================
ERROR: test_correct_usage (__main__.PyArg_ParseTupleAndKeywordsTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "testcpychecker.py", line 223, in test_correct_usage
    self.assertNoErrors(src)
  File "testcpychecker.py", line 88, in assertNoErrors
    self.build_module(bm)
  File "testcpychecker.py", line 79, in build_module
    self.compile_src(bm)
  File "testcpychecker.py", line 75, in compile_src
    bm.compile_src(extra_cflags)
  File "/builddir/build/BUILD/gcc-python-plugin-0.16/building-for-python3/testcpybuilder.py", line 98, in compile_src
    raise CompilationError(self)
testcpybuilder.CompilationError:
returncode: 1
  compiling: gcc -o /tmp/tmpo4qiynz3/example.cpython-37m-x86_64-linux-gnu.so -I/usr/include/python3.7m -I/usr/include/python3.7m -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -Wall -Werror -fPIC -shared -fplugin=/builddir/build/BUILD/gcc-python-plugin-0.16/building-for-python3/python3.so -fplugin-arg-python3-script=cpychecker.py -fno-diagnostics-show-caret /tmp/tmpo4qiynz3/example.c
Stdout:
Stderr:
    <built-in>: error: Unhandled Python exception raised within callback
    Traceback (most recent call last):
      File "/builddir/build/BUILD/gcc-python-plugin-0.16/building-for-python3/libcpychecker/attributes.py", line 54, in register_our_attributes
        gcc.define_macro('WITH_CPYCHECKER_RETURNS_BORROWED_REF_ATTRIBUTE')
    ValueError: gcc.define_macro("WITH_CPYCHECKER_RETURNS_BORROWED_REF_ATTRIBUTE") called without a compilation unit
    lto1: fatal error: errors during merging of translation units
    compilation terminated.
    lto-wrapper: fatal error: gcc returned 1 exit status
    compilation terminated.
    /usr/bin/ld: error: lto-wrapper failed
    collect2: error: ld returned 1 exit status
Latest commit ef78bdc Jun 28, 2018
Permalink
Failed to load latest commit information.
docs 0.16 May 4, 2018
examples use separate filenames for each function in examples/show-{gimple|ssa… Feb 22, 2013
gcc-c-api gcc-callgraph.c: fix includes for gimple.h for gcc 4.9 May 4, 2018
gccutils Support C++11/C++14 within get_global_typedef Feb 15, 2016
libcpychecker Avoid multiline warning strings in libcpychecker.formatstrings Feb 13, 2016
libcpychecker_html python 3 fixes Sep 30, 2014
misc/fedora Python3: makeindex.py: fix print-as-function SyntaxError Apr 8, 2012
tests tests/plugin/callbacks/exception: ensure we have a column number May 4, 2018
.gitignore add print-gcc-version to .gitignore Apr 22, 2013
.travis.yml .travis.yml: try with gcc 8 May 4, 2018
COPYING Add COPYING and COPYING.LESSER, for the LGPLv3 Jun 8, 2011
LICENSES.txt fix up references to the moved directory May 19, 2014
Makefile testsuite: add mini-DejaGnu support Feb 14, 2017
README.rst README.rst: update May 4, 2018
configbuilder.py Work-in-progress support for gcc 4.9 Dec 3, 2013
contributors.rst add David Narvaez to contributors.rst Jan 10, 2012
cpybuilder.py mass renaming of python symbols to use PyGcc_CamelCase Feb 15, 2013
cpychecker.py Add GPLv3 license headers to all source files Jun 20, 2011
dejagnu.py testsuite: add mini-DejaGnu support Feb 14, 2017
demo.c add example of refcounting bugs to "make demo" Jan 18, 2013
gcc-python-attribute.c Work-in-progress support for gcc 4.9 Dec 3, 2013
gcc-python-callbacks.c mass renaming of python symbols to use PyGcc_CamelCase Feb 15, 2013
gcc-python-callgraph.c Work-in-progress support for GCC 5 May 6, 2015
gcc-python-cfg.c ensure uniqueness of gcc.Cfg for an underlying gcc_cfg Mar 3, 2013
gcc-python-closure.c mass renaming of python symbols to use PyGcc_CamelCase Feb 15, 2013
gcc-python-closure.h mass renaming of python symbols to use PyGcc_CamelCase Feb 15, 2013
gcc-python-compat.h Conditionalize declaration of dump_generic_node May 4, 2018
gcc-python-diagnostics.c PyGcc_inform: fix -Wformat-security May 4, 2018
gcc-python-docs fix gcc-python-docs Mar 27, 2012
gcc-python-function.c mass renaming of python symbols to use PyGcc_CamelCase Feb 15, 2013
gcc-python-gimple.c Work-in-progress support for GCC 5 May 6, 2015
gcc-python-location.c Fix build problems with gcc < 7 Dec 3, 2017
gcc-python-option.c work-in-progress porting to gcc 4.8 Feb 22, 2013
gcc-python-parameter.c mass renaming of python symbols to use PyGcc_CamelCase Feb 15, 2013
gcc-python-pass.c Fix build against GCC 4.8 May 6, 2015
gcc-python-plugin.spec Add 0.15 release notes; bump versions Feb 15, 2016
gcc-python-pretty-printer.c Work-in-progress support for gcc 4.9 Dec 3, 2013
gcc-python-rtl.c Work-in-progress support for GCC 5 May 6, 2015
gcc-python-tree.c print_integer_cst_to_buf: prevent -Werror=unused-function on gcc 4.9 … May 4, 2018
gcc-python-variable.c mass renaming of python symbols to use PyGcc_CamelCase Feb 15, 2013
gcc-python-version.c mass renaming of python symbols to use PyGcc_CamelCase Feb 15, 2013
gcc-python-wrapper.c gcc_register_root_tab expects a NULL-terminated list of ggc_root_tab Apr 29, 2013
gcc-python-wrappers.h Handle gcc 8's changes to TYPE_FIELDS and TYPE_METHODS Dec 5, 2017
gcc-python.c Update for gcc 8 C++ FE changes to global_namespace Dec 5, 2017
gcc-python.h Fix build problems with gcc < 7 Dec 3, 2017
gcc-with-cpychecker fix mixing tabs and spaces May 20, 2014
gcc-with-python allow the use of $CC to override "gcc" in gcc-with-python and gcc-wit… Mar 27, 2012
generate-callgraph-c.py mass renaming of python symbols to use PyGcc_CamelCase Feb 15, 2013
generate-casts-c.py Attempt to fix the build on gcc 4.6 May 14, 2014
generate-cfg-c.py Ignore exception edges in CFG Jun 25, 2013
generate-config-h.py Work-in-progress support for gcc 4.9 Dec 3, 2013
generate-function-c.py mass renaming of python symbols to use PyGcc_CamelCase Feb 15, 2013
generate-gimple-c.py add a "label" attribute to gcc.GimpleLabel Mar 3, 2013
generate-location-c.py Fix build problems with gcc < 7 Dec 3, 2017
generate-option-c.py mass renaming of python symbols to use PyGcc_CamelCase Feb 15, 2013
generate-parameter-c.py mass renaming of python symbols to use PyGcc_CamelCase Feb 15, 2013
generate-pass-c.py mass renaming of python symbols to use PyGcc_CamelCase Feb 15, 2013
generate-passes-svg.py Work-in-progress support for gcc 4.9 Dec 3, 2013
generate-pretty-printer-c.py mass renaming of python symbols to use PyGcc_CamelCase Feb 15, 2013
generate-rtl-c.py Work-in-progress support for GCC 5 May 6, 2015
generate-tables-of-passes-rst.py Work-in-progress support for gcc 4.9 Dec 3, 2013
generate-tree-c.py Handle gcc 8's changes to TYPE_FIELDS and TYPE_METHODS Dec 5, 2017
generate-variable-c.py mass renaming of python symbols to use PyGcc_CamelCase Feb 15, 2013
gimple-types.txt.in Generate appropriate gcc.Gimple subclasses for each statement; add so… Apr 25, 2011
maketreetypes.py Initial wrapping of the Register Transfer Language Jul 28, 2011
print-gcc-version.c GCC 6: #include fixes Feb 11, 2016
rtl-types.txt.in Initial wrapping of the Register Transfer Language Jul 28, 2011
run-test-suite.py tests/cpychecker/absinterp/comparisons/conditionals: exclude for gcc … May 4, 2018
test-builder.py Derive gcc name from $CC, not $GCC Mar 25, 2012
test.c Add GPLv3 license headers to all source files Jun 20, 2011
test.py Add GPLv3 license headers to all source files Jun 20, 2011
testcpybuilder.py Filter out -flto from sysconfig's CFLAGS Jun 29, 2018
testcpychecker.py Avoid multiline warning strings in libcpychecker.formatstrings Feb 13, 2016
tree-types.txt.in Try a more flexible approach to autogenerating Tree wrapper subclasses Apr 20, 2011
wrapperbuilder.py mass renaming of python symbols to use PyGcc_CamelCase Feb 15, 2013

README.rst

gcc-python

This is a plugin for GCC, which links against libpython, and (I hope) allows you to invoke arbitrary Python scripts from inside the compiler. The aim is to allow you to write GCC plugins in Python.

The plugin is Free Software, licensed under the GPLv3 (or later).

It's still at the "experimental proof-of-concept stage"; expect crashes and tracebacks (I'm new to insides of GCC, and I may have misunderstood things).

It's already possible to use this to add additional compiler errors/warnings, e.g. domain-specific checks, or static analysis. One of my goals for this is to "teach" GCC about the common mistakes people make when writing extensions for CPython, but it could be used e.g. to teach GCC about GTK's reference-counting semantics, or about locking in the Linux kernel, or about signal-safety in APIs.

Other ideas include visualizations of code structure. Given a gcc.CFG instance, gccutils.render_to_dot(cfg) and gccutils.invoke_dot(cfg) will use graphviz and eog to plot a handy visualization of a control flow graph, showing the source code interleaved with GCC's GIMPLE internal representation.

The documentation can be seen at:

http://gcc-python-plugin.readthedocs.io/en/latest/index.html

Requirements

  • GCC: 4.6 or later (it uses APIs that weren't exposed to plugins in 4.5)

    • tested with 4.8, 4.9, 5, 6, 7, and 8.
  • Python: requires 2.7 or 3.2 or later

  • "six": The libcpychecker code uses the "six" Python compatibility library to smooth over Python 2 vs Python 3 differences, both at build-time and run-time:

    http://pypi.python.org/pypi/six/

Usage

I use:

make

to build the plugin and run the tests

You can also use:

make demo

to demonstrate the new compiler errors.

Development has been on x86_64 and I don't know to what extent it will be compatible with other architectures.

There isn't an installer yet. In theory you should be able to add these arguments to the gcc invocation:

gcc -fplugin=python.so -fplugin-arg-python-script=PATH_TO_SCRIPT.py OTHER_ARGS

and have it run your script as the plugin starts up.

The plugin automatically adds the absolute path to its own directory to the end of its sys.path, so that it can find support modules, such as gccutils.py and libcpychecker.

The exact API is still in flux; you can currently connect to events by registering callbacks e.g. to be called for each function in the source at different passes.

It exposes GCC's various types as Python objects, within a "gcc" module. You can see the API by running:

import gcc
help(gcc)

from within a script.

Overview of the code

This is currently three projects in one:

gcc-python-*: the plugin for GCC. The entrypoint (init_plugin) is in gcc-python.c

libcpychecker and cpychecker.py: a Python library (and a driver script), written for the plugin, in which I'm building new compiler warnings to help people find bugs in CPython extension code.

cpybuilder: a handy module for programatically generating C source code for CPython extensions. I use this both to generate parts of the GCC plugin, and also in the selftests for the cpychecker script. (I initially attempted to use Cython for the former, but wrapping the "tree" type hierarchy required more programatic control)

Coding style: Python and GCC each have their own coding style guide for C. I've chosen to follow Python's (PEP-7), as I prefer it (although my code is admittedly a mess in places).

You'll find API documentation within the "docs" directory, written in the reStructuredText format (as is this file, in fact). If you have Sphinx installed (http://sphinx.pocoo.org/), you can regenerate these docs using:

make html

within the docs directory. Sphinx is the python-sphinx package on a Fedora/RHEL box.

More detailed documentation can be seen within docs/getting-involved.rst

Enjoy! David Malcolm <dmalcolm@redhat.com>