Skip to content
Ramsay is a Bazel BUILD file generator for Python 2/3.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
ramsay
.gitignore
LICENSE
README.md
requirements.txt

README.md

Ramsay

Ramsay is a Bazel BUILD file generator for Python 2/3 using the pyz_rules rule set.

At a Glance

Installation

$ python2.7 -m virtualenv env
$ . env/bin/activate
$ python2.7 -m pip install --user -r requirements.txt

Usage

$ ln -s $(pwd -P)/ramsay/ramsay.py ~/bin/ramsay
$ export PATH="$PATH:~/bin"  # if ~/bin isn't already in your path
$ cd <directory with .py files>
$ ramsay *.py > BUILD.bazel

Overview

Ramsay is a Bazel BUILD file generator for Python 2/3 code. It (currently) emits Bazel target that use the pyz_rules set for binaries and tests.

Configuration

You can configure Ramsay through command-line arguments and .ramsayrc files. Only a small subset of generator directives can be influenced through the command line. .ramsayrc is where the main action happens.

Directive Description Example Default Command-Line .ramsayrc
workspace_dir Ramsay will automatically discover the Bazel workspace directory. Should this heuristic fail you, you can override the (absolute) path with this option. <automically discovered> yes yes (but watch for portability)
module_aliases Maps the name of a module to another names. { "bson": "pymongo" } {} no yes
ignored_modules Ignores these module imports. [ "uwsgi" ] [] no yes
ignored_files Excludes these files from being parsed. [ "broken_syntax.py" ] [] no yes
ignored_test_files Excludes test targets from being generated for these files. [ "my_test_case_base.py"] [] no yes
manual_imports Applies extra imports to these files. { "uses_futures": "futures" } {} no yes
manual_dependencies Adds dependencies to these files. { "utils.py": "//my/project:file_py" } {} no yes
manual_data_dependencies Adds data dependencies to these files. { "test_foobar.py": "//my/project:fixtures" } {} no yes
manual_tags Adds tags to these files. { "tag_me.py": ["this_is_a_tag"] } {} no yes
manual_sizes Adds test sizes to these files. { "test_mouse.py": "small" } {} no yes
manual_timeouts Adds test timeouts to these files. { "test_gotta_go_fast.py": "short" } {} no yes
manual_flaky Sets the flaky flag for these files. { "test_a_couple_of_times.py": true } {} no yes
pattern_deps Applies extra import to files matched by patterns { "^test_.*\\.py$": { "manual_dependencies": [ "//my/project:file_py" ] } } {} no yes
post_sections Adds free-form text to the generated BUILD file. { "post_sections": [ "# this is a test" ] } {} no yes
third_party_modules Ramsay will query Bazel's dependency graph for third-party modules (see Caveats below). Should this lookup fail you, you can override the list with this option. [ "werkzeug" ] <queried with bazel> no yes
allow_scoped_imports Whether or not to allow scoped imports. true false yes yes
generate_library_targets Whether or not to generate pyz_library targets. true no yes
generate_test_targets Whether or not to generate pyz_test targets. true no yes
generate_shared_library Whether or not to generate pyz_library targets containing all non-test files in the current directory. false no yes
enable_debug Whether or not to raise the log level to debug. false yes yes

Caveats

  • Ramsay invokes bazel query to query the Bazel dependency graph to discover third-party Python modules. These target would've been generated by the pip_generate_wrapper of rules_pyz.
  • Ramsay will ignore scoped imports (imports that don't occur at the top level) by default. This can be enabled by setting allow_scoped_imports to true in the .ramsayrc file or by providing the --allow_scoped_imports command-line options.

Roadmap

This is an informal roadmap of features that we want to see in the future, in order of most desired first.

  • Add pyz2_image/pyz3_image targets: We haven't yet looked into how these targets could be discovered. (A cheap strategy would be manually adding post sections or introducing a new manual_images section to the .ramsayrc file.)
  • Lift restrictions: The generator references and relies on the current working directory being the directory that the files to-be-parsed are in.
  • Add forwards-backwards generation: Parse existing BUILD files and incorporate their contents into the generator so that hand-tailored targets don't get lost. (A possible strategy here is to use Bazel's tags to tag ramsay-generated targets as such.)
  • Refactor to event-driven rules: Increase flexibility of the generator by making rules match on the AST instead of hardcoding rules in the core. Forwards-backwards generation would benefit from this feature.
  • Add rulesets: Generate targets that use [rules_python][https://github.com/bazelbuild/rules_python] and/or other competing rulesets.
  • Add language support: Add support for other programming languages like Go, Rust, etc.

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.

History

At Zenreach, our web UI is a large Django-based Python webapp with several hundreds of files and tests. We tried to hand-tailor our own BUILD files, but this quickly became a monumental task. At first, we collected files from the each directory into a python_shared_library target (mirroring gazelle's go_default_library) and all python_shared_library targets into a top-level python_main_library target. This target was added as a dependency to each test file. This worked for the first 3-4 tests, but failed spectacularly for the remaining tests when we ran out of inodes on ext4-based filesystems. Because bazel symlinks all test dependencies into the runfiles tree, so far the build consumed upward of 7 million inodes. We considered 1.) switching filesystems or 2.) sandboxfs, but found that 1.) put too much of a burden on Linux-based developers, and 2.) didn't work for us as expected.

We realized that we needed to automate the process of parsing Python source code, which would allow us manage dependencies in a fine-grained manner to eliminate the inode problem. Ramsay is the result of this effort.

License

BSD-3-Clause

You can’t perform that action at this time.