Ramsay

Ramsay is a Bazel BUILD file generator for Python 2/3 using the pyz_rules rule set.

At a Glance

Installation

$ python2.7 -m virtualenv env
$ . env/bin/activate
$ python2.7 -m pip install --user -r requirements.txt

Usage

$ ln -s $(pwd -P)/ramsay/ramsay.py ~/bin/ramsay
$ export PATH="$PATH:~/bin"  # if ~/bin isn't already in your path
$ cd <directory with .py files>
$ ramsay *.py > BUILD.bazel

Overview

Ramsay is a Bazel BUILD file generator for Python 2/3 code. It (currently) emits Bazel target that use the pyz_rules set for binaries and tests.

Configuration

You can configure Ramsay through command-line arguments and .ramsayrc files. Only a small subset of generator directives can be influenced through the command line. .ramsayrc is where the main action happens.

Directive	Description	Example	Default	Command-Line	.ramsayrc
workspace_dir	Ramsay will automatically discover the Bazel workspace directory. Should this heuristic fail you, you can override the (absolute) path with this option.		<automically discovered>	yes	yes (but watch for portability)
module_aliases	Maps the name of a module to another names.	{ "bson": "pymongo" }	{}	no	yes
ignored_modules	Ignores these module imports.	[ "uwsgi" ]	[]	no	yes
ignored_files	Excludes these files from being parsed.	[ "broken_syntax.py" ]	[]	no	yes
ignored_test_files	Excludes test targets from being generated for these files.	[ "my_test_case_base.py"]	[]	no	yes
manual_imports	Applies extra imports to these files.	{ "uses_futures": "futures" }	{}	no	yes
manual_dependencies	Adds dependencies to these files.	{ "utils.py": "//my/project:file_py" }	{}	no	yes
manual_data_dependencies	Adds data dependencies to these files.	{ "test_foobar.py": "//my/project:fixtures" }	{}	no	yes
manual_tags	Adds tags to these files.	{ "tag_me.py": ["this_is_a_tag"] }	{}	no	yes
manual_sizes	Adds test sizes to these files.	{ "test_mouse.py": "small" }	{}	no	yes
manual_timeouts	Adds test timeouts to these files.	{ "test_gotta_go_fast.py": "short" }	{}	no	yes
manual_flaky	Sets the flaky flag for these files.	{ "test_a_couple_of_times.py": true }	{}	no	yes
pattern_deps	Applies extra import to files matched by patterns	{ "^test_.*\\.py$": { "manual_dependencies": [ "//my/project:file_py" ] } }	{}	no	yes
post_sections	Adds free-form text to the generated BUILD file.	{ "post_sections": [ "# this is a test" ] }	{}	no	yes
third_party_modules	Ramsay will query Bazel's dependency graph for third-party modules (see Caveats below). Should this lookup fail you, you can override the list with this option.	[ "werkzeug" ]	<queried with bazel>	no	yes
allow_scoped_imports	Whether or not to allow scoped imports.	true	false	yes	yes
generate_library_targets	Whether or not to generate pyz_library targets.		true	no	yes
generate_test_targets	Whether or not to generate pyz_test targets.		true	no	yes
generate_shared_library	Whether or not to generate pyz_library targets containing all non-test files in the current directory.		false	no	yes
enable_debug	Whether or not to raise the log level to debug.		false	yes	yes

Caveats

• Ramsay invokes bazel query to query the Bazel dependency graph to discover third-party Python modules. These target would've been generated by the pip_generate_wrapper of rules_pyz.
• Ramsay will ignore scoped imports (imports that don't occur at the top level) by default. This can be enabled by setting allow_scoped_imports to true in the .ramsayrc file or by providing the --allow_scoped_imports command-line options.

Roadmap

This is an informal roadmap of features that we want to see in the future, in order of most desired first.

• Add pyz2_image/pyz3_image targets: We haven't yet looked into how these targets could be discovered. (A cheap strategy would be manually adding post sections or introducing a new manual_images section to the .ramsayrc file.)
• Lift restrictions: The generator references and relies on the current working directory being the directory that the files to-be-parsed are in.
• Add forwards-backwards generation: Parse existing BUILD files and incorporate their contents into the generator so that hand-tailored targets don't get lost. (A possible strategy here is to use Bazel's tags to tag ramsay-generated targets as such.)
• Refactor to event-driven rules: Increase flexibility of the generator by making rules match on the AST instead of hardcoding rules in the core. Forwards-backwards generation would benefit from this feature.
• Add rulesets: Generate targets that use [rules_python][https://github.com/bazelbuild/rules_python] and/or other competing rulesets.
• Add language support: Add support for other programming languages like Go, Rust, etc.

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.

History

At Zenreach, our web UI is a large Django-based Python webapp with several hundreds of files and tests. We tried to hand-tailor our own BUILD files, but this quickly became a monumental task. At first, we collected files from the each directory into a python_shared_library target (mirroring gazelle's go_default_library) and all python_shared_library targets into a top-level python_main_library target. This target was added as a dependency to each test file. This worked for the first 3-4 tests, but failed spectacularly for the remaining tests when we ran out of inodes on ext4-based filesystems. Because bazel symlinks all test dependencies into the runfiles tree, so far the build consumed upward of 7 million inodes. We considered 1.) switching filesystems or 2.) sandboxfs, but found that 1.) put too much of a burden on Linux-based developers, and 2.) didn't work for us as expected.

We realized that we needed to automate the process of parsing Python source code, which would allow us manage dependencies in a fine-grained manner to eliminate the inode problem. Ramsay is the result of this effort.

License

BSD-3-Clause