Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Python to generate `.travis.yml` #8112

Eric-Arellano opened this issue Jul 26, 2019 · 2 comments · Fixed by #8163


Copy link

commented Jul 26, 2019


While travis.yml.mustache is a big improvement over what we had a year ago—before we used Mustache templates and anchors which prevented any sort of deduplication—the file is difficult to understand. YAML anchors are tough to track, especially when you start using multiple anchors like we currently do. Likewise, the Mustache files are better than what we had before but require jumping to multiple files to see the result.

There is no particular reason we are using YAML + Mustache to generate the final .travis.yml. While the final config file must be in YAML format for Travis to understand it, there is no requirement on what language we use to actually generate the file, so long as it ends up being .travis.yml.


  1. The generated .travis.yml must be functionally identical to the current one, even if the actual YAML files differs. What does this mean? YAML is ultimately a superset of JSON. To check that the current version is identical to the new version, you can run both through, which will convert the YAML into JSON, applying things like anchors for you. We need the JSON to resolve the same for the current version and whatever final version we end up with.
  2. We should stop using anchors / inheritance in the generated .travis.yml. These make it very difficult to understand, and were only used to deduplicate things. We don't care if the generated .travis.yml ends up being 5k lines long due to duplication - more important is that you can go to any individual Travis job and see all of its config in one spot.
  3. The file used to generate should be readable and maintainable.

Suggestion: Use PyYAML

PyYAML allows us to use readable Python to generate a YAML file via its safe_dump() function.

We only pass PyYAML the dictionary we want. For example:

  "conditions": "v1",
  "env": {"global": ["PANTS_CONFIG_FILES=...", ...]},

How is this any better? We can use Python's control flow and abstractions to deduplicate things and make them more readable. For example, we can use functions + dictionary unpacking via ** to define an entry like this:

unit_tests_v2_py36 = {**linux_shard(python_version="3.7", include_test_config=True), **unit_tests_v2_base, "my_special_config": "test"}

This comment has been minimized.

Copy link
Contributor Author

commented Aug 12, 2019

See #8163 for an initial proof of concept. @gshuflin feel free to build off of this - know you're on vacation right now. I gotta get back to teaching the next few days haha.


This comment has been minimized.

Copy link

commented Aug 17, 2019

This seems like a great idea!!! Mustache is the lowest common denominator sometimes but being able to introduce more logic in the language we love to use seems like a big win.

Eric-Arellano added a commit that referenced this issue Aug 20, 2019
patliu85 added a commit to twitter/pants that referenced this issue Aug 20, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
3 participants
You can’t perform that action at this time.