Contributing to Dpl
Table of Contents
- Navigating the Codebase
- Lifecycle of the Deployment Process
- Deployment Tooling
- Runtime Dependencies
- Unit Tests
- Runtime Dependency Installation Tests
- Integration Tests
- Testing Dpl Branches or Forks on Travis CI
- Code Conventions
- Naming Conventions
- Updating the README
Dpl is a central component in Travis CI, and has been around for a long time.
This library always has been a community effort first. There probably is not a single person in the world who is very familiar with all deployment providers supported by Dpl.
Thank you all for this!
This document is for you if you are looking to contribute to dpl, be it by adding a new deployment provider, fixing a bug, or adding a new feature.
Dpl has a code of conduct, please follow it in all interactions with the project.
Dpl is written in Ruby, and we assume that you familiarize yourself with our documentation as much as needed.
Hopefully helpful resources are:
Navigating the Codebase
All provider specific classes live in dpl/providers.
These represent the CLI commands that are executed when the command line
dpl is run with a given provider name as the first argument.
Each provider is a subclass of
Dpl::Provider, which is defined in
dpl/provider.rb. The provider base class itself
Cl::Cmd, so it represents an executable sub command of the
For instance, the command
dpl s3 --bucket bucket instantiates and runs the
provider class S3.
Dpl::Provider adds, amongst other things, the order of stages
(methods) that make up the deployment process:
Implementors of concrete provider classes may or may not choose to implement any of these instance methods according to their needs, and semantics of their tooling and service providers. Please refer to Dpl::Provider for details.
The DSL that is used to declare features, dependencies, environment integration
etc. on the concrete provider classes is defined in the module
Dpl::Provider::DSL, in dpl/provider/dsl.
Also of interest is Dpl::Ctx::Bash,
the Bash execution context, that runs shell commands, installs dependencies
etc. (while the
Test context class is used for testing in order to keep your
development machine clean and safe when you run tests locally).
lib └── dpl ├── assets # Stores larger shell scripts ├── ctx │ ├── bash.rb # Bash execution context │ └── test.rb # Test execution context ├── provider.rb # Base class for all providers ├── provider │ ├── dsl.rb # DSL for defining providers │ └── example.rb # Generating example commands for help output └── providers ├── anynines.rb # Concrete providers ├── atlas.rb ├── azure_webapps.rb ├── bintray.rb ├── bitballoon.rb └── ⋮
Lifecycle of the Deployment Process
When a provider class is instantiated and run it will go through a number of stages that make up the deployment process.
These are documented in dpl/provider.rb. If you are adding a new deployment provider please familiarize youself with this lifecycle.
Feel free to pick and interpret these stages according to the needs and
semantics of the service provider you are adding. By no means do all of these
stages have to be filled in or implmented. The
Provider base class checks for
these methods, and runs them, if present, so that implementors can choose
semantically fitting names for their providers.
If you are adding a new deployment provider please choose the tooling you are going to use carefully.
Dpl is a long lived library, and it has outlived many tools that once were supported, and no longer are. Thus tooling stability is a major concern for this project.
Ideally use official CLI tooling supported by the company who's service
provider you are about to add. Often, such CLI tools can be installed via
standard package managers, or manually downloaded using
curl and installed
with a few simple Bash commands.
Such CLI tooling is preferrable over Ruby gem runtime dependencies as they can be executed in a child process, and won't introduce any dependency resolution problems later on.
If no such CLI is available, or it does not look well supported, and your provider implementation needs to talk to an external HTTP API then please consider using Net::HTTP from Ruby's standard library.
If you absolutely have to rely on a runtime Ruby gem dependency, such as a provider client implementation, please only do so if the gem is supported by the respective company officially. We may choose to reject including runtime dependencies that do not look stable or widely supported.
Runtime dependencies can be declared on the provider class using the DSL.
In the case of APT, NPM, and Pip dependencies these will be installed via shell commands at the beginning of the deployment process.
Ruby gem dependencies will be installed using Bundler's inline API, at the beginning of the deployment process, so they are available in the same Ruby process from then on.
Dpl uses RSpec for tests. The specs reside in
spec, and each provider class has a corresponding file
spec/dpl/providers/*_spec.rb to hold tests.
Provider tests should be implemented on an input/output acceptance level, as much as possible.
They use a Ctx::Test execution context in order to avoid running actual shell commands, or actually installing dependencies at test time. There are custom RSpec matchers in place that help with making assertions against this execution context.
If your provider has to talk to an external HTTP API then ideally use Webmock to stub external requests. If by any means possible try to avoid mocking or stubbing Ruby client classes (this is not always possible, but should be considered).
Running Unit Tests Locally
You can run the unit test suite locally as follows:
bundle install bundle exec rspec
In order to execute tests only for a certain provider you can run:
bundle exec rspec spec/dpl/providers/[provider]_spec.rb
In order to execute a single test or group of tests add a line number like so:
bundle exec rspec spec/dpl/providers/[provider]_spec.rb:25
These tests can be run safely on any development machine, anywhere.
Runtime Dependency Installation Tests
We additionally run tests that exercise runtime dependency installation on Travis CI.
These live in .travis/test_install.rb. It is not advisable to run these tests outside of an ephemeral VM or container that can be safely discarded, as they are going to leave various artifacts around.
In order to ensure proper integration with the service providers supported we also periodically run a test suite that exercises actual deployments to these providers.
An integration test consists of:
- A setup script that creates an application (or artifact) to deploy (or upload).
- A YAML config snippet that configures and triggers the deployment as part of a build on Travis CI.
- A test script that tests if the deployment was successful.
creates a minimal Git repository that serves an
index.htmlon GitHub Pages in a temporary directory.
- github-pages/travis.yml configures the build to use Dpl 2.0 to deploy this repository to GitHub Pages.
- github-pages/test tests if the deployment was successful.
The tests can be run on Travis CI individually, or combined, by triggering a build via our API, using the script .travis/trigger. This takes a provider name as an argument, and requires a Travis CI API token.
For example, this triggers a build that executes the GitHub Pages test on Travis CI:
.travis/trigger github-pages --token [token]
The token can also be set as an environment variable:
export TRAVIS_API_TOKEN=[token] .travis/trigger github-pages
trigger script accepts multiple provider names as arguments. If no
arguments are given then tests for all providers will be run.
Integration Test Configuration
In the build config YAML snippet make sure to use the branch of your fork for the deployment tooling, and allow the deployment to run on your branch:
deploy: - provider: [name] edge: source: [your-login]/dpl branch: [your-branch] on: branch: [your-branch]
Ideally use credentials for an isolated account on the service you are deploying to. This is generally good practice, and way you can hand things off to someone else.
In order to get things working encrypt the credentials against your fork, and add them to the build config YAML snippet. If you are in the root directory of your fork then this command should do the trick:
travis encrypt password=[password]
If you do not have the
travis CLI installed you can install it using:
gem install travis
When you add encrypted credentials to the build config YAML snippet also add a comment that allows others to identify the account used. E.g:
deploy: - provider: pages github_token: # personal access token with repo scope on the account [name] secure: "[encrypted token]"
Open a pull request. In order for us to merge your test, and get it working on
our repository you will need to re-encrypt the credentials against
travis-ci/dpl, like so:
travis encrypt -r travis-ci/dpl password=[password]
Whatever minimal deployment you can get working is be a great contribution. Even if for some reason it proves hard to test the deployment in an automated fashion, but you have a successful deployment that can be verified manually, please still open a pull request, and talk to us. Any test is better than no test.
Testing Dpl Branches or Forks on Travis CI
It is possible to test a new deployment provider or new functionality of dpl on
Travis CI. In order to do so, add proper configuraiton on the
edge key to
.travis.yml like so:
deploy: provider: [name] edge: source: [github-handle]/dpl branch: [branch] on: branch: TEST_BRANCH # or all_branches: true ⋮ # rest of your provider's configuration
This builds the
dpl gem on the Travis CI build environment from your
repository, on the given branch. Then it installs the gem built from this code
base, and uses it to run your deployment.
When submitting a pull request, please be sure to run at least one deployment with the new configuration, and provide a link to the build in your pull request.
Dpl does not follow any strict code styleguide.
Please take a look around other providers, and try to follow a similar code style to what you find.
Try to use the DSL as much as possible. It keeps the code declarative and readable, so that people not familiar with Ruby or programming in general can still follow it, and make sense of it.
If you find yourself trying to achieve something that should be, but is not supported by the DSL please open an issue about it.
If you are rather unfamiliar with Ruby, and have trouble following our code style then please submit your pull request anyway, or get in touch, so we can help.
Dpl uses constant names following Ruby naming conventions. I.e. constant
CamelCase, and they live in files named in
If you pick such names for a new provider please try to follow these conventions.
Real world service provider or company names do not always translate to such conventional Ruby names one-to-one. That is ok, they don't have to. These Ruby constant names are representations of real world service and company names in Ruby code.
Other Ruby libraries often (not always) follow a similar thinking. E.g.
even though Amazon Web Services brand name is
AWS the module name
they chose in their aws-sdk is
Updating the README
In order to update the README please edit the template, and run:
gem install ffi-icu bin/readme > README.md