Skip to content

Latest commit

 

History

History
362 lines (249 loc) · 30.7 KB

CONTRIBUTING.md

File metadata and controls

362 lines (249 loc) · 30.7 KB
Raw

Contributing Guide

First of all, thanks for visiting this page 😊 ❤️ ! We are stoked that you may be considering contributing to this project. You should read this guide if you are considering creating a pull request or plan to modify the code for your own purposes.

Table of Contents

Code of Conduct

This project and everyone participating in it is governed by the Code of Conduct. By participating, you are expected to uphold this code. Please report unacceptable behavior to help@megabyte.space.

Overview

All our Dockerfiles are created for specific tasks. In many cases, this allows us to reduce the size of the Dockerfiles by removing unnecessary files and performing other optimizations. Our Dockerfiles are broken down into the following categories:

  • Ansible Molecule - Dockerfile projects used to generate pre-built Docker containers that are intended for use by Ansible Molecule
  • Apps - Full-fledged web applications
  • **CodeClimate - CodeClimate engines
  • CI Pipeline - Projects that include tools used during deployments such as linters and auto-formatters
  • Software - Docker containers that are meant to replace software that is traditionally installed directly on hosts

Style Guide

In addition to reading through this guide, you should also read our Docker Style Guide template repository which outlines implementation methods that allow us to manage a large number of Dockerfile projects using automated practices.

Philosophy

When you are working on one of our Dockerfile projects, try asking yourself, "How can this be improved?" By asking yourself that question, you might decide to take the project a step further by opening a merge request that:

  • Reduces the size of the Docker container by converting it from a Ubuntu image to an Alpine image
  • Improves the security and reduces the size of the Docker container by including a DockerSlim configuration
  • Lints the Dockerfile to conform with standards set in place by Haskell Dockerfile Linter

All of these improvements would be greatly appreciated by us and our community. After all, we want all of our Dockerfiles to be the best at what they do.

Choosing a Base Image

  • Whenever possible, use Alpine as the base image. It has a very small footprint so the container image downloads faster.
  • Whenever possible, choose an image with a slim tag. This is beneficial when, say, Alpine is incompatible with the requirements and you must use something besides an Alpine image.
  • Avoid using the latest tag (e.g. node:latest). Instead use specific versions like node:15.4.2. This makes debugging production issues easier.
  • When choosing a base image version, always choose the most recent update. There are often known vulnerabilities with older versions.
  • If all else fails, feel free to use other base images as long as they come from a trusted provider (i.e. using ubuntu:latest is fine but using bobmighthackme:latest is not).

Requirements

Before getting started with development, you should ensure that the following requirements are present on your system:

Although most requirements will automatically be installed when you use our build commands, Docker should be installed ahead of time because it usually requires a reboot.

Getting started should be fairly straight-forward. All of our projects should include a file named start.sh. Simply run, bash start.sh to run a basic update on the repository. This should install most (if not all of the requirements).

Other Requirements

Here is a list of a few of the dependencies that will be automatically installed when you use the commands defined in our Taskfile.yml files

  • DockerSlim - Tool used to generate compact, secure images. Our system will automatically attempt to build both a regular image and a DockerSlim image when you adhere to the guidelines laid out in our Docker Style Guide. DockerSlim allows us to ship containers that can be significantly smaller in size (sometimes up to 90% smaller).
  • container-structure-test - A Google product that allows you to define and run tests against Dockerfile builds. It is used to perform basic tests like making sure that a container can start without an error.
  • GitLab Runner - Tool that allows for the simulation and execution of GitLab CI workflows. It can run tests / scenarios defined in the .gitlab-ci.yml file.

Having an understanding of these tools is key to adhereing to our Docker Style Guide and ensuring that each project can integrate into our automation pipelines.

Getting Started

To get started when developing one of our Dockerfile projects (after you have installed Docker), the first command you need to run in the root of the project is:

bash start.sh

This command will ensure the dependencies are installed and update the project to ensure upstream changes are integrated into the project. You should run it anytime you step away from a project (just in case changes were made to the upstream files).

Creating DockerSlim Builds

Whenever possible, a DockerSlim build should be provided and tagged as :slim. DockerSlim provides many configuration options so please check out the DockerSlim documentation to get a thorough understanding of it and what it is capable of. When you have formulated and fully tested the proper DockerSlim configuration, you will need to add to the blueprint section of the package.json file. More details on this are in our Docker Style Guide.

How to Determine Which Paths to Include

In most cases, the DockerSlim configuration in package.json will require the use of --include-path. If you were creating a slim build that included jq, for instance, then you would need to instruct DockerSlim to hold onto the jq binary. You can determine where the binary is stored on the target machine by running:

npm run shell
which jq

You would then need to include the path that the command above displays in the dockerSlimCommand key of blueprint section in package.json. The package.json might look something like this:

{
  ...
  "blueprint": {
    ...
    "dockerSlimCommand": "--http-probe=false --exec 'npm install' --include-path '/usr/bin/jq'"
  }
}

Determining Binary Dependencies

Note: The method described in this section should not usually be required if you use the --include-bin flag (e.g. --include-bin /usr/bin/jq). The --include-bin option will automatically perform the steps outlined below.

If you tried to use the dockerSlimCommand above, you might notice that it is incomplete. That is because jq relies on some libraries that are not bundled into the executable. You can determine the libraries you need to include by using the ldd command like this:

npm run shell
ldd $(which jq)

The command above would output something like this:

	/lib/ld-musl-x86_64.so.1 (0x7fa35376c000)
	libonig.so.5 => /usr/lib/libonig.so.5 (0x7fa35369e000)
	libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7fa35376c000)

Using the information above, you can see two unique libraries being used. You should then check out the slim build to see which of the two libraries is missing. This can be done by running:

echo "***Base image libraries for jq***"
npm run shell
cd /usr/lib
ls | grep libonig.so.5
cd /lib
ls | grep ld-musl-x86_64.so.1
exit
echo "***Slim image libraries for jq***"
npm run shell:slim
cd /usr/lib
ls | grep libonig.so.5
cd /lib
ls | grep ld-musl-x86_64.so.1
exit

You should then compare the output from the base image with the slim image. After you compare the two, in this case, you will see that the slim build is missing /usr/lib/libonig.so.5 and /usr/lib/libonig.so.5.1.0. So, finally, you can complete the necessary configuration in package.json by including the paths to the missing libraries:

{
  ...
  "blueprint": {
  ...
    "dockerSlimCommand": "--http-probe=false --exec 'npm install' --include-path '/usr/bin/jq' --include-path '/usr/lib/libonig.so.5' --include-path '/usr/lib/libonig.so.5.1.0'"
  }
}

Using a paths.txt File

In the above example, we use --include-path to specify each file we want to include in the optimized Docker image. If you are ever including more than a couple includes, you should instead create a line return seperated list of paths to preserve in a file named local/paths.txt. You can then include the paths in the dockerSlimCommand by using utilizing --preserve-path-file. The dockerSlimCommand above would then look like this if you create the local/paths.txt file:

{
  ...
  "dockerSlimCommand": "--http-probe=false --exec 'npm install' --preserve-path-file 'local/paths.txt'"
}

Basics of CodeClimate

CodeClimate is an eco-system of plugins that inspect code and report possible errors. It combines many different linting engines into a single tool. It works by providing a CLI which spins up a Docker instance for each plugin that is included in the .codeclimate.yml configuration. Because it uses Docker to spin up each linting instance, the Docker instance has to support Docker-in-Docker to function appropriately.

There is somewhat of a learning curve. To save some time, it is well worth it to read through the Getting Started section in their help guide.

It is also imperative that once you have a grasp of how CodeClimate works that you read the CodeClimate Spec which contains the technical details (including the format you should use when returning data). You should make your best effort to return as much useful data as possible using the CodeClimate Spec. However, sometimes it is not feasible (like in a case where you need to label 500 different linting rules with a category). At the very minimum, the engine should work flawlessly with GitLab's implementation of CodeClimate. You can view information (including sample data) on GitLab's Code Quality help guide.

Developing a New CodeClimate Engine

Developing a CodeClimate engine basically boils down to a few key elements:

  1. Creating a Dockerfile which includes your linting engine and your custom scripts that adapt the linting engine for CodeClimate (sample Dockerfile)
  2. Developing scripts that parse the CodeClimate configuration and report the results in the CodeClimate format
  3. Including a binary-like file that serves as the entrance point for the scripts (sample)

You should also fill out the information in the blueprint section of the package.json file. Please refer to some of our other CodeClimate engines for examples of how this section should look. Our automated build scripts will read from package.json to populate the engine.json file which is kept in the local/ folder.

You can find samples of CodeClimate engines we maintain by checking out our CodeClimate Docker engine group on GitLab. You can also browse through the source code of official CodeClimate engines by browsing through CodeClimate's GitHub.

When developing any project (including a CodeClimate engine) that is part of our eco-system, you should also make sure you structure the project according to the guidelines in our Docker Style Guide. Using the same design patterns across all of our repositories helps with automation and testing so it is very important.

How to Run CodeClimate

To run CodeClimate, add a .codeclimate.yml file in the root of the repository you would like to inspect (locally, if you wish). Then run codeclimate analyze. Be sure to checkout codeclimate --help for some other useful commands. The following codeclimate.yml file would run the ESLint and ShellCheck CodeClimate engine:

engines:
  eslint:
    enabled: true
  shellcheck:
    enabled: true
exclude_paths:
- "node_modules/**"

CodeClimate Configuration

Each of our CodeClimate engines should support the ability to specify the location of the linting tool's configuration. The path of this configuration should be relative to the root of the repository. So, for instance, if the .codeclimate.yml file in the root of the repository (or anywhere, if the location is specified to the CodeClimate CLI) looks like this:

---
engines:
  eslint:
    enabled: true
    config:
      path: .config/eslint.js
  duplication:
    enabled: true
    config:
      path: .config/duplication.js
      languages:
        - javascript
        - python
exclude_paths:
- "node_modules/**"
- "test/**"
- "integration/**"

You can see that we have assigned two new fields called path under each plugin's config section. Each of our CodeClimate engines should run with either no configuration path specified or with one specified. The .codeclimate.yml file might not always be in the root of the repository though so you cannot parse the user's .codeclimate.yml file. Instead, you will have to check the /config.json file that gets injected into the Docker container that the CodeClimate CLI spins up to run each plugin's linter. For the ESLint plugin above, the /config.json file will look something like this:

{
  "config": {
    "path": ".config/eslint.js"
  }
  "include_paths": [
    ...
    ...
  ]
}

TODO: To improve this documentation, include an actual example of the /config.json which might be useful to determine what include_paths is and how it can be useful. It would also be nice to see the other pieces of data that /config.json contains.

Using the /config.json file placed in the root of the container, you can access the appropriate settings regardless of where the .codeclimate.yml file is loaded from. Judging from other published repositories, you should include logic that handles the case where /config.json is not available though as well. You should assume that the /config.json and all of its data is optionally included. Whenever you think there is a useful option that you would pass directly to the engine's linter tool, you should include an option that is stored under the config of that particular linter. Whenever you do this, please add a section, brief note, and .codeclimate.yml sample to the docs/partials/guide.md file.

The Dockerfile

There needs to be a Dockerfile included in each CodeClimate engine project. This Dockerfile should pre-install all the dependencies of the engine as well as perform optimizations to slim down the image size. The Dockerfile should strictly follow the same format outlined in our Docker Style Guide. When designing an engine, please read the guide, add the tests it asks for, and refer to our other CodeClimate engines if you need examples.

It is important to note that when the CodeClimate engine runs it will load all of the code in the current (or specified) directory into the /code directory. The /work directory you see in the Dockerfile is just an arbitrary folder used when building the image. So, for example, if you are constructing the path to the configuration mentioned above, you would combine /code + the config.path defined in the /config.json which is only available when the CodeClimate engine is being run.

Testing

Testing is an extremely important part of contributing to this project. Before opening a merge request, you must test all common use cases of the Docker image. The following chart details the types of tests, when they are required, and provides examples:

Test Type Test Description Required Example
Docker.test.yml ContainerStructureTest If the project has a Docker.test.yml file defined, then a ContainerStructureTest will be run on both the regular and slim build (if there is one). If there are not multiple build targets defined in the Dockerfile, then you should use this type of test (by naming the test Docker.test.yml). Ideally, you should leverage some sample project files which should be stored in a folder in the test/ directory. Required when the Dockerfile has 1 build target Example Docker.test.yml
Multiple Docker.target.test.yml ContainerStructureTests If the project has multiple build targets defined in the Dockerfile, then multiple ContainerStructureTests are required. Each one should be named Docker.target.test.yml where target is replaced with the build target name defined in the Dockerfile. This will also test both the regular and slim builds. Similar to the previous test type, it is preferrable to make use of test files stored in a directory in the test/ directory. Required when the Dockerfile has multiple build targets Example Repository
test-output Tests For each folder in the test-output directory, both the regular and slim build will be run in the directory. The container is run by running docker run -it image:tag .. If the console output of the regular build does not match the output of the slim build, then this test will fail. ContainerStructureTests are preferrable but this test type is provided for cases where ContainerStructureTests might not suite your needs. Optional Coming soon.. maybe..
CodeClimate CLI Test This test is unique to CodeClimate Docker projects. For each folder in the test/ directory that begins with codeclimate, the CodeClimate CLI will run with options specified in the lint:codeclimate task. Before running the CodeClimate CLI, our slim custom versions of the CodeClimate engines will be pulled from DockerHub and then tagged as codeclimate/engine:latest. Required for CodeClimate engine projects Example codeclimate folder
GitLab Runner Test This test will ensure that the container can be run on GitLab CI by using a local instance of a GitLab Runner. The GitLab Runner test simulates GitLab CI pipeline steps by running each stage defined in .gitlab-ci.yml that starts with integration:. Required for any project that might be run on GitLab CI Example .gitlab-ci.yml

Note: If you are interested in testing GitHub Actions, then you might be interested in act. This is not integrated into our automated test flow but you can install it by running task install:software:act.

Testing DockerSlim Builds

It is especially important to test DockerSlim builds. DockerSlim works by removing all the components in a container's operating system that it thinks are unnecessary. This can easily break things.

For example, if you are testing a DockerSlim build that packages ansible-lint into a slim container, you might be tempted to simply test it by running docker exec -it MySlimAnsibleLint ansible-lint. This will ensure that the ansible-lint command can be accessed but that is not enough. You should also test it by passing in files as a volume and command line arguments. You can see an example of this in the Ansible Lint repository.

It is important to test all common use cases. Some people might be using the ansible-lint container in CI where the files are injected into the Docker container and some people might be using an inline command to directly access ansible-lint from the host.

Testing Web Apps

When testing Docker-based web applications, ensure that after you destroy the container along with its volumes you can bring the Docker container back up to its previous state using volumes and file mounts. This allows users to periodically update the Docker container while having their settings persist. This requirement is also for disaster recovery.

Linting

We utilize several different linters to ensure that all our Dockerfile projects use similar design patterns. Linting sometimes even helps spot errors as well. The most important linter for Dockerfile projects is called Haskell Dockerfile Linter (or hadolint). We also use many other linting tools depending on the file type. Simply run git commit to invoke the pre-commit hook (after running bash start.sh ahead of time) to automatically lint the changed files in the project.

Some of the linters are also baked into the CI pipeline. The pipeline will trigger whenever you post a commit to a branch. All of these pipeline tasks must pass in order for merge requests to be accepted. You can check the status of recently triggered pipelines for this project by going to the CI/CD pipeline page.

Pull Requests

All pull requests should be associated with issues. You can find the issues board on GitLab. The pull requests should be made to the GitLab repository instead of the GitHub repository. This is because we use GitLab as our primary repository and mirror the changes to GitHub for the community.

Pre-Commit Hook

Even if you decide not to use npm run commit, you will see that git commit behaves differently because there is a pre-commit hook that installs automatically after you run npm i (or bash start.sh). This pre-commit hook is there to test your code before committing and help you become a better coder. If you need to bypass the pre-commit hook, then you may add the --no-verify tag at the end of your git commit command and HUSKY=0 at the beginning (e.g. HUSKY=0 git commit -m "Commit" --no-verify).

Style Guides

All code projects have their own style. Coding style will vary from coder to coder. Although we do not have a strict style guide for each project, we do require that you be well-versed in what coding style is most acceptable and best. To do this, you should read through style guides that are made available by organizations that have put a lot of effort into studying the reason for coding one way or another.

Recommended Style Guides

Style guides are generally written for a specific language but a great place to start learning about the best coding practices is on Google Style Guides. Follow the link and you will see style guides for most popular languages. We also recommend that you look through the following style guides, depending on what language you are coding with:

For more informative links, refer to the GitHub Awesome Guidelines List.

Strict Linting

One way we enforce code style is by including the best standard linters into our projects. We normally keep the settings pretty strict. Although it may seem pointless and annoying at first, these linters will make you a better coder since you will learn to adapt your style to the style of the group of people who spent countless hours creating the linter in the first place.