DC/OS - The Datacenter Operating System
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github CODEOWNERS for dcos/dcos repo. Nov 28, 2017
cloud_images SRE-222: Builds AMIs with support for fifth gen EC2 instances Apr 23, 2018
config Adding required backslash to download_url Nov 7, 2018
dcos_installer platform: bump version to 1.13-dev Oct 8, 2018
docs Merge branch 'master' into telegraf_config_statsd_soak Nov 6, 2018
flake8_dcos_lint Pin flake8 version to 3.3.0. Jul 28, 2017
gen Merge #3887 Dec 12, 2018
packages Merge branch 'master' into mergebot-integration-branch/201814121030/3… Dec 14, 2018
pkgpanda Merge #2877 May 17, 2018
release Handle Azure Exception During Upload. Dec 12, 2018
ssh Remove unused class SyncCmdDelegate May 18, 2018
test_util Update dcos-vagrant pinned version. Feb 13, 2018
.DS_Store Adding required backslash to download_url Nov 7, 2018
.editorconfig Add editorconfig Jun 1, 2016
.gitignore .gitignore: ignore common build artifacts and dot-files Nov 22, 2018
CHANGES.md chore(dcos-ui): bump DC/OS UI v2.40.6 Dec 13, 2018
Jenkinsfile Bump used sec-ci-libs to v2, adjust Jenkinsfiles Sep 11, 2017
Jenkinsfile-insecure.groovy Admin Router: rename `make flake8` target to `make lint` as per new g… Aug 4, 2017
LICENSE Include LICENSE, NOTICE file in dcos-image. Dec 4, 2017
NEWS Bump master to 1.10-dev. May 11, 2017
NOTICE Collect Admin Router Nginx metrics from agents Dec 6, 2018
PULL_REQUEST_TEMPLATE.md PR template: make changelog entry obligatory Jan 26, 2018
README.md Update README example for running command to new DC/OS E2E CLI Nov 9, 2018
build_dcos_launch.sh Modify mkpanda CLI to specify variants Mar 23, 2017
build_local.sh Fix the release create stage. Aug 30, 2018
build_local_windows.ps1 Update build_local_windows.ps1 Mar 26, 2018
build_teamcity Parameterize tree variants in build scripts Mar 26, 2018
build_teamcity_windows.ps1 Add missing error check Nov 2, 2018
conftest.py Cleanup Jan 23, 2018
contributing.md Update contributing.md to be a little nicer to newcomers Jul 20, 2016
fetch_cluster_logs.bash Update fetch_cluster_logs.bash Nov 2, 2018
mergebot-config.json Update jira fix version for 1.11 to 1.11.9 Nov 30, 2018
owners.json Add mergebot to owners. Dec 11, 2018
prep_local Squash ext/dcos-installer into the base dcos python package Sep 22, 2016
prep_local_windows.ps1 windows-based pkgpanda builds Mar 26, 2018
prep_teamcity Use `pip download` to prepare TeamCity rather than the removed `pip i… Apr 16, 2018
prep_teamcity_windows.ps1 Add prep_teamcity_windows.ps1 script Oct 8, 2018
setup.py setup.py: use pathlib for windows / linux compat Aug 8, 2018
symlink_check DC/OS 💥 Apr 19, 2016
tox.ini tox.ini: ignore pedantic bracket indenetation rule Nov 22, 2018

README.md

DC/OS - The Datacenter Operating System

The easiest way to run microservices, big data, and containers in production.

What is DC/OS?

Like traditional operating systems, DC/OS is system software that manages computer hardware and software resources and provides common services for computer programs.

Unlike traditional operating systems, DC/OS spans multiple machines within a network, aggregating their resources to maximize utilization by distributed applications.

To learn more, see the DC/OS Overview.

How Do I...?

Releases

DC/OS releases are publicly available on http://dcos.io/releases/

Release artifacts are managed by Mesosphere on Amazon S3, using a CloudFront cache.

To find the git SHA of any given release, check the latest commit in the versioned branches on GitHub: https://github.com/dcos/dcos/branches/

Release Type URL Pattern
Latest Stable https://downloads.dcos.io/dcos/stable/dcos_generate_config.sh
Latest Master https://downloads.dcos.io/dcos/testing/master/dcos_generate_config.sh
Specific PR, Latest Build https://downloads.dcos.io/dcos/testing/pull/<github-pr-number>/dcos_generate_config.sh

Development Environment

Linux is required for building and testing DC/OS.

  1. Linux distribution:
    • Docker doesn't have all the features needed on OS X or Windows
    • tar needs to be GNU tar for the set of flags used
    • unzip needs to be installed
  2. tox
  3. git 1.8.5+
  4. Docker 1.11+
    • Install Instructions for various distributions. Docker needs to be configured so your user can run docker containers. The command docker run alpine /bin/echo 'Hello, World!' when run at a new terminal as your user should just print "Hello, World!". If it says something like "Unable to find image 'alpine:latest' locally" then re-run and the message should go away.
  5. Python 3.6
    • Arch Linux: sudo pacman -S python
    • Fedora 23 Workstation: Already installed by default / no steps
    • Ubuntu 16.04 LTS:
      • pyenv-installer
      • Python dependencies: sudo apt-get install make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev xz-utils liblzma-dev python3-venv
      • Install Python 3.6.3: pyenv install 3.6.3
      • Create DC/OS virtualenv: pyenv virtualenv 3.6.3 dcos
      • Activate environment: pyenv activate dcos
  6. Over 10GB of free disk space and 8GB of RAM
    • The build makes use of hard links, so if you're using VirtualBox the disk space cannot be a synced folder.
  7. Optional pxz (speeds up package and bootstrap compression)
    • ArchLinux: pxz-git in the AUR. The pxz package corrupts tarballs fairly frequently.
    • Fedora 23: sudo dnf install pxz

Unit Tests

Unit tests can be run locally but require the development environment specified above.

tox

Tox is used to run the codebase unit tests, as well as coding standard checks. The config is in tox.ini.

Integration Tests

Integration tests can be run on any deployed DC/OS cluster. For installation instructions, see https://dcos.io/install/.

Integration tests are installed via the dcos-integration-test Pkgpanda package.

Integration test files are stored on the DC/OS master node at /opt/mesosphere/active/dcos-integration-test. Therefore, in order to test changes to test files, move files from packages/dcos-integration-test/extra/ in your checkout to /opt/mesosphere/active/dcos-integration-test on the master node.

The canonical source of the test suite's results is the continuous integration system. There may be differences between the results of running the integration tests as described in this document and the results given by the continuous integration system. In particular, some tests may pass on the continuous integration system and fail locally or vice versa.

Minimum Requirements

  • 1 master node
  • 2 private agent nodes
  • 1 public agent node
  • Task resource allocation is currently insignificantly small
  • DC/OS itself requires at least 2 (virtual) cpu cores on each node

Instructions

  1. SSH into a master node The tests can be run via Pytest while SSH'd as root into a master node of the cluster to be tested.

  2. Switch to root

    sudo su -
    
  3. Add the test user

    dcos-shell python /opt/mesosphere/active/dcos-oauth/bin/dcos_add_user.py albert@bekstil.net
    

    Running the above mentioned command will result in an output

    User albert@bekstil.net successfully added
    

    This test user has a known login token with far future expiration. DO NOT USE IN PRODUCTION. After the test, remember to delete the test user.

    For more information, see User Management.

  4. Run the tests using pytest in the cluster.

    cd /opt/mesosphere/active/dcos-integration-test
    dcos-shell pytest
    

Using a Docker Cluster with DC/OS E2E

One way to run the integration tests is to use the DC/OS E2E CLI.

This lets you create, run and manage clusters in test environments. Each DC/OS node is represented by a Docker container.

  1. Setup DC/OS in containers using the DC/OS E2E CLI.

For example, after installing the DC/OS E2E CLI, create a cluster:

dcos-docker create /tmp/dcos_generate_config.sh \
    --masters 1 \
    --agents 2 \
    --public-agents 1 \
    --cluster-id default
  1. Run dcos-docker wait

Wait for DC/OS to start. Running wait command allows to make sure that the cluster is set up properly before any other actions that could otherwise cause errors in pytest command in the next step.

  1. Run pytest on a master node.

For example:

dcos-docker run --test-env pytest
  1. Destroy the cluster.
dcos-docker destroy

Build

DC/OS can be built locally but requires the development environment specified above.

DC/OS builds are packaged as a self-extracting Docker image wrapped in a bash script called dcos_generate_config.sh.

WARNING: Building a release from scratch the first time on a modern dev machine (4 cores / 8 hyper threads, SSD, reasonable internet bandwidth) takes about 1 hour.

Instructions

./build_local.sh

That will run a simple local build, and output the resulting DC/OS installers to $HOME/dcos-artifacts. You can run the created `dcos_generate_config.sh like so:

$ $HOME/dcos-artifacts/testing/`whoami`/dcos_generate_config.sh

Build Details

If you look inside of the bash script build_local.sh there are the commands with descriptions of each.

The general flow is to:

  1. Check the environment is reasonable
  2. Write a release tool configuration if one doesn't exist
  3. Setup a python virtualenv where we can install the DC/OS python tools to in order to run them
  4. Install the DC/OS python tools to the virtualenv
  5. Build the release using the release tool

These steps can all be done by hand and customized / tweaked like standard python projects. You can hand create a virtualenvironment, and then do an editable pip install (pip install -e) to have a "live" working environment (as you change code you can run the tool and see the results).

Release Tool Configuration

This release tool always loads the config in dcos-release.config.yaml in the current directory.

The config is YAML. Inside it has two main sections. storage which contains a dictionary of different storage providers which the built artifacts should be sent to, and options which sets general DC/OS build configuration options.

Config values can either be specified directly, or you may use $ prefixed environment variables (the env variable must set the whole value).

Storage Providers

All the available storage providers are in release/storage. The configuration is a dictionary of a reference name for the storage provider (local, aws, my_azure), to the configuration.

Each storage provider (ex: aws.py) is an available kind prefix. The dictionary factories defines the suffix for a particular kind. For instance kind: aws_s3 would map to the S3StorageProvider.

The configuration options for a storage provider are the storage provider's constructor parameters.

Sample config storage that will save to my home directory (/home/cmaloney):

storage:
  local:
    kind: local_path
    path: /home/cmaloney/dcos-artifacts

Sample config that will store to a local archive path as well as AWS S3. To authenticate with AWS S3, reference the boto3 docs to learn how to configure access.

storage:
  aws:
    kind: aws_s3
    bucket: downloads.dcos.io
    object_prefix: dcos
    download_url: https://downloads.dcos.io/dcos/
  local:
    kind: local_path
    path: /mnt/big_artifact_store/dcos/

Repo Structure

DC/OS itself is composed of many individual components precisely configured to work together in concert.

This repo contains the release and package building tools necessary to produce installers for various on-premises and cloud platforms.

Directory Contents
cloud_images Base OS image building tools
config Release configuration
docs Documentation
flake8_dcos_lint Flake8 plugin for testing code quality
dcos_installer Backend for Web, SSH, and some bits of the Advanced installer. Code is being cleaned up
gen Python library for rendering yaml config files for various platforms into packages, with utilities to do things like make "late binding" config set by CloudFormation
packages Packages which make up DC/OS (Mesos, Marathon, AdminRouter, etc). These packages are built by pkgpanda, and combined into a "bootstrap" tarball for deployment.
pkgpanda DC/OS baseline/host package management system. Tools for building, deploying, upgrading, and bundling packages together which live on the root filesystem of a machine / underneath Mesos.
release Release tools for DC/OS. (Building releases, building installers for releases, promoting between channels)
ssh AsyncIO based parallel ssh library used by the installer
test_util various scripts, utilities to help with integration testing

Pull Requests Statuses

Pull requests automatically trigger a new DC/OS build and run several tests. These are the details on the various status checks against a DC/OS Pull Request.

Status Check Purpose Source and Dependencies
continuous-integration/jenkins/pr-head Admin Router Endpoint tests dcos/dcos/packages/adminrouter/extra/src/test-harness Docker Dependency: dcos/dcos/packages/adminrouter
mergebot/enterprise/build-status/aggregate EE Test Enforcement Private mesosphere/dcos-enterprise repo is tested against the SHA.
mergebot/enterprise/has_ship-it Code Review Enforcement Private Mergebot service in prod cluster
mergebot/enterprise/review/approved/min_2 Code Review Enforcement Mergebot service in prod cluster
mergebot/has_ship-it Code Review Enforcement Mergebot service in prod cluster
mergebot/review/approved/min_2 Code Review Enforcement Mergebot service in prod cluster
teamcity/dcos/build/dcos Builds DCOS Image (dcos_generate_config.sh) gen/build_deploy/bash.py
teamcity/dcos/build/tox Runs check-style, unit-tests tox.ini
teamcity/dcos/test/aws/cloudformation/simple Deployment using single-master-cloudformation.json and runs integration tests gen/build_deploy/aws.py, Uses dcos-launch binary in CI
teamcity/dcos/test/aws/onprem/static Installation via dcos_generation_config.sh and runs Integration Tests gen/build_deploy/bash.py, Uses dcos-launch binary in CI
teamcity/dcos/test/azure/arm Deployment using acs-1master.azuredeploy.json and runs integration tests. gen/build_deploy/azure.py, Uses dcos-launch binary in CI
teamcity/dcos/test/docker Exercises dcos-docker by launching dcos-docker against this PR and running integration tests within the docker cluster. dcos-docker repo
teamcity/dcos/test/docker/smoke Exercises dcos-docker by launching dcos-docker against this PR and running smoke tests within the docker cluster. dcos-docker repo
teamcity/dcos/test/upgrade Upgrade from stable minor version mesosphere/advanced-tests repo (transitively, dcos/dcos-test-utils , dcos/dcos-launch)
teamcity/dcos/test/upgrade-from-previous-major Upgrade from previous major version mesosphere/advanced-tests repo (transitively, dcos/dcos-test-utils, dcos/dcos-launch)
teamcity/dcos/test/upgrade-to-next-major Upgrade to Next Major version mesosphere/advanced-tests repo (transitively, dcos/dcos-test-utils, dcos/dcos-launch)