Python Other
Switch branches/tags
Nothing to show
Latest commit 4706bde Sep 18, 2017 @ianmiell release: 1.0.23
Permalink
Failed to load latest commit information.
fixterm_src assets Apr 12, 2016
shutit-docs @ c4efb0b submodules Jun 1, 2016
shutit-test @ 845b94d release: 1.0.22 Sep 12, 2017
shutit_patterns no checking sudo Jul 25, 2017
.gitignore #292: reinstate previous check that was failing before Feb 17, 2017
.gitmodules stray colon Nov 23, 2016
.pylintrc Update pylintrc May 8, 2014
CONTRIBUTING.md Create CONTRIBUTING.md Jun 20, 2017
Jenkinsfile lock at a higher level May 23, 2017
LICENSE license Mar 26, 2014
README latest Apr 1, 2016
README.md fix README.md syntax in docker build section Jun 11, 2017
banner.sh merged Apr 11, 2016
emailer.py logging explicit Sep 14, 2017
package_map.py package improvements and og->log Jun 23, 2017
release.sh test Jul 26, 2017
requirements.txt six Mar 4, 2017
setup.py release: 1.0.23 Sep 18, 2017
shutit shutit Apr 23, 2017
shutit.py release: 1.0.23 Sep 18, 2017
shutit_assets.py more pylinting May 21, 2017
shutit_background.py logging Sep 14, 2017
shutit_class.py logging explicit Sep 14, 2017
shutit_exam.py remove todo Jun 16, 2017
shutit_global.py rm rm Aug 26, 2017
shutit_login_stack.py logging explicit Sep 14, 2017
shutit_module.py more pylinting May 21, 2017
shutit_pexpect.py background the history command in case it fails Sep 15, 2017
shutit_pexpect_session_environment.py finalize May 1, 2017
shutit_sendspec.py follow_on_commands documented Jun 16, 2017
shutit_setup.py cleanups May 25, 2017
shutit_skeleton.py Exception made broad again May 26, 2017
shutit_util.py logging explicit Sep 14, 2017

README.md

ShutIt

Join the chat at https://gitter.im/ianmiell/shutit

A versatile automation framework.

ShutIt is an automation tool that models a user's actions on a terminal.

It can automate any process that can be run by a human on the command line with little effort.

It was originally written to manage complex Docker builds, but is a now general-purpose automation tool that supports bash, Docker, Vagrant, ssh and arbitrary build contexts.

ShutIt can also be used as an educational tool, as it can produce videos of demos, capture reproducible steps required to set environments up, and even challenge you to get the right output (see grep-scales).

If you want to know more about Docker, see the official site or take a look at the book by the creators of ShutIt - Docker in Practice.

Really Quick Overview

Some use cases:

  • You like bash, want to automate tasks, have structure and support, but don't want to learn a configuration management framework that takes you away from the command line you know and love.

  • Want to create complex Vagrant environments to model clusters of machines.

  • Want to create instructive walkthroughs:

  • Are interested in "phoenix deployment".

  • Want to take your scripts and turn them into stateless containers quickly, without needing to maintain (or learn) a configuration management solution designed for moving-target systems.

  • You're programmer who wants highly configurable stateless containers development, testing, and production.

  • Want to build everything from source in a way that's comprehensible and auditable.

What Does it Do (bash Builds)?

ShutIt acts as a modular and easy to use wrapper around pexpect.

Here is a simple example of a script that creates a file and a directory if they are not there already:

Simple Example

What Does it Do (Tutorials)?

This builds on the docker features (see below), but allows you to interrupt the run at points of your choosing with 'challenges' for the user to overcome.

Two types of 'challenge' exist in ShutIt:

  • scales
  • free form

Scales tell you to run a specific command before continuing. This is useful when you want to get certain commands or flags 'under your fingers', which does not happen without dedicated and direct practice.

grep Scales

Free form exercises give you a task to perform, and free access to the shell. This is to give the user a realistic environment in which to hone their skills. You can check man pages, look around the directories, search for useful utils (even install new ones!). When you are finished, a pre-specified command is run to check the system is in an appropriate state. Here's an example for the basics of git:

git 101 Tutorial

If you use a Docker-based tutorial and you mess the environment up, the state can be restored to a known one by hitting CTRL-G.

What Does it Do (Vagrant)?

Uses a bash build to set up n vagrant machines, and uses Landrush to give them useful hostnames accessible from the hosts and in the guest VMs.

It supports both Virtualbox and Libvirt providers.

This allows another kind of contained environment for more infrastructural projects than Docker allows for.

This example demonstrates a reproducible build that sets up Docker on an Ubuntu VM (on a Linux host), then runs a CentOS image within Docker wihing the Ubuntu VM.

It deposits the user into a shell mid-build to interrogate the environment, after which the user re-runs the build to add a directive to ensure ps is installed in the image.

Docker on Ubuntu VM running a CentOS image

What Does it Do (Docker Builds)?

Example Setup

We start with a "ShutIt Module", similar to a shell script, or a Dockerfile (see bash builds above).

In the image above there are five of these. At a high level they each have the following attributes:

  • a list of zero or more dependencies on other modules
  • a unique number that represents its ordering within the available modules
  • a set of steps (bash commands) for building the module

In the image we imagine a scenario where we want to build our blog into a docker image, with all its attendant content and config.

We instruct ShutIt to build the MyBlog module, and it runs the build as per the image on the right.

The container environment is set up, the modules are ordered, and the build steps are run. Finally, the image is committed, tagged and pushed as configured.

This is a core function of ShutIt - to manage dependencies and image building for complex image setups.

But it doesn't just run build steps, it also manages The ShutIt Lifecycle to make the build more robust and flexible.

The ShutIt Lifecycle

  • gathers all the modules it can find in its path and determines their ordering
  • for all modules, it gathers any build-specific config (e.g. passwords etc.)
  • it checks dependencies and conflicts across all modules and figures out which modules need to be built
  • for all modules, it checks whether the module is already installed
  • for all modules, if it needs building, it runs the build
  • for all modules, run a test cycle to ensure everything is as we expect
  • for all modules, run a finalize function to clean up the container
  • do any configured committing, tagging and pushing of the image

These correspond to the various functions that can be implemented in the ShutIt module file.

Auto-Generate Modules

ShutIt provides a means for auto-generation of modules (either bare ones, or from existing Dockerfiles) with its skeleton command. See here for an example.

Really Quick Start

Full User Guide

API

Installation

Contributing

We always need help, and with a potentially infinite number of libraries required, it's likely you will be able to contribute. Just mail ian.miell@gmail.com if you want to be assigned a mentor. He won't bite

Tests

Mailing List

https://groups.google.com/forum/#!forum/shutit-users shutit-users@groups.google.com

Known Issues

Since a core technology used in this application is pexpect - and a typical usage pattern is to expect the prompt to return. Unusual shell prompts and escape sequences have been known to cause problems. Use the shutit.setup_prompt() function to help manage this by setting up a more sane prompt. Use of COMMAND_PROMPT with echo -ne has been seen to cause problems with overwriting of shells and pexpect patterns.

ScreenShot

Licence

The MIT License (MIT)

Copyright (C) 2014 OpenBet Limited

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.