An ssh cluster shell for EC2
Python Shell
Switch branches/tags
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
bin
docs bugfixes Oct 13, 2017
dustcluster bump version Mar 8, 2018
tests
.travis.yml fix for travis CI Sep 4, 2017
INSTALL.md
README.md
dust
pylint.rc
requirements.txt
setup.cfg
setup.py fix setup Oct 31, 2017

README.md

dustcluster

DustCluster v0.2.0 is an AWS command line interface in an REPL shell with cluster-aware ssh and EC2 node operations, with plugin commands to bring up some pre-configured clusters on EC2.

Status:

  • Linux and OSX only
  • Alpha/work in progress
  • Python 2.7

Build Status PyPI version

Quickstart:

bash$ dust

[eu-west-1] show

dustcluster

Rationale

This can be useful for developing, prototyping, and one-off configurations of (usually ephemeral) EC2 clusters. Such as when developing/testing custom data engineering stacks and distributed systems.

Summary of Features

As an EC2 web console replacement

dustcluster ec2 operations

  • start/stop/terminate

    stop [filter-expression]  
    stop *           		# stop all nodes
    stop 1,3,4       		# by index
    stop worker*     		# by name
    stop state=running 	# by EC2 attribute
    
  • list nodes

    show [-v] [-vv] [filter-expression | search term]
    show      # list all nodes
    show -v   # list with extended attributes
    show -vv  # list with all attributes
    
    show -v  1,3,4  # using filter expresison
    show -v  worker*
    
  • search

    $show searches all ec2 attributes and tags
    
    show mongo          # matches tag app=mongodb
    show running        # matches state=running attribute
    show 192.168.0.15   # matches public_ip_address, private_ip_address etc
    show 0.15           # same as above, partial matches work
    show run
    
  • tag/untag nodes

    tag [filter-expression] tags
    tag worker* env=dev
    tag 3,4 stack=spark1
    
  • refresh cache

    refresh
    
  • view cloudwatch logs

    logs                            # shows log groups
    logs /aws/lambda/hello_world    # show log streams in this group
    logs -t /aws/lambda/hello_world # show the most recent log events in this group
    logs -N 50 -t  /aws/lambda/xyz  # increase output limit to 50
    

Note: Filter expressions can contain index numbers, node names, or all EC2 instance attributes from show -vv .e.g. image=ami-123145. More on filter expressions below.

Parallel ssh operations

You need to configure logins first.

  • run non-interactive commands

    @[filter-expression] command
    
    e.g. Run  "uptime" on ..
    
    @ uptime                    # all running nodes
    @1,3,4 uptime               # nodes with indexes 1,3 and 4 
    @worker* uptime             # node names matching worker*
    @image=ami-1234 uptime      # EC2 attribute
    @tags=key:value uptime      # by tags
    
  • run interactive commands

    Same as above. This is **stateful** ssh - the connection is kept open.
    
    @1,2,3 sudo apt-get install nginx
    
      .
      .
      [worker3] After this operation, 5,342 kB of additional disk space will be used.
      [worker3] Do you want to continue? [Y/n]
    
          
    @1,2,3 y
    
    // sends a Y to all the nodes named work\* and the apt-get script continues.
    
  • run ncurses/full screen commands

    @[filter_exp]  -  drops to a shell
    
    [eu-west-1]$ @9
    dust:11:48:43 | *** Entering raw shell, press ctrl-c thrice to return to cluster shell. Press Enter to continue.***
    
    [ec2-user@ip-172-31-44-125 ~]$ 
    [ec2-user@ip-172-31-44-125 ~]$ vim /etc/resolv.conf
    
  • secure copy files

    put [filter_exp] localfile* remote_dir
    get [filter_exp] remotefile local_dir
    
    put 1,3 /home/alice/data*.csv /home/ec2-user
    get 1,3 /home/ec2-user/data3.csv .
    
Cluster-aware ssh and node operations

When you are working with a 3 node cluster but you have 25 ec2 nodes...

  • use a working set
    use [clustername | *]
    
    use spark1
    show              # shows cluster nodes only
    @ uptime          # incvoke command over ssh to cluster nodes only
    tag * cool=yes    # tags cluster nodes only
    

You specify cluster membership in the login configuration.

  • remove working set restriction
    use *
    
Spin up a new compute cluster
  • Create a N node compute cluster with 10 GBps interconnect with in a placement group in the default VPC, with keys downloaded and security groups configured. More details here.
  cluster new           - asks for node count and creates a cluster spec   

  cluster create base1  - converts the spec to a CloudFormation template and instantiates it

  cluster status base1  - shows the Cloudformation status

  cluster delete base1  - terminates the cluster nodes  
Regions
use us-east-1     # switch to region
Run commands on localhost

If dustcluster doesn't recognize a command, it drops it to the calling shell.

[eu-west-1]$ ls -plart /var/log
dust:12:06:08 | dustcluster: [ls -plart /var/log] unrecognized, trying system shell...

total 25828
drwx------.  2 root    root               4096 Feb  5  2016 ppp/
drwx------.  3 root    root               4096 Nov 14  2016 libvirt/
drwxr-xr-x.  2 chrony  chrony             4096 Nov 21  2016 chrony/
.
.

[eu-west-1]$ vim /home/alice/topsecret.txt    # runs vim locally

Command History
  • Reverse search (ctrl-R) to retrieve old commands
    (reverse-i-search)`':@1,3,5,7 grep -c ERROR /var/logs/applogs/app.log
    
    up/down arrow to navigate 
    

More on Filter expressions

Most commands take a target. e.g. show -v [target]

Here, [target] is a comma separated list of filter expressions.

Filter expression can be:

A node index

dust$ stop 1,2,3

dust$ @1,2,3 service restart nginx

By node name

Name comes from Instance tags with Key=Name

dust$ stop worker*

dust$ start wo*

dust$ terminate worker[0-2]

dust$ @worker* vmstat

By EC2 attribute or tags

dust$ show state=stopped

dust$ start state=stop* # filters can have wildcards

dust$ @ip=54.12.* uptime

dust$ @tags=owner:devops uptime

dust$ stop tags=env:* # tags can have wildcards too

dust$ show -v launch_time=2016-04-03*

"show" takes a search term as well so this is equivalent:

dust$ show -v 2016-04-03

By cluster name

dust$ show -v cluster=mystack1

dust$ stop cluster=slurm1

Configure ssh Logins

Before using ssh you need to configure logins. Use filter expressions to select nodes with the same login user and key:

i) by using the assign command

This command writes login rules to ~/.dustcluster/login_rules.yaml.

Example:

$assign
selector: tags=key:value 
login-user: ec2-user
keyfile : /path/to/kyfile
member-of: webapp

1] selector help:
selector is a filter expression.`

selector: *                    # selects all nodes
selector: id=0-asd1212         # selects a single node
selector: subnet=s-012sccas    # selects all nodes in a subnet

2] member-of help:
member-of adds nodes to a cluster, these nodes are grouped together in "show", 
and can be made into a working set with the "use" command. cluster names can be
used in filter expressions.

3] login rules precedence:

login rules are applied in order that they appear in the file. 
So given a login_rules.yaml containing:

- selector: tags=env:prod   
    ...
- selector: tags=env:dev
    ...
- selector: *
    ...

a command like "@node5 service restart xyz" will search for a login rule by matching prod, dev, 
and then default (*) in that order'''

ii) by manually editing the login rules file. See a sample login rules file here.

Writing plugin commands

Write commands to an interface and drop them into dustcluster...

i) Write a plugin command that list all your instances in all regions.

TBD

ii) Write a plugin command that runs a set of ssh commands on a cluster

TBD

Install

Install:

The python cryptography package now pulls in a boatload of deps, so the install is bit cumbersome:

Ubuntu:

# pip
sudo apt-get update
sudo apt-get install python-pip
sudo pip install pip --upgrade

# deps
sudo apt-get install build-essential python-dev libffi-dev libssl-dev
sudo pip install enum34 cffi bcrypt cryptography

# dust
sudo pip install dustcluster

AmazonLinux/CentOS:

#deps
yum install gcc gcc-c++ make openssl-devel python-devel libffi-devel libssl-devel        
sudo  pip install enum34 cffi bcrypt cryptography

# dust
sudo pip install dustcluster

On a python 3 default system, use sudo pip2.7 instead of pip above.