Manage multiple servers using SSH and tmux.
For the demo and short intro, read this blog post.
- Quick tease
- CLI description
- Sessions without configuration files
- Comments, ideas or if you feel like chatting
- connect to multiple servers at once
- expressive layouts customizable for different server groups available
- issue commands to all servers or just a selected (custom) subgroup
- work on multiple sessions in parallel
- multiple connections to individual servers possible
- sessions can be persisted (actually recreated) after computer restarts
- they are lost only if you delete them explicitly
- panes not associated with hosts available
tcon start staging_config.yml -n staging -p 'all staging servers'
- crate a session (name: 'staging', description: 'all staging servers') with complex layout structure (multiple windows with (potentially) different pane layouts)
- connect to all servers
- attach to tmux session
tcon send staging 'sudo su'
- send to all servers (in 'staging' session)
tcon send staging -v -g 'lbs' 'top'
topcommand to all loadbalancing nodes in 'staging' session and report the number of affected nodes
tcon send production -f 'worker' 'tail -f /var/log/syslog'
tail -f /var/log/syslogcommand to all worker nodes in 'production' session
tcon send production -f 'worker :: <,3]; 7; <9,13>' 'C-c'
Ctrl-Cto some worker nodes in 'production' session, if executed after the
tail -f /var/log/syslogcommand, it will exit the
- if there are worker nodes numbered from 1 to 20 (e.g. staging-worker-1, staging-worker-2, etc.) this command affects worker nodes: 1, 2, 3, 7, 10, 11 and 12
tcon resume s#3
- resume (recreate) 's#3' session, even after computer restart
tcon start -q "dev.node-staging-[[42,]]"
- starts quick session (no config file); selects all 'node' servers that have index of at least 42
tcon enables establishing connections (ssh) to multiple servers and executing commands on those servers. The sessions can be persisted (actually recreated) even after computer restarts. Complex sessions with different layouts for different kinds of servers can be easily created. Usage: tcon start ( <config-file> | --quick-session=<qs-args> ) [--ssh-config=<file>] [--session-name=<name>] [--purpose=<description>] tcon resume <session-name> tcon delete (<session-name> | --all) tcon list tcon send <session-name> ( <command> | --command-file=<file> ) [ --server-filter=<filter> | --group-filter=<regex> | --filter=<regex> | --window=<index> ] [--verbose] tcon --help tcon --version Options: <config-file> Path to configuration file. Configuration file describes how new session is started. YAML. <qs-args> Arguments needed to start a quick session. <session-name> Name to identify the session. Must be unique. <command> Command to be executed on remote server[s]. <regex> String that represents valid Ruby regex. <index> 0-based index. <filter> Filter consisting of a valid ruby regex and optionally of a special predicate. For more information see README file. -q --quick-session=qs-args Start the seesion without a configuration file. Specify necessary argumenst instead. -s --ssh-config=file Path to ssh config file. [default: ~/.ssh/config] -n --session-name=name Name of the session. -p --purpose=description Description of session's purpose. --all Delete all existing sessions. -f --server-filter=filter Filter to select a subset of the servers via host names. -g --group-filter=regex Filter to select a subset of the servers via group membership. -r --filter=regex Filter to select a subset of the servers via host names or group membership. Combines --server-filter and --group-filter. -w --window=index Select a window via (0-based) index. -c --command-file=file File containing the list of commands to be executed on remote server[s]. -v --verbose Report how many servers were affected by the send command. -h --help Show this screen. --version Show version.
Send command sends user specified command(s) to chosen panes.
Once more, here's the command syntax:
tcon send <session-name> (<command> | --command-file=<file>) [ --server-filter=<filter> | --group-filter=<regex> | --filter=<regex> | --window=<index> ] [--verbose]
where long flags can be shortened to:
There are 4 flags used to filter the servers (panes) that are going to receive the command(s):
-ffilters the servers via host names
- accept valid ruby regex and optionally intervals specification
<ruby-regex> :: <intervals-specification>
- accept valid ruby regex and optionally intervals specification
-gfilters the servers via group names
- accepts valid ruby regex
-rfilters the servers via host names or group names
- accepts valid ruby regex
-wfilters the servers that belong to particular layout window
- accepts 0-based index
-f filter can accept optional interval specification:
- interval specification operates on 'sort-by' part of host names (defined in
- intervals specification consists of one or more elements separated
by semicollon (the split regex is:
- individual element is either an interval description or white-listed element
- interval description consists of 4 parts:
- '[' or '<' - include or exclude the first element
- first element (can be empty)
- ',' (can have trailing whitespace)
- second element (can be empty)
- ']' or '>' - include or exclude the last element
- regex used:
2; 4- 2, 4
[1,3]- 1, 2, 3
[1, 3>- 1, 2
<1, 3>- 2
<,5>- ... 3, 4
[5, >- 5, 6, ...
- intervals specification consists of one or more elements separated by semicollon (the split regex is:
Hopefully now the following examples make more sense:
tcon send staging 'sudo su' tcon send staging -v -g 'lbs' 'top' tcon send production -f 'worker' 'tail -f /var/log/syslog' tcon send production -f 'worker :: <,3]; 7; <9,13>' 'C-c'
To use this gem, you usually need to create a configuration file.
Let's get to it.
The configuration file is in YAML format.
Let's say the following ssh config file that will be used:
KeepAlive yes ServerAliveInterval 2 StrictHostKeyChecking no UserKnownHostsFile=/dev/null Host staging.cache-staging-1 Hostname ec2-111-42-111-42.eu-west-1.compute.amazonaws.com Port 4242 IdentityFile /Users/some-user/.ssh/some-pem-file.pem User ubuntu Host dev.database-staging-1 << omitted >> Host dev.database-staging-3 << omitted >> Host dev.mongodb-single-replica-1 << omitted >> Host dev.haproxy-staging-72 << omitted >> Host dev.haproxy-staging-73 << omitted >> Host dev.nginx-staging-11 << omitted >> Host dev.nginx-staging-15 << omitted >> Host dev.node-staging-127 << omitted >> Host dev.node-staging-129 << omitted >> << ... >>
Every configuration file needs to define at least the folowing fields: 'regex', 'regex-parts-to', 'group-by' and 'sort-by'
Here's a minimal configuration file that could be used:
regex: !ruby-regexp '^(\w+)\.(\w+)-([\w-]+)-(\d+)$' regex-parts-to: group-by:  sort-by: 
And here's a 'real world' configuration file that shows off all the available options and could be use with previous ssh config file:
regex: !ruby-regexp '^(\w+)\.(\w+)-([\w-]+)-(\d+)$' reject-regex: !ruby-regexp '-(nodes|to_ignore)-' regex-parts-to: group-by:  sort-by:  name: regex-ignore-parts: [0, 2] separator: '-' prefix: 'dev--' hostless: ctrl: 1 additional: 2 merge-groups: misc: ['cache', 'db', 'mongodb'] lbs: ['haproxy', 'nginx'] group-ranges: cache: [1, 4] node: [125, 131] multiple-hosts: regexes: - !ruby-regexp '(nginx|haproxy)-' - !ruby-regexp '(db)-' counts: [2, 3] layout: default: custom: max-horizontal: 3 max-vertical: 3 panes-flow: vertical group-layouts: misc: tmux: layout: tiled max-panes: 6 node: tmux: layout: tiled
(required) 'regex' field is the most important field. Some other field reference this one. It provides a rule on how to parse host names from ssh config file. The regex should be a valid ruby regex. (If you're not familiar with ruby regexes, consider visiting rubulator and playing around.)
All host whose host names fail the regex will be ignored.
For example if ssh config file include the following host definition:
and the following regex is used:
regex: !ruby-regexp '^(\w+)\.(\w+)-([\w-]+)-(\d+)$'
the name 'dev.database-staging-1' will be broken to 4 groups:
'dev', 'database', 'staging', 1
This regex is used for all the host names and should be designed accordingly.
The idea behind the regex is to enable sorting and grouping of hosts from regex groups extracted from host names. Those groups are used to crate meaningful layouts. I know, sounds more complex than it really is...
In (required) 'regex-parts-to' section, fields 'group-by' and 'sort-by' are referencing before mentioned 'regex' field. As their names suggest, they decide which servers constitute a group (and share layout and potentially commands) and how to sort serves in a group. Both fields can reference more than one regex group.
In the above example, for 'dev.database-staging-1' host name, a group to which the host belongs would be 2nd group, which is: 'database'.
(optional) 'reject-regex' field is used to ignore some hosts while starting a session.
(optional) field 'name' and it's (optional) subfields 'regex-ignore-parts', 'separator' and 'prefix' decide how to name the servers. If those fields are omitted, ssh host name is used instead.
Filed 'regex-ignore-parts' potentially removes some regex groups from name, 'separator' is used to separate left-over groups and it's possible to specify 'prefix' for the name.
(optional) field 'hostless' defines groups of panes that are not associated with hosts. Each group is defined by the name and the number of panes.
hostless: ctrl: 1 additional: 2
defines 2 groups of panes without hosts. First group, 'ctrl', has only one pane, while the second group, 'additional', has 2 panes.
Hostless panes can be used to:
- issue the tcon commands to other panes
- e.g. one control pane, in a separate window
- manage local work
- connect to a host at some point in the future
(optional) field 'merge-groups' contains groups that should be merged (for layout purposes) together. This can be used to group a few servers that are unique in type or small in numbers. E.g. grouping different DB servers.
lbs: ['haproxy', 'nginx']
In this example two different kinds of loadbalancers are grouped together.
Note that the servers/panes from merge groups can later be referenced with both original and merge-group name.
Hostless groups can also be merged.
(optional) field 'group-ranges' contains groups that should be limited to specific ranges of servers. Ranges are defined by lower and upper limits, both inclusive. They are enforced on sort values (defined by aforementioned 'sort-by' field).
If the following group ranges were defined:
group-ranges: cache: [1, 4] node: [125, 131]
Only cache servers with sort ids 1, 2, 3 or 4 would be included in the session. Similarly for node servers, only servers with ids between 125 and 131 would be included.
(optional) field 'multiple-hosts' contains 'regexes' and 'counts' fields. With those, some hosts can have multiple connections established, not just the default one connection per host.
multiple-hosts: regexes: - !ruby-regexp '(nginx|haproxy)-' - !ruby-regexp '(db)-' counts: [2, 3]
creates 2 connections for each of nginx or haproxy nodes, as well as 3 connection for db nodes.
Fields 'regexes' and 'counts' must have the same number of elements. Each element in 'regexes' must contain valid ruby regex.
Finally, what's left is the (optional) layout definition:
There are 2 main ways to specify a layout for a (merge-)group:
- with option tmux, built-in tmux layouts: even-horizontal, even-vertical,
main-horizontal, main-vertical or tiled
- defines a tmux layout, option 'layout' and (optionally) maximum number of panes in one window, 'max-panes', default 9
- with option custom, custom tiled layout
- defines filed layout with maximal size of rows, 'max-vertical', and columns, 'max-vertical'. There is also an (optional) option 'panes-flow' to specify if the panes flow from left to right (horizontal - default) or from top to bottom (vertical)
If you don't specify layout, 'tmux tiled' will be used
The layouts are applied individually to any merge group and to any normal (regex) group not belonging to some merge group. If there are more servers in a group then layout allows on a single window, next window for that group is added. Servers from different groups never share a window.
Take a look at
spec/fixtures/configs.yml for some configuration
possibilities (sections under 'input' fields).
Sessions without configuration files
As mentioned in previous section, normally you need the configuration file to start the session.
This is not true for some smaller sessions where you can directly specify all necessary configuration in command line when starting the session.
You can use
-q) to start a quick session. Quick
session arguments must conform to the following format:
<regex>is similar to 'regex' filed in configuration file, but without the sorting part (which is presumably last part of the full regex)
<upper-limit>are optional elements and define range of valid servers. Range is enforced on sorting part (which is not part of previous regex, but presumably comes after it)
<optional-additional-arguments>is list of optional arguments. This list starts with ' :: ' and consists of key-value pairs separated by semicolon. Currently only
varguments are supported. Format:
:: h <max-horizontal-panes>; v <max-vertical-panes>
tcon start -q "dev.node-staging-[[1,24]]"
tcon start -q "dev.node-staging-[[42,]]"
tcon start -q "dev.node-staging-[[,]]"
tcon start -q "dev.node-staging-[[,42]] :: h 2; v 2"
Example with more complex regex (selects 2 kinds of servers):
tcon start -q "dev.(nginx|haproxy)-staging-[[10,30]] :: h 2; v 2"
To be able to use the gem you should have ruby 1.9+ and tmux installed on a *nix (Mac OS X, Linux, ...) machine. (Windows: here be dragons)
Interaction with tmux is done via bash commands.
Minimal familiarity with tmux is required. For a start, switching between panes/windows and attaching/detaching is enough:
- detach session:
- attach session:
tmux attach -t <session-name>
- navigate windows (next/previous):
- navigate panes:
- toggle pane zoom:
(prefix is by default
The gem provides CLI and currently it is not intended to be used as part of bigger ruby apps.
Install it with:
$ gem install tmux-connector
If tmux isn't already installed, install it using your favorite mathod, e.g.:
apt-get install tmux
- Mac OS X:
brew install tmux
SSH config files
If you plan on specifying separate ssh config file when starting session, consider adding the following lines on top:
StrictHostKeyChecking no UserKnownHostsFile=/dev/null
That way you won't have problems with known hosts changes or with infinite questions to approve new hosts. Do this only if you understand security consequences and are sure that it is safe to do so.
Sending the commands
You need to run the send command from somewhere. There are (at least) 2 options:
- have a controller pane (or the whole window) inside the tcon session
- a pane not connected to a server to issue the commands to other panes
- have a separate terminal pane for issuing the send commands
- I'm a iTerm2 user, and so I split the window vertically: much bigger top pane for the tcon session, and a smaller bottom pane to issue the send commands
- Fork it
- Create your feature branch (
git checkout -b my-new-feature)
- Commit your changes (
git commit -am 'Add some feature')
- Push to the branch (
git push origin my-new-feature)
- Create new Pull Request
Or just mail me, mail: ivan@<< username >>.com
This is my first real gem, so all your comments are more than welcome. I'd really appreciate ruby code improvements/refactoring comments or usability comments (all other are welcome too). Just drop me a line. :)
Comments, ideas or if you feel like chatting
ivan@<< username >>.com
I'd be happy to hear from you.
Also, visit my homepage.