Skip to content
/ clussh Public

Stream task execution, through ssh, to one or many host, one or many times, in parallel or in series.

License

Notifications You must be signed in to change notification settings

nodys/clussh

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

62 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Clussh

Build Status JavaScript Style Guide Clussh on npm

Stream task execution, through ssh, to one or many host, one or many times, in parallel or in series.

clussh

Features:

  • Distribute tasks upon many hosts
  • Stream task from stdin
  • Stream task outputs and progression stats to ldjson stream
  • Support scaling, parallel execution and retry, network failure and host unavailability

Installation

npm install -g clussh

This will install three command-line cli:

  • clussh - The main command line
  • clussh-board - A streamable clussh progress board
  • clussh-log - Pretty print clussh logs

Pre-requirement: NodeJS and its awesome universe

Usages & exemples

# Run "Hello $(hostname)" on the default worker ssh://yourusername@localhost
clussh

# Run given command on default worker
clussh --cmd 'echo "Hello world"'
clussh --cmd 'echo "Hello world"' --scale 10 # Teen times
clussh --cmd 'echo "Hello world"' --scale 10 --concurrency 2 # Two in parallel

# Run given bash script on default worker
clussh ./install.sh

# Run given bash script on to agents (with a concurrency of 2 for agent-b)
# (NB: for ssh cooking & baking reasons, the shell script must ends with an exit statement)
clussh -w ssh://user@agent-a.local -w ssh://user@agent-b.local?concurrency=2 ./runme.sh

# Pipe task from line delimited json stream
echo '
{ "id": "task-a", "cmd": "echo hello; sleep 1" }
{ "id": "task-b", "cmd": "echo hello; sleep 2", "scale": 10 }
{ "id": "task-c", "script": "./runme.sh", args: ["--option", "value"] }
{ "id": "task-d", "script": "./runme.sh", "worker": "ssh://only@on.host" }
' | clussh -w ssh://user@agent-a -w ssh://user@agent-b

# You can pipe infinite line delemited json stream (eg. event logs for a file wacher...)
infinite-stream | clussh

# Print human readable logs
clussh | clussh-log

# Print nice-looking progress board
clussh | clussh-board

# Print human readable logs
clussh | clussh-log

# Save logs for further exploration (and still display the board)
clussh | tee output.log | clussh-board
tail -f output.log | clussh-log   # Another console...
tail -f output.log | clussh-board # Another console...
tail -f output.log | ndjson-filter 'd.type === "fail"' | prettyldjson # Another console...

Command line usage and configuration

clussh [options] [script filepath]

Options:
  --help             Show help                                         [boolean]
  --version          Show version number                               [boolean]
  --retry, -r        How many retry for each task          [number] [default: 0]
  --timeout, -t      Task timeout in millisecond or parseable ms time (eg. 1h,
                     2d or `3 days`)                   [string] [default: "10d"]
  --worker, -w       Repeatable worker uri list
                                   [array] [default: "ssh://jponchon@localhost"]
  --concurrency, -c  Concurrency per worker                [number] [default: 1]
  --scale, -s        Default scale                         [number] [default: 1]
  --cmd              Command to execute                                 [string]
  --script           Script to execute (please ensure proper exit)      [string]
  --identity, -i     List ssh identity files to use        [array] [default: []]

Clussh can be configured with a .clusshrc file (see https://www.npmjs.com/package/rc)

{
  "worker": [
    "ssh://master@agent-a.local",
    "ssh://master@agent-b.local",
    "ssh://master@agent-b.local?concurrency=2"
  ],
  "concurrency": 4,
  "retry": 10
}

Above, one worker uri overide the default clussh configuration using url query string (?concurrency=2 for host agent-b.local). This will result in two workers for agent-b.local: one with the default concurrency (4) and another with a concurrency of 2.

Log message format

Clussh output line delemited json stream consumable by clussh-board or clussh-log or any other line delemited json tools (see prettyldjson for pretty print and ndjson-cli for map, reduce, filtering, etc.)

TODO: Fields documentation & messages types

API

const clussh = require('clussh')

const clusshStream = clussh({
  cmd: 'hello',
  worker: [ 'ssh://foo@bar.com', 'ssh://bar@foo.com' ],
  retry: 3,
  concurrency: 2
})

clusshStream.pipe(process.stdout)

License: MIT - Novadiscovery

About

Stream task execution, through ssh, to one or many host, one or many times, in parallel or in series.

Topics

Resources

License

Stars

Watchers

Forks

Packages

No packages published