Networked Process Manager to execute functions in a Computer Grid
Switch branches/tags
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
docs documentation Apr 28, 2017
grid-api fix grid-api broadcast Apr 28, 2017
grid-cli install script with yarn Apr 28, 2017
research add pako for potencial test in future Jul 3, 2016
sample-proj changes Dec 19, 2016
src restore waiting exec Apr 28, 2017
subprojects back tcp May 15, 2016
test fix test Apr 28, 2017
.gitignore progressive spread Apr 28, 2017
.npmignore small fixed Jun 22, 2016
.travis.yml (test) add behavioral tests Sep 4, 2016
CHANGELOG.md #29 Apr 28, 2017
CONTRIBUTING.md documentation Apr 28, 2017
Dockerfile 0.9.0 Aug 13, 2016
LICENSE.txt new structure Apr 25, 2016
README.md README update Apr 27, 2017
TODO.md fix grid-api broadcast Apr 28, 2017
index.js download progress May 28, 2016
jsdoc.json prefix namespace + update jsdoc gridcontrol + update doc + add CONTRI… May 14, 2016
logo.png blue logo - cyan list May 25, 2016
logo2.png compare both logos May 25, 2016
npm-shrinkwrap.json 0.12.0 Dec 19, 2016
package.json gridcontrol@0.13.0 Apr 28, 2017

README.md

Build Status

GridControl

GridControl provisions and links multiple servers together to form a Grid.

Files are synchronized, Opinionated Pub/Sub system is implemented, Servers get linked together.

You develop, you play, in a scalable way. The more Servers you add to the Grid, the more calculation power you get.

Welcome to the Grid

5 minutes to get started. By the authors of PM2.

Behind the scenes: GridControl is a network layer built on top of PM2 allowing file synchronization, inter-process communication via an opionated PUB/SUB system and a wide-range system discovery

Features

  • 0 conf Auto discovery system via multiple DNS
  • 0 conf P2P application source sharing
  • Ecosystem Grid management toolbox (CLI, provisioning, Logs, Monitoring)
  • Secure Diffie Hellman key exchange and password authentication
  • Decentralized Each Node can trigger actions executed by another Nodes
  • Fast Grid interconnected via TCP sockets
  • Fast Services are started once, then stay alive waiting for inputs. This saves non-negligible startup time
  • Polyglot Services can be written in any language
  • Compatible with Amazon Lambda, Google Cloud Functions
  • Rock Solid PM2 behind the scene for process management and cluster capabilities
  • And a lot more like Buffering, Retry on Failure...

Creating a Grid

Install your Swiss Army Knife to manage a Grid:

$ npm install grid-cli -g

Now the bin grid is available via the CLI.

Commands

1/ Generate a new Gridfile in the current path that contains grid name, grid password, host and SSH keys:

$ grid new

The Gridfile will look like this:

grid_name     = 'grid-name'
grid_password = 'xxxx'

servers = [
  'user@ip1',
  'user@ip2',
  'user@ip3'
]

ssh_key = '''
1024_PRIVATE_SSH_KEY
'''

ssh_public_key = '''
PUBLIC_SSH_KEY
'''

Change each attribute to the desired value. Note that an SSH client should be running on the defaut port (22) on each remote machine

2/ Provision every host listed in the Gridfile:

$ grid provision

This will copy the SSH pub key and install NVM, Node.js, PM2 and Gridcontrol This installation does not need ROOT access rights at any time

3/ Grid management

$ grid dash

Grid dashboard

Commands to manage your grid:

# List all nodes linked to the grid
$ grid ls

# Display Dashboard
$ grid dash

# Execute a command on each server
$ grid multissh <bash_command>

# Restart/Recover the current Grid
$ grid restart

# Upgrade Gridcontrol to latest version
$ grid upgrade

# Display realtime logs of all tasks
$ grid logs

# Monitor the whole Grid with Keymetrics
$ grid monitor <secret> <public>
$ grid unmonitor

# Interactively SSH into desired machine
$ grid ssh

Interact with the Grid

Now let's play with the Grid. You can generate a sample project by typing:

$ grid sample [project-name]
$ cd [project-name]
$ npm install

Now you'll have a project that looks like this:

.
├── index.js
├── package.json
└── tasks
    └── get-ip.js

Let's look at the content of tasks/get-ip.js, this is a task that will be propagated in the grid:

var request = require('request');

module.exports = function(context, cb) {
  request('https://api.ipify.org/?format=json', function (error, response, body) {
    if (error)
      return cb(error);

    if (!error && response.statusCode == 200) {
      return cb(null, body);
    }
  });
};

To call this function, look at how it's done in index.js:

var grid = require('grid-api').init({
  instances   : 1,
  env         : {
    NODE_ENV  : 'development'
  }
});

function triggerTask() {
  // 'get-ip' is the filename
  grid.exec('get-ip', function(err, data, server) {
    if (err) {
      console.log(err);
      return false;
    }
	  console.log('Got response from server pub_ip=(%s) priv_ip=(%s):',
                server.public_ip,
                server.private_ip);
    console.log(data);
  });
}

grid.on('ready', function() {
  console.log('Gridcontrol Ready');
  setInterval(triggerTask, 1000);
});

Start the main application:

$ node index.js

At the beginning, only the local gridcontrol will respond. Once the other peers are synchronized they will also process the queries:

Got response from server pub_ip=(88.123.12.21) priv_ip=(10.2.2.1):
$HTML
Got response from server pub_ip=(88.123.12.22) priv_ip=(10.2.2.8):
$HTML
Got response from server pub_ip=(88.125.120.20) priv_ip=(10.2.2.10):
$HTML
Got response from server pub_ip=(88.123.23.42) priv_ip=(10.2.2.12):
$HTML

Distributed processing, on-premise!

Contributing

If you find any issues please open an issue on Github

For Contributing please refer to docs/CONTRIBUTING.md

License

Apache V2 (see LICENSE.txt)