Gagent is a self-replicating, messaging network of remote agents capable of running commands visualizing or processing the output.

Overview

Start an interactive command from a UNIX shell which allows you to target (a potentially large set of) machines by sending commands and responding to output. For example, run the "date" command on all machines of a cluster to eyeball whether NTP is likely running.

Chef and Ansible are heavyweight with bulky syntax ruby and python footprints. They solve higher level problems.

Salt and Puppet are declarative, also solving much more complicated problems.

Warning

Misuse of this module may be highly insecure and destructive. Please use a measure of caution who can connect to your daemon and what necessary access to remote systems your daemon may expose when you enable insecure settings.

Why Do This

Besides potentially offering a small-footprint solution for remote execution which might be extended to orchestrate node drain replacement activities, I am brand new to Go and there will be a lot of opportunity to exercise many useful features of Go -- for example maybe channels to notify on received message, and non-trivial parallel execution tasks to transmit commands to multiple agents and then to coalesce output for presentation.

This is an exercise to learn how to write good Go.

What is Involved

Execute remote agent (after possible replication). Remote agent subscribes to request messages. Upon request, run "bash -c 'command'" and capture its stdout, stderr, exit status, timing info, etc. into a message and publish this response somewhere. Remote agent also handles: ping, read file, write file, and info.

A daemon can send and receive messages with agents. A daemon may use SSH to start and manage multiple remote agents, and communicate with STDIN and STDOUT of those connections to transmit messages. A daemon may also listen (e.g. TLS on port 9007) for connections from other daemons, or console.

Readline-based interactive console accepts commands from user, sends command requests to remote agents through a daemon. Console attempts connection to local daemon, and on failure, starts a local daemon, and re-attempts connection to it. Other daemon clients may generate requests and process responses. Daemon may remain resident after the daemon client exits.

When daemon receives a request for a machine for which it cannot already send and receive messages, the daemon uses SSH to do the following activities: test for existence of agent installation, install agent, execute remote agent, send and receive command requests and responses with that remote agent.

The console binary is a single file capable of handling all the roles above. As such, this binary can be copied to the remote machine before being invoked with arguments to apply the remote agent behavior.

Command requests have unique IDs, may be sent asynchronously, so responses may arrive out of order. Agent is expected to respond immediately to pings, so daemon sends regular heartbeats, and can use SSH to verify whether the remote process is running, and can kill one or more remote sub-processes to recover control of agent or can kill the agent itself.

Agent replies with the PID of sub-process immediately after starting the command. Then after the sub-process exits, the agent sends another message containing the response. Agent may refuse command requests if its command queue exceeds a maximum threshold. Each command executes in its own directory, which is typically removed upon successful communication of the response.

A config file and a private state file can be used to define tags related to machines and to keep track of whether or where the remote replication exists. The config file can decide whether to connect to a resident daemon, and for shared accounts, whether to require user identify herself upon console startup.

Concepts

• agent
• agent/handler
• handler
• handler/agent
• process
• config
• state
• queue
• daemon
• console

Use Cases

• console interacts with user
• console attempts message to daemon
• console creates daemon process
• console sends and receives messages to daemon
• daemon attempts message to agent
• daemon replicates to remote machine
• daemon creates agent process
• daemon sends and receives messages to agent
• daemon logs consoles, commands, and responses
• agent replies to ping
• agent replies to info
• agent reads or writes a file
• agent runs a command, captures and transmits output
• queue notifies listeners of availabilty of incoming message
• queue allows atomic, thread-safe pop() for multiple threads

Keys

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
chmod +x cfssl_linux-amd64
sudo mv cfssl_linux-amd64 /usr/local/bin/cfssl

wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
chmod +x cfssljson_linux-amd64
sudo mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

mkdir keys

echo '{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "kubernetes": {
        "usages": ["signing", "key encipherment", "server auth", "client auth"],
        "expiry": "8760h"
      }
    }
  }
}' > ca-config.json

echo '{
  "CN": "Gagent",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "US",
      "L": "Austin",
      "O": "Gagent",
      "OU": "CA",
      "ST": "Texas"
    }
  ]
}' > ca-csr.json

cfssl gencert -initca ca-csr.json | cfssljson -bare ca