Remove RabbitMQ and scheduler

[jvehent]

RabbitMQ is powerful but hard to operate, particularly when managing clusters of multiple nodes that break from time to time. We don't benefit from the pub/sub features of AMQP as most interactions are 1 to 1: scheduler to agent, agent to scheduler. Only event workers use round robin distributions, but those can easily be rewritten to poll the DB.

With HTTP/2 now standard in Golang, we should simply the backend architecture and get rid of RabbitMQ entirely in favor of a full Rest API interaction. Agents should authenticate to the API using tokens, maintain a connection open, and receive actions via server push.

This is a tough one. The amount of code we're touching is very large.

[lukebeer]

Over the past year or so I've produced about 6 variants of Mig that in attempt to remove the dependancy on RabbitMQ in various ways. End goal I'd like to see on-demand data streams rather than persistant TCP connections to a central point. Some kind of automatic protocol selection with plugable transports (eg: allow staff laptops to traverse strict customer networks) but nothing worth sharing yet.

After much experimentation, it seems Serf and its gossip protocol implemented in Mig coupled with optional central long term storage would be the best approch for my use cases however I've not even considered HTTP/2 so will have a read up on that.

ps. Blockchain/DHT?

[jvehent]

We'd like to go with a full HTTP/2 implementation and a simplified architecture. The end goal is to get rid of both RabbitMQ and the scheduler, and centralize all features on the API (then renamed mig-server).

The API will need 5 new endpoints:

1. POST /authenticate will receive a jwt from an agent, verify it and create a persistent HTTP/2 connection with the agent.
2. POST /agent/heartbeat will receive heartbeats from agents and write them to database. In the database, we'll associate the agent heartbeat with the API node currently holding the connection so we can do server push later on.
3. push command will implement HTTP/2 server push of mig commands to agents. This is the bit that needs identification of the API endpoints that holds an agent connection so we can scale the number of API nodes horizontally.
4. GET /agent/command will be used for agents that may have been disconnected to check if there are pending commands for them to run
5. POST /agent/result will be used by agents to post command results to the API.
6. POST /agent/event is a future endpoint we may need to stream events coming from agents to destinations, like a local disc or a queue. This endpoint should not write to the database, it's for log streaming.

We'll also need to move the periodic tasks from the scheduler to the api, but that's just adding a new goroutine that will run on a single host (lock in db?).