New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove RabbitMQ and scheduler #238

Open
jvehent opened this Issue Jun 6, 2016 · 2 comments

Comments

Projects
None yet
3 participants
@jvehent
Copy link
Contributor

jvehent commented Jun 6, 2016

RabbitMQ is powerful but hard to operate, particularly when managing clusters of multiple nodes that break from time to time. We don't benefit from the pub/sub features of AMQP as most interactions are 1 to 1: scheduler to agent, agent to scheduler. Only event workers use round robin distributions, but those can easily be rewritten to poll the DB.

With HTTP/2 now standard in Golang, we should simply the backend architecture and get rid of RabbitMQ entirely in favor of a full Rest API interaction. Agents should authenticate to the API using tokens, maintain a connection open, and receive actions via server push.

This is a tough one. The amount of code we're touching is very large.

@jvehent jvehent added this to the 1.0 milestone Jun 6, 2016

@lukebeer

This comment has been minimized.

Copy link

lukebeer commented Jun 13, 2016

Over the past year or so I've produced about 6 variants of Mig that in attempt to remove the dependancy on RabbitMQ in various ways. End goal I'd like to see on-demand data streams rather than persistant TCP connections to a central point. Some kind of automatic protocol selection with plugable transports (eg: allow staff laptops to traverse strict customer networks) but nothing worth sharing yet.

After much experimentation, it seems Serf and its gossip protocol implemented in Mig coupled with optional central long term storage would be the best approch for my use cases however I've not even considered HTTP/2 so will have a read up on that.

ps. Blockchain/DHT?

@jvehent jvehent changed the title Remove RabbitMQ Remove RabbitMQ and scheduler Jun 17, 2016

@jvehent

This comment has been minimized.

Copy link
Contributor Author

jvehent commented Jun 17, 2016

We'd like to go with a full HTTP/2 implementation and a simplified architecture. The end goal is to get rid of both RabbitMQ and the scheduler, and centralize all features on the API (then renamed mig-server).

The API will need 5 new endpoints:

  1. POST /authenticate will receive a jwt from an agent, verify it and create a persistent HTTP/2 connection with the agent.
  2. POST /agent/heartbeat will receive heartbeats from agents and write them to database. In the database, we'll associate the agent heartbeat with the API node currently holding the connection so we can do server push later on.
  3. push command will implement HTTP/2 server push of mig commands to agents. This is the bit that needs identification of the API endpoints that holds an agent connection so we can scale the number of API nodes horizontally.
  4. GET /agent/command will be used for agents that may have been disconnected to check if there are pending commands for them to run
  5. POST /agent/result will be used by agents to post command results to the API.
  6. POST /agent/event is a future endpoint we may need to stream events coming from agents to destinations, like a local disc or a queue. This endpoint should not write to the database, it's for log streaming.

We'll also need to move the periodic tasks from the scheduler to the api, but that's just adding a new goroutine that will run on a single host (lock in db?).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment