Revised architecture with an Elixir-based test agent manager

[charypar]

More of a place for conversation, rather than a serious suggestion:

How would you like an architecture with a manager in the middle?

Manager could be responsible for

1. Collecting and queuing test jobs
2. Managing the pool of containers, potentially reusing them and using ECS to spawn and stop them
3. Running test jobs on containers
4. Handling results (processing, storage) and streaming them back to the browser clients

Given this is mostly orchestration, processing and data streaming, it could be quite good to build this in Elixir.

The benefits I see:

• Removes the central dependency on ECS and the necessity of running two servers (one long running web server, one server on the agent itself) and managing the web server
• Makes the architecture independent of cloud provider, as long as there is a container platform
• Removes the need for streaming data between the two servers in favour of Elixir messages
• Gives you more control over resource usage - potentially a faster response to demand but also a cap on the number of running containers.
• Should make the entire system very, very reliable and scalable (yay OTP)
• Elm front-end can be served from S3 (this is actually also true in the current system)

Drawback is obviously that none of us really know much Elixir and you also need to worry about running the Elixir system.

Thoughts?

[lpil]

Drawback is obviously that none of us really know much Elixir

[charypar]

I was hoping for a wild @lpil to appear!

[lpil]

I think this would be a fine use case for Elixir. Streaming little bits of data about through persistent connections is the name of the game.

request
-> spin up a worker process
-> process spins up the chrome ECS worker
-> ECS worker connects to Elixir app to push results stream
-> Elixir app pushes data to the user, possibly via phoenix channels
-> Elixir worker process cleans up afterwards

I think it'd be easier to do this stateful programming in Elixir than Node, for sure.

[lpil]

Actually is there any reason why the result data needs to go via the app? The VPS could host a little server streaming the data over websockets and the client can connect directly to it.

Less work for the persistent server and no more requirement to route the results data across the cluster to the node where the client socket is held, which will make things simpler.

[jameshopkins]

Yep, I think this would be an awesome idea to explore/spike.

The architecture as it is, was purposefully extracted into single concerns so that we could converge on solution that involved less dependencies in the future.

Comments inline:

Removes the central dependency on ECS and the necessity of running two servers (one long running web server, one server on the agent itself) and managing the web server

My initial thinking was that the orchestration of the test runs would be deferred to cloud infrastructure. The current proposed interface between the application and test agent pool (ECS) is a naive call using the AWS SDK to spawn a container. However, it'd be more manageable if the interface between the two mechanisms would sit closer together.

Removes the need for streaming data between the two servers in favour of Elixir messages

That's a big win as far as I'm concerned. Internalising this communication would be ideal 👍

I'm going to go away and do a bit more of an investigation into Elixir, but at this early stage of the project, I'm not concerned with changing the underlying interfaces frequently and dramatically - i.e I'm up for considering anything that people deem would be a better fit. Greenfields, people!