Hyper.sh integration as a container alternative to ECS

[Gnep]

Hi,

Hyper.sh is a container-native IaaS, which allows developers to provision secure containers on demand without hosting VMs. It uses a per-secong billing model, a perfect fit to your serverless CI model.

[mhart]

The general usage of a container service isn't compatible with LambCI as it's not a FaaS solution.

However, I notice that Hyper has a new beta product, called "Hyper Func", which looks like it might be compatible, but it's unclear from their documentation how the pricing works (ie, are you only charged when your function is invoked, or do you have to start paying as soon as you deploy the function container?)

It still seems like it's in very early stages – but if you have any success with it, let me know!

[Gnep]

Hi @mhart

I'm a cofounder of Hyper.

We have the plugin for Jenkins and Buildbot. https://medium.com/buildbot/using-hyper-sh-to-deploy-your-buildbot-126888e1164d.

In Hyper, you don't provision a VM cluster to run containers. You simply launch containers on-demand. For CI/CD, think the build job as on-demand containers in the cloud, launch in 5 seconds, charged per second. And with Docker images as the function source, you can run any arbitrary code to build.

[mhart]

@Gnep that's all well and good, but as I'm sure you've read that doesn't match the LambCI model – with LambCI there are no idle containers or instances running.

Unless you have the ability in Hyper.sh to have zero containers running, and only spin one up when a web hook (or SNS hook or similar) is triggered, then it's not really the same model.

The link you posted implies that you need a buildbot-travis container running at all times to receive the github webhook.

[Gnep]

@mhart

Pls believe me that it is the SAME model. I wouldn't propose if user need to keep the slave container running. The container in the post is the master server (it does NOT have to run in Hyper.sh, it can run on your laptop).

Buildbot team has migrated their own CI/CD pipeline from EC2 to Hyper.sh (they built the integration plugin anyway), and saw a 5x build speed up ("unlimited" build concurrency), and 10x cost reduction from their previous EC2 bill!

I saw your docs has a section using ECS as the Docker build backend. Trust me, Hyper.sh is the true "No idle containers" service.

[mhart]

What do you mean you can run the master server your laptop? Do you mean... exposing an http server on your laptop to the world and then setting up your laptop's IP address as the GitHub webhook address? I'm still not sure how that's the same model as FaaS where you don't need to run anything on your laptop? And how does that work in a team? That seems completely different to me.

If you're talking about a solution using AWS Lambda as well as Hyper.sh, that might be possible. Let me know if you get hyper running on Lambda!

[Gnep]

@mhart

Think the Jenkins single node mode, Jenkins runs the build job using local Docker container. And actually Jenkins has no idea where the container, e.g. it calls the API and a container is provisioned. Therefore, as long as Hyper's REST API is consistent with Docker engine, Jenkins doesn't care.

As in the LambCI step, very tiny changes:

• Receives notification from GitHub (via SNS)
• Looks up config in DynamoDB
• Clones git repo using a bundled git binary
• Looks up config files in repo
• Runs install and build cmds on Lambda (or starts ECS task) --- or start a container in Hyper.sh to run cmd
• Updates Slack and GitHub statuses along the way (optionally SNS for email, etc)
• Uploads build logs/statuses to S3

And deliver the similar value as:

• 1000 concurrent builds out of the box (can request more) --- YES
• No maintenance of web servers, build servers or databases --- YES
• Zero cost when not in use (ie, 100% utilization) --- YES
• Easy to integrate with the rest of your AWS resources --- Not Hyper's focus

[mhart]

Ok cool, let's get back to this issue (Hyper.sh integration) – how are you proposing to integrate Hyper.sh with LambCI?

[mhart]

(more specifically – does hyper run on Lambda?)

[Gnep]

I think what you did in LambCI is a step of the disruption of CI/CD market, e.g. serverless CI/CD.

FaaS is one of the backend driver options. Container-native IaaS, like Hyper.sh or Azure ACI is another. And given that CI pipeline tends to be long running and requires arbitrary code, container is a better delivery format than FaaS.

As to Hyper.sh, I think the change would be: call Hyper's API to launch containers, clone git repo and run cmd. Just my two cents.

[mhart]

Cool – can hyper's API be called from Lambda?

[Gnep]

Yes, it is just REST API, and consistent with Docker. https://docs.hyper.sh/Reference/API/2016-04-04%20[Ver.%201.23]/index.html

[kshmir]

Given the problems with AWS ( #92 ), I think hyper as a secondary option would be really cool!

I tried hyper.sh for other reasons back in the day and it worked OK.

As I mentioned to you @mhart in the other PR, anything I could do to help you can contact me.

[mhart]

@kshmir this doesn't really make sense – using Hyper.sh here is an alternative to using ECS – not an alternative to building natively on Lambda, which is what #92 is about.

[Gnep]

@kshmir @mhart

• We have an integration with Jenkins: github.com/hyperhq/hykins
• Drone.io has a plugin too: Hyper.sh Integration cncd/pipeline#24
• And CNCF has a serverless WG, in their whitepaper, Hyper.sh is quoted as an example of Serverless CI: https://docs.google.com/document/d/1UjW8bt5O8QBgQRILJVKZJej_IuNnxl20AJu9wA8wcdI/edit#

I think people would prefer container for the build jobs, given the various limitations of FaaS.

[mhart]

Looks like hyper.sh shut down a while ago: https://news.ycombinator.com/item?id=18734658