Enable CI

[kirs]

This is the scrappy version of the CI running for master and branches. Surprisingly, it took me less than an hour to set it up (big thanks to @sander-m for help with BK and VMs) 🎉

Important things:

• Only Shopify employees can access the CI page, however the CI status is reported back to GH and visible for everyone
• I didn't enable it for the PRs on purpose because it's a source of vulnerabilities. This is a public project and anyone could submit a PR with malicious code in bin/ci that would be executed on Shopify VMs. Technically the VMs are isolated but still I wouldn't like to give this opportunity to people outside of the company.

How it works:
Mobile CI at Shopify provides disposable Ubuntu VMs with enabled KVM. We have them integrated with BuildKite, and these computing resources are available to run CI for this project. In bin/ci script we install kubectl, minikube, docker kvm driver, and ruby dependencies. Them we simply run rake test.

It takes 2m40s to start minikube, and the complete test suite runs in 20 seconds. This sounds good enough for me, but in future we can speed it up by pre-building the VM template with minikube and all dependencies installed.

Preview:

review @sirupsen @KnVerey @ibawt @wfarr

[ibawt]

:O

[KnVerey]

😻

• Is it named rubernetes-deploy gem instead of kubernetes-deploy gem on purpose?
• Can we include the pipeline.yml for this in the repo?
• I'm ok with doing it a bit later, but I think prebuilding the VM template will definitely be worthwhile. The suite takes 20 seconds now, but that's with a single integration test. On my branch that adds ~10 more, we're already up to 70 seconds.

[KnVerey]

Couple of things I'd like to confirm about the defaults we're using here:

• --cpus is 2 and --memory is 2048. Seems reasonable to me but ¯\_(ツ)_/¯
• --disk-size is 20g. Seems a bit excessive? MinimumDiskSizeMB = 2000 (from minikube code), and I've been using that much locally.
• --kubernetes-version is available. Perhaps we should link this to the version we download above.

[kirs]

Do you mean you'd like to not rely on defaults and run it with minikube start --cpus 2 --memory 2048 --disk-size=2gb --kubernetes-version=x.x.x?

[sirupsen]

The idea of specifying this explicitly with values that the Dev Acceleration team is comfortable with (cc @sander-m) sounds good to me. 👍 Any opinions on values Sander?

[sander-m]

• --cpu 2 👍 because these VM instances have 2 cores.
• --memory 2048 The VM has 8GB RAM available, so this is fine but you could bump it if you want
• --disk-size=2gb You can expect roughly 20GB of free space on the VM, but we can increase this for the custom template if you need more.

[KnVerey]

Can we wait for this to be "Running" before trying to start the tests? Looks like it was immediately in your test runs, but perhaps that wouldn't always be the case.

[kirs]

minikube start waits until the cluster is running and then connects to it to verify things. So the check shouldn't be required.

If you think the check is necessary, we can put test -n "$(minikube status | grep 'Running')" into the bash script. It will fail if it couldn't grep the status for "Running".

[sirupsen]

Why have minikube status at all then? In case it fails? What's the timeout on minikube start in case it fails?

[kirs]

There's no timeout in minikube start that I could find. It may be worth adding it - I'll discuss it with the mikube maintainer.

I agree that taking the facts that I mentioned above, minikube status isn't necessary. I'll remove it.

[kirs]

It's also worth mentioning that if in the worst case minikube start gets crazy and starts to take forever, we have the global Buildkite timeout.

[sirupsen]

I am assuming Shopify employees can trigger a build manually? Can you put that in the README as well to be required? If we ship the gem with ShipIt already, this shouldn't be an issue. I'm really concerned about this test suite getting rusty... complicated VM test suites that rely on multiple versions of things to work together are very prone to rust.

[sirupsen]

This is pretty awesome Kir... I'd +1 to all of Katrina's comments. Great work. Thanks @sander-m!

[sirupsen]

Can you extract all these versions to environment variables? I'd be a little concerned that these things get out of date. How about writing a test that asserts the versions used in this file are the same as we use in development?

Just a sketch in code (haven't run any of this so don't know if it works, but it illustrates an idea):

test "check versions with CI" do
  versions = File.read("omg.sh").scan(/([\w_]+)_VERSION="([\d\.]+)"/)
  versions.each do |(util, version)|
    case util
    when "KUBECTL"
      dev_version =`kubectl version | grep -oP "(?<=v)[0-9]+\.[0-9]+\.[0-9]+"`.strip
      assert_equal dev_version, version
    when "DOCKER_MACHINE"
      # compare somehow
    when "MINIKUBE"
       # compare somehow
    else
       flunk "#{util} was not defined to compare with development..."
    end
  end
end

Alternatively we could enforce this somehow else, e.g. booting test/test_helper.rb to make sure both dev and CI runs against the same versions.

[kirs]

I'm concerned about minikube and kubectl versions too. The problem is that in future they will be installed in the VM template that doesn't have the context of the repo.

I'm going to move versions to environmental variables and in the next PR I'll introduce the version check in test helper.

[kirs]

I am assuming Shopify employees can trigger a build manually? Can you put that in the README as well to be required? If we ship the gem with ShipIt already, this shouldn't be an issue. I'm really concerned about this test suite getting rusty... complicated VM test suites that rely on multiple versions of things to work together are very prone to rust.

Shopify employees push code to branches, and the branch builds are automatically triggered. AFAIK it also means that PRs from Shopify employees are also built automatically. It will also build git tags (== gem releases).

[wfarr]

Shopify employees push code to branches, and the branch builds are automatically triggered. AFAIK it also means that PRs from Shopify employees are also built automatically. It will also build git tags (== gem releases).

This was the only thing I was waiting to hear back on. This all looks good to me!

[kirs]

Can we include the pipeline.yml for this in the repo?

I had exactly the same suggestion, but then we discussed it with @sander-m and came to the conclusion that since CI is a private thing, in this case we can keep the config in BK and don't expose it to the public.

I'm ok with doing it a bit later, but I think prebuilding the VM template will definitely be worthwhile. The suite takes 20 seconds now, but that's with a single integration test. On my branch that adds ~10 more, we're already up to 70 seconds.

Sure, working on the VM template is one of the next things in my list.

[sirupsen]

Shopify employees push code to branches, and the branch builds are automatically triggered. AFAIK it also means that PRs from Shopify employees are also built automatically. It will also build git tags (== gem releases).
This was the only thing I was waiting to hear back on. This all looks good to me!

That's... magic. ❇️

[sander-m]

LGTM. I can help out with the custom template.

[kirs]

I've addressed all concerns that were mentioned. Please let me know if there's any other feedback.

I have another updated about Buildkite:

BK will build branches as you're working on them, but as soon as you create a PR it will stop building new commits from that branch because it's a PR and we disabled PR builds. The workaround for that is to trigger the build manually, which is very easy:

[ibawt]

fyi @stefanmb

[KnVerey]

BK will build branches as you're working on them, but as soon as you create a PR it will stop building new commits from that branch because it's a PR and we disabled PR builds. The workaround for that is to trigger the build manually, which is very easy:

Can we add this to the readme in a "maintainers" section or something? Otherwise LGTM.

[KnVerey]

kubernetes-deploy-gem (I see you fixed it in buildkite)

[kirs]

🎉🎉🎉

@KnVerey great suggestions. I've fixed both things.