Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(k8s): in-cluster building #808

Merged
merged 8 commits into from
Jun 5, 2019
Merged

feat(k8s): in-cluster building #808

merged 8 commits into from
Jun 5, 2019

Conversation

edvald
Copy link
Collaborator

@edvald edvald commented May 31, 2019

Moving this from our closed-source version. This adds an option to build
container images inside Kubernetes clusters, and by extension removes the
need for Docker and Kubernetes running on developer machines.

Use it by setting buildMode: "cluster-docker" in the kubernetes provider
configuration.

@edvald edvald force-pushed the remote-build branch 2 times, most recently from fd27897 to 88f5b8e Compare May 31, 2019 13:34
eysi09
eysi09 previously approved these changes May 31, 2019
@edvald
Copy link
Collaborator Author

edvald commented May 31, 2019

Note: This needs a little more testing before merging. I think I see some merge-related issues that the compiler isn't catching.

@eysi09 eysi09 dismissed their stale review May 31, 2019 13:39

Something's not right.

@edvald
Copy link
Collaborator Author

edvald commented May 31, 2019

Ok, resolved the issues I could find. But it'd be great to get more hands on this, to test and make sure I didn't mess anything else up in the merge.

@edvald edvald force-pushed the remote-build branch 4 times, most recently from aaf0257 to c4df7c2 Compare May 31, 2019 15:31
@eysi09
Copy link
Collaborator

eysi09 commented Jun 1, 2019

Commands always fail with:

Plugin local-kubernetes has been updated or hasn't been configured, and requires user input.. Please r
un `garden init` and then re-run this command.

Even though I've already run garden init for the respective env.

This issue seems to have been introduced in #803.

@eysi09
Copy link
Collaborator

eysi09 commented Jun 1, 2019

Also, getSystemServicesStatuses always returns:

{
  'docker-daemon':
  {
    state: 'outdated',
      version: undefined,
        detail: { remoteObjects: [Array] }
  },
  'docker-registry':
  {
    state: 'outdated',
      version: undefined,
        detail: { remoteObjects: [Array] }
  },
  'registry-proxy':
  {
    state: 'ready',
      version: 'v-cc8c98d873',
        detail: { remoteObjects: [Array] }
  }
}

which results in Garden initialising the env on every command.

EDIT:

This is after I comment out the line which always sets needManualInit to true.

@edvald edvald changed the title feat(k8s): in-cluster building feat(k8s): in-cluster building WIP Jun 3, 2019
@@ -158,8 +158,8 @@ export class ActionHelper implements TypeGuard {
if (!allowUserInput && needManualInit.length > 0) {
const names = needManualInit.map(s => s.name).join(", ")
const msgPrefix = needManualInit.length === 1
? `Plugin ${names} has been updated or hasn't been configured, and requires user input.`
: `Plugins ${names} have been updated or haven't been configured, and require user input.`
? `Provider ${names} has been updated or hasn't been configured, and requires manual initialization.`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a dot at the end of the prefix and where the message is printed, resulting in two dots.

? `Plugin ${names} has been updated or hasn't been configured, and requires user input.`
: `Plugins ${names} have been updated or haven't been configured, and require user input.`
? `Provider ${names} has been updated or hasn't been configured, and requires manual initialization.`
: `Providers ${names} have been updated or haven't been configured, and require manual initialization.`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same.

@eysi09
Copy link
Collaborator

eysi09 commented Jun 4, 2019

There's still the issue with needsManualInit being always set to true for the remote provider. As a result I never get passed:

Provider kubernetes has been updated or hasn't been configured, and requires manual initialization.. Please run `garden init` and then re-run this command.

@edvald
Copy link
Collaborator Author

edvald commented Jun 4, 2019

Ah, I misunderstood the flow, I thought that would only kick in if the environment wasn't reported as ready. I'll fix that.

),
})
} else if (systemServiceStatuses.state !== "ready") {
systemReady = false
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this not be accounted for where systemReady is set above? Would be more readable imo.

Copy link
Collaborator Author

@edvald edvald Jun 4, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The statement would then look like this:

systemReady = systemTillerReady && sysNamespaceUpToDate && (
      systemServiceStatuses.state === "ready"
      ||
      (needManualInit && systemServiceStatuses.state === "outdated")
    )

Guess that's in the eye of the beholder, I'm happy to go with either.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not the prettiest of assignments but IMO the control flow is slightly more readable this way. But I can swing either way of course.

@eysi09
Copy link
Collaborator

eysi09 commented Jun 4, 2019

LGTM. Did some very basic tests but still working on running the whole thing through our integ suite, got stuck on a tangent earlier today.

@edvald edvald changed the title feat(k8s): in-cluster building WIP feat(k8s): in-cluster building Jun 5, 2019
Moving this from our closed-source version. This adds an option to build
container images inside Kubernetes clusters, and by extension removes the
need for Docker and Kubernetes running on developer machines.

Use it by setting `buildMode: "cluster-docker"` in the `kubernetes` provider
configuration.
…iffs

Also improved the display of diffs when verbose logging, to make these
types of issues a little easier to debug.
eysi09 and others added 3 commits June 5, 2019 14:13
This is a temporary fix before we manage to fully normalize manifests while
diffing, i.e. when server-side apply is widely available for all/most
resources.
@edvald edvald merged commit 1fb89fc into master Jun 5, 2019
@edvald edvald deleted the remote-build branch June 5, 2019 13:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants