Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Manifest Factorization through .flux.yaml config files #1848

Merged
merged 11 commits into from May 30, 2019

Conversation

@2opremio
Copy link
Collaborator

commented Mar 20, 2019

Want to test this? Go to https://github.com/weaveworks/flux-kustomize-example and follow the instructions.

This feature (behind the flag --manifest-generation) is now available in our prereleases repo: https://hub.docker.com/r/weaveworks/flux-prerelease/tags

This PR implements Generators and Updaters as specified in Generalizing Manifest Factorization in Weave Flux.

It is meant as a Proof of Concept and will only be merged if/when there is positive feedback.

One major wart is that Kubernetes doesn't offer Ephemeral containers yet. It's been in the works for quite a while ( kubernetes/kubernetes#59416 ) but until then there is no simple and (reasonably) secure way to run generators and releasers. For now, we will simply assume that the command executables must live in the flux container.

  • Abstract out manifest file operations (done by using an adapter of Manifests: ResourceStore)
  • Implement Generators and Updaters
  • Add the most popular factorization tools to the Flux container image (only Kustomize for now)
  • Create a comprehensive example (https://github.com/2opremio/flux-kustomize-demo)
  • Address FIXMEs TODOs and performance issues
  • Hide behind an experimental flag
  • Add more unit tests
  • Circulate concept through the community to gather feedback

Fixes #943 #1261 #1811 (and #1420 to certain extend)

@2opremio 2opremio referenced this pull request Mar 20, 2019
3 of 3 tasks complete
@2opremio 2opremio force-pushed the 2opremio:generators-releasers branch from df0cd55 to 977a866 Mar 20, 2019
@2opremio 2opremio force-pushed the 2opremio:generators-releasers branch from 977a866 to f5b31c2 Mar 29, 2019
@2opremio 2opremio changed the title Proof of Concept: Generators and Releasers Proof of Concept: Generators and Updaters Mar 29, 2019
@2opremio 2opremio force-pushed the 2opremio:generators-releasers branch 5 times, most recently from 3fc3952 to 7f54431 Apr 2, 2019
cluster/manifests.go Outdated Show resolved Hide resolved
resourcestore/resourcestore.go Outdated Show resolved Hide resolved
@2opremio 2opremio force-pushed the 2opremio:generators-releasers branch 12 times, most recently from c83d6de to 15dad2f Apr 4, 2019
@2opremio 2opremio force-pushed the 2opremio:generators-releasers branch 5 times, most recently from 5cf9625 to db644b0 Apr 11, 2019
cmd/fluxd/main.go Outdated Show resolved Hide resolved
@2opremio 2opremio marked this pull request as ready for review Apr 11, 2019
@2opremio 2opremio force-pushed the 2opremio:generators-releasers branch 5 times, most recently from fd0e871 to d11ea84 May 29, 2019
@2opremio 2opremio force-pushed the 2opremio:generators-releasers branch 3 times, most recently from eeb9cf4 to 8d27f07 May 29, 2019
@2opremio

This comment has been minimized.

Copy link
Collaborator Author

commented May 29, 2019

Post-merge checklist:

@2opremio 2opremio force-pushed the 2opremio:generators-releasers branch from 8d27f07 to 2258bf7 May 29, 2019
2opremio and others added 11 commits Mar 20, 2019
* Remove dependency on `git.Checkout` and rename `checkoutManager` to
  `fileResourceStore`
* Inline configfilemanager and manifestfilemanager into the new
  `fileResourceStore`
PolicyTranslator exists because the `.flux.yaml` files give commands
for updating annotations, rather than updating policies, following on
from how KubeYAML was used. But the fact that policies appear as
annotations in Kubernetes resources is really up to the code in
cluster/kubernetes.

This commit changes the format of `.flux.yaml` files so a command for
updating _policies_ is supplied, rather than for annotations. All the
translation of policies to annotations is then done in
cluster/kubernetes.

There still needs to be a little code to evaluate the qualified policy
updates (i.e., tag_all) against a particular workload, but this can be
done without reference to whether they end up as annotations.
Also, fix git file adding (the CommitAndPush check doesn't detect untracked files).
@squaremo squaremo force-pushed the 2opremio:generators-releasers branch from 2258bf7 to b55d0ff May 30, 2019
Copy link
Member

left a comment

This is a substantial amount of work Fons. Well done on the thoughtful restructuring of the code, and pushing the PR through reviews, pre-releases, and other trials.

As a last smoke test, I tried the image built from this branch locally, both with the flux-example repo (and --manifest-generation=false) and the repo flux-kustomize-demo (--manifest-generation=true).

OK, let's merge this and let it soak in master ..

@squaremo squaremo merged commit e81b697 into fluxcd:master May 30, 2019
1 check passed
1 check passed
ci/circleci: build Your tests passed on CircleCI!
Details
@2opremio 2opremio deleted the 2opremio:generators-releasers branch May 30, 2019
@2opremio

This comment has been minimized.

Copy link
Collaborator Author

commented May 30, 2019

Thanks @squaremo !!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.