Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow flux to watch a git tag #2568

Open
antonosmond opened this issue Oct 31, 2019 · 11 comments
Open

Allow flux to watch a git tag #2568

antonosmond opened this issue Oct 31, 2019 · 11 comments
Labels

Comments

@antonosmond
Copy link
Contributor

@antonosmond antonosmond commented Oct 31, 2019

Describe the feature
Flux currently allows you to watch a git branch and a path within a repo but you can't watch a git tag, as far as I can tell.

It'd be really useful if flux could watch a tag in git.

Before I explain the problem I'm trying to solve, it's worth noting that we're using flux for applying static manifests only and we're not using helm/tiller or any of the docker image watching functionality.

Currently, we're adopting git flow and have release branches with SemVer for versioning which are eventually merged to master and tagged.
Unfortunately, we also have a requirement to support multiple "active" versions of our software releases across multiple k8s clusters, for example:

cluster-a: my-app/v1.2.0
cluster-b: my-app/v1.1.3

This means we can't configure flux to watch the master branch of the my-app repo as HEAD may not be the version we want. Instead we have to watch the respective release branches. The problem here is that to upgrade from one version to another it requires re-configuring flux i.e. changing the branch that it watches.

If flux could watch a tag, it'd work nicely with git flow i.e. once we've merged our release branch to master and tagged it with the relevant version, we could create an additional tag for flux to watch and move the tag as needed. This negates the need to reconfigure flux. As an example, assume the below are tags in the master branch for the my-app repo:

c17f64b4: v1.2.0, flux-cluster-a
73f2c51d: v1.1.3, flux-cluster-b
7cea7fbd: v1.1.2

Given the above, if we wanted to upgrade the version of my-app in cluster-b, we'd just "move" the flux-cluster-b tag to be aligned with the v1.2.0 tag.

This would enable us to upgrade/rollback releases with ease simply by moving the tags around in git and we wouldn't have to reconfigure flux each time.

What would the new user story look like?
How would the new interaction with Flux look like? E.g.
What are the prerequisites for this?

  1. User starts up Flux providing the --git-tag flag

Expected behavior

  1. Flux checks out revision that --git-tag points to and applies the manifests
  2. Git tag is moved to point at a new revision
  3. When flux next polls the repo, it detects the revision to which the tag points has changed, checks it out and kubectl applies the changes

If people agree that this functionality could be useful, I'd be happy to start working on it and submit a PR.

@Perdjesk

This comment has been minimized.

Copy link

@Perdjesk Perdjesk commented Nov 4, 2019

I have been looking at the support for tag triggered fluxcd but with slightly different use case than yours.

Based on https://github.com/weaveworks/flux-kustomize-example how to handle the code in the repository using a trunk-based git workflow with SemVer releases instead of triggering a sync at each commit in refs/heads/master (using --git-branch). The end-goal would be to be able with the above repository and fluxcd instances per environment (using --git-path) to have different SemVer releases in each environment. This would help to validate and test changes done in the shared basemanifests in a stage environment before applying them in production. At the moment with the mentioned repository and two running fluxcd instance per environment both environments will be update at the same time when a new commit in master branch modifying files for basemanifests.
In the case of a git tag trigger support (taking your proposed --git-tag and adding support of wildcard) it could be improved by having for example:

  • Staging fluxcd instance with --git-path=staging and --git-branch=master
  • Production fluxcd instance with --git-path=production and --git-tag=v2.*

Thus having the staging environment tracking the tip of master branch and being updated at each pull-request merged to it and production limited to SemVer releases (tags) of a specific major.
In that example it would require to update the fluxcd instance configuration before deploying a new major in production. Which in a trunk-based development would be welcomed to be done via pull-request in the repository which configures fluxcd itself. It would allow as well to be more restrictive on the selection of the git tag if necessary by using an explicit tag (e.g --git-tag=v2.1.3).

Side point to your proposal with moving tags which does not change the validity of the proposal.
The idea of moving tags has the benefits of flexibility and not to have to change the configuration of the fluxcd instances like you mentioned. However its lacks proper workflow for review before triggering a deployment. To my knowledge there is no tool providing a review workflow when creating/writing tags. Thus it is not possible to "protect" or peer-review deployment into production which might wanted for some use cases. In the end this choice should be done by the user and they should be able to chose whether they want this flexibility or a reviewable workflow. This point could be described as part of documentation.

Other relevant discussion:
#1071

Reference to code that only looks for refs/heads:

ok, err := refExists(ctx, dir, "refs/heads/"+r.branch)

Some other tool have one parameter for triggering on git tags/branches with some fallback mechanism to switch between refs/heads and refs/tags (see: https://jenkins.io/doc/pipeline/steps/atlassian-bitbucket-server-integration/). This is cumbersome and the approach of two flags per type of git references seems cleaner or have common one only accepting explicit git references.

@antonosmond

This comment has been minimized.

Copy link
Contributor Author

@antonosmond antonosmond commented Nov 5, 2019

@Perdjesk I totally agree with you in terms of a poor workflow when moving tags and a lack of peer-review. Essentially moving a tag is a force push which I would strongly advise anyone against! But like you say, it's down to the end user how they use any functionality we're proposing and that's not really the point here. It's more about whether the ability to watch git tags would be useful to some people, regardless of their deployment strategy. It sounds like it'd be helpful for any situation where you want to target multiple versions/points-in-time on a single branch which I think is what we're both talking about.
That being said, I guess it'd make sense to support any ref, not just tags?

@linuxbsdfreak

This comment has been minimized.

Copy link

@linuxbsdfreak linuxbsdfreak commented Nov 7, 2019

@antonosmond . This is what i would also wish. However you had mentioned

User starts up Flux providing the --git-tag flag

Would it not be better that if flux via a K8s custom resource provide a tag release feature. Which can be manager via Gitops. The startup of flux providing a flag sounds very manual everytime a tag has to be rolled out to a K8s cluster.

@antonosmond

This comment has been minimized.

Copy link
Contributor Author

@antonosmond antonosmond commented Nov 7, 2019

@linuxbsdfreak in my use case I was thinking about moving tags which I appreciate is not a great approach but it'd work for my use case. Given I had the following commits/tags on a branch:

SHA      TAGS
5b722484 v2.0.0
b5798f18 v1.1.1
968c8164 v1.1.0,flux
a87b7edb v1.0.0

Rather than telling flux to watch v1.1.0, I'd tell flux to watch the flux tag for example, and then if I wanted to upgrade an app on my cluster to v2.0.0, I'd move the flux tag to align it with the v2.0.0 tag. In this example, I wouldn't need to reconfigure flux when I want to upgrade my release. Instead I'd just move the flux tag along.

@linuxbsdfreak

This comment has been minimized.

Copy link

@linuxbsdfreak linuxbsdfreak commented Nov 7, 2019

@antonosmond . Sounds reasonable. How would it work to promote the tags to different cluster envs? At the moment i have configured flux to pull its config from a Branch. Give the below scenario you mentioned.

c17f64b4: v1.2.0, flux-cluster-a
73f2c51d: v1.1.3, flux-cluster-b
7cea7fbd: v1.1.2
@antonosmond

This comment has been minimized.

Copy link
Contributor Author

@antonosmond antonosmond commented Nov 7, 2019

@linuxbsdfreak we actually run a flux per env on a given cluster so we have we have a flux namespace with a bunch of flux deployments e.g.
flux-dev
flux-test
flux-staging
That means we can have tags per cluster/env e.g.

c17f64b4: v1.2.0, flux-cluster-a-dev, flux-cluster-b-dev
73f2c51d: v1.1.3, flux-cluster-a-test, flux-cluster-a-staging
7cea7fbd: v1.1.2, flux-cluster-b-test,
48e20af6 v1.0.0, flux-cluster-b-staging

We'd handle moving the tags in our CI process.

@linuxbsdfreak

This comment has been minimized.

Copy link

@linuxbsdfreak linuxbsdfreak commented Nov 8, 2019

@antonosmond . Thx for the info. My setup as a follows

1 cluster per env which is mapped to a git branch.

for eg. K8s clusters (poc, dev, stg) and flux is configured to pull for the relevant branches in git. It would be be nice to have flux handle the scenario mentioned above along with what i have for doing a tagged release along with a rollback scenario.

@AustinSmart

This comment has been minimized.

Copy link

@AustinSmart AustinSmart commented Dec 3, 2019

@antonosmond This would be great to have. I have the same use case as you and I think your solution would solve things nicely.

@casibbald

This comment has been minimized.

Copy link

@casibbald casibbald commented Dec 16, 2019

@antonosmond I really would like to see this functionality implemented.
So +1 from me

@marsavela

This comment has been minimized.

Copy link

@marsavela marsavela commented Jan 3, 2020

Even though following a tag is something I could find useful, I'd like Flux to be able to detect when a tag has been made and same for releases. In this way I could have:

Dev cluster: latest and greatest.
Stage cluster: updated when I create a new tag.
Prod cluster: updated when I create a new release.

Is any of this already possible? I guess this is also manageable with branches though.

@2opremio

This comment has been minimized.

Copy link
Collaborator

@2opremio 2opremio commented Jan 10, 2020

This means we can't configure flux to watch the master branch of the my-app repo as HEAD may not be the version we want. Instead we have to watch the respective release branches. The problem here is that to upgrade from one version to another it requires re-configuring flux i.e. changing the branch that it watches.

@antonosmond would it be enough to:

  1. Decouple Flux's configuration from Flux itself using a e.g. ConfigMap
  2. Have Flux automatically reload when the configuration changes

(1) is already in progress (see #2544 , Christmas vacation stalled it though)

You would still need to update Flux's configuration though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.