support triggering via webhook #1128
Comments
We have a prototype of this in weave cloud, and the hooks are there in the daemon (i.e., it's just a matter of some engineering in weave cloud keeping it back). We could also have these in the daemon. It's a bit trickier, because
|
Great news - I guess it can be a bit nicer in weave cloud (UI etc). It would make sense to have in the daemon for users running flux only / on-prem / no weave cloud. Since the git setup is really in the daemon the format should not matter much if it is just a signal to pull the configured repo (e.g. no need to parse payload). |
+1 It would be most elegant to create a dashboard service (with a nice UI showing all flux releases), which would be accessible to these app generated webhooks. That service would then publish those events to the flux instances, triggering them to do their thing. Maybe that is possible with hosted Flux, but I am not aware of the paid offering's options... |
Has it already in available open source version? |
This feels like a real problem to support clear pipelines for CI/CD workflows. Our dev team wants to know their deployment is activated as they would then execute secondary flows such as cache clears, routing behaviour or smoke/QA testing against the deployment. Without having an explicit hook (and ideally a blocking wait) we can't use Flux for any deployment on our systems as the async delay reduces our options for automated hands-off continuous delivery and notification. |
This is less to do with webhooks and more with providing a layer of interpretation on the Kubernetes API. But I do grasp the requirement: to be able to spin up tests, you need to know when the system under test is ready.
What would the wait wait for? To put my cards on the table: I think it's possible to give a practical answer, or a general answer, and very difficult to give a practical, general answer :-) |
True, but isn't that basically what Flux is for? Providing the bridge between deployment state and deployment intent?
Maybe something like
Isn't this covered by the existing features in the stack; |
The interface should also include freezing the deployed configuration once it is consistent with a given commit. The freeze is necessary to ensure that tests can run without a change to the environment.
|
I fear feature creep away from the original scope. I understand the desire to avoid a potential several-minute lag between a new commit/image becoming available and the normal Flux poll cycle discovering it. Not by providing a new behaviour, but by jumping the clock and polling now. Which barely requires auth, as no directed change is possible (eg specific image/commit, locking), and no information is returned (eg whether an image was found/deployed). Those are already exposed in the API |
For sure. Perhaps @warmfusion, @derrickburns, we could move this discussion to the mailing list. It seems like there's a bunch of related requirements and desiderata beyond simple webhooks, that need some curation. |
I'm also very interested by the feature, do you have any plan to implement this in the daemon? Is it an option to expose an API endpoint like /api/v1/git-sync and protect it by using Basic, JWT, Oauth2.0... |
I would like authentication (along with only exposing that one /route) to be the providence of an Ingress controller in front of Flux, with Flux documentation recommending an appropriate config. Rate Limiting though... I suspect we really only want a maximum of one in-flight sync, rather than a request rate limit. If a sync is in-flight and one or more sync webhooks arrive, they should be batched into a single needs-another-sync-soon state that can be consumed/cleared. Is that equal to a golang buffered channel of length 1? |
Yes, that and an non-blocking put; and in fact that's what fluxd does: https://github.com/weaveworks/flux/blob/master/daemon/loop.go#L143 |
Hello, @squaremo pointed me to #2211 and I wanted to confirm with the people on the thread if the If this is not the case do you have an ETA when the functionality requested with this issue will be available? I don't see it included in any of the available milestones at the moment. Is there design guidance we can follow to try to implement it? Thank you! |
The
But the client can't be trusted to specify the git repo or image name, and an Ingress controller or reverse-proxy is not in a position to iterate through various urls/images. There doesn't appear to be an method to check all configured repos/images here. The older
Neither of these are pleasant for use as-is as a webhook interface. |
As per the discussion in fluxcd#1128, there are currently no convenient API methods that could be used as a webhook interface. For the simple need of letting Flux know that there are changes to be synced, this API method can be used. It requires no arguments, only triggers a Repo.Notify() call, same as /api/flux/v11/notify does if the supplied payload checks out.
Since I was looking for this functionality and couldn't find it, I took a look at how the above mentioned API methods work and added one that syncs the git repo without requiring any arguments. I'm not sure how it may fit in the larger scheme of Flux and what the maintainers have planned, so if this isn't something they'd like to merge, I hope that at least someone looking for this functionality might find it helpful. With the above API in place, all you really need to do is add an Ingress that exposes it and preferably protect it in some way, at least with basic auth:
Now you can call whatever address your Ingress exposes on /flux/sync-git, spcifying the basic auth as user:pass@ in front of the address and it should work :) |
That's the same approach I had in mind, though I was really trying to use an existing API. I didn't find one and got distracted by other work. |
Me too, but unfortunately I didn't find one either. Though at least for the purposes of notifying Flux about new Docker images, I think I will be able to use |
It was never the intention that an ingress or reverse proxy would call |
I was actually thinking about doing that, but I wasn't sure how to get started, since I'm not very familiar with writing stuff in Go. So just making that "dumb" method, which could theoretically only do very little harm was the way I chose. It does sound interesting though and I would like to try help with it if that's something you're considering adding. Edit: another thing is though, why the need to specify a repo in the first place? Flux can't watch multiple repos, can it? So I can really only see the usefulness of a "webhook client" for the image updates. |
No, but conceivably the service invoking the webhook might only let you install a webhook for several git repos (e.g., for a team or account). There is already a similar limitation with github: you can install a webhook for pushes to a particular repo, but you don't get to choose the refs you care about. That's why we pass the git repo and ref to the daemon in ../notify, so it can choose what to pay attention to; otherwise it would be constantly pinging the git service for updates, only to learn nothing has changed. |
I've said elsewhere, and it's worth repeating here: I think webhook receivers would make a nice contribution to fluxcloud*. It's not that I want to disown or deflect the premise of this issue -- my reasoning is that 1. because public endpoints ought to be separate from the flux API, they should be in another component, and 2. fluxcloud is another component that already takes the role of "adapting flux to the outside world". (*NB I have not verified that the fluxcloud maintainers share my view) |
Please have a look at https://github.com/fluxcd/flux-recv and see if it will serve your purpose. That is built specifically to receive webhooks and pass them on to fluxd. Currently it understands GitHub, GItLab and Docker; it is a fairly simple matter to add support for another webhook source. |
I honestly haven't reviewed all the details of this thread, but I think both sides of this request are covered well in Flux v2:
https://toolkit.fluxcd.io/guides/webhook-receivers/
Thanks for contributing something to Flux through your discussion. Flux v1 is in maintenance mode now, and is not adding any new features unless they are critical. As Flux contrib efforts have been focused on Flux v2, the Flux project has moved to a new repo, fluxcd/flux2 In the interest of reducing the number of open issues not directly related to supporting Flux v1 in maintenance mode, and respecting you may have moved on already, I will go ahead and close out this issue for now. |
Having to wait for flux to detect changes in the git repo is a bit of a drag / delay. It would be nice if flux could support being called via a webhook to trigger immediate refresh of the repo.
The text was updated successfully, but these errors were encountered: