Repository navigation
Description
Proposed Feature
There are use cases of promotions where the act of promotion may include copying files or directories from one path to another. Additionally, or alternatively, the promotion may need to modify/patch files in a certain way as part of the promotion.
Conceptually, this is no different than what already happens when Kargo promotes container images on the main branch using kustomize, at different paths:
# promote to dev
$ cd env/dev && kustomize edit set image acme/guestbook:v2.0.0
# promote to staging
$ cd env/staging && kustomize edit set image acme/guestbook:v2.0.0
# promote to prod
$ cd env/prod && kustomize edit set image acme/guestbook:v2.0.0
For each promotion, you could describe what Kargo is doing as promoting a "patch" to kustomization.yaml files from path to path, where the patch is upserting an entry in the images: stanza:
images:
- name: acme/guestbook
newTag: v2.0.0This proposal is to generalize the ability to modify files into something that is not kustomize/kubernetes specific, and could apply to anything configuration-related:
- json
- yaml
- hcl
Motivation
Users of Kargo shouldn't be limited to, or beholden to the choice of tooling Kargo natively decided to support. This feature will expand the scope and capabilities of what Kargo is able to promote, such as Terraform.
Suggested Implementation
Largely TBD but I imagine we'll need ways to describe:
- filepaths
- exclusion filters
- extraction/parsing logic
- ways to manipulate a configuration file (e.g. yq, jq, sed)
Open Issues
When we move to support patch-based promotion, I think Kargo has to start treating "rollback" as something different than "promotion". Because while we may easily know how to apply a patch as part of the promotion, we would not easily know how to "undo" that patch for a rollback.
Ultimately, I think we may need to require the use of rendered branches to support rollback, and then rollback becomes nothing more than re-applying the contents of an older commit in that branch as the tip.
Activity
jessesuen commentedon Dec 11, 2023
I forgot about one use case that we even practice internally. I would like a way for kargo to monitor git-repo-1/path for changes, and then whenever changes are made in that path, copy/replace the directory contents of git-repo-1/path to git-repo-2/path.
We constantly make changes/adjustments to helm chart contents in an upstream git repo. Let's say:
https://github.com/akuity/kargo/tree/main/charts/kargo
We infrequently release the helm chart (once every few weeks), but would still like a way to immediately deploy the changes of that helm chart before any release is made (whenever changes are made to
chart/kargodirectory in git).You could think of this use case as similar to constantly building and pushing a container image at the tip of
main, but instead of a container image, it's the contents of a file path in git.jmgilman commentedon Feb 23, 2024
Apologies if I am misunderstanding the proposal, but looking for a goldilocks solution seems infeasible given the wide range of activities that must be accounted for in the act of promotion. Wouldn't it be better to implement some contractual API akin to Argo's Custom Management Plugin interface? If I recall correctly, that was born out of trying to avoid people having to hack the repo server with all sorts of workarounds due to Argo's limited support for anything other than Helm/Kustomize.
We don't currently use Kargo (due to this exact limitation), but from my conceptual understanding, the "before" and "after" that the CMP primarily revolves around isn't much different here.
krancour commentedon Feb 23, 2024
@jmgilman this and promotion mechanism plugins aren't mutually exclusive.
Plugins are definitely happening. The output of this issue could end up being a plugin.
jmgilman commentedon Feb 23, 2024
Gotcha, thanks for clarifying. Is there an issue tracking plugins?
krancour commentedon Feb 23, 2024
@jmgilman oddly there isn't. It's an epic anyway. But it is on the roadmap with plans for us to be designing during the current release cycle and implementing in the next.
20 remaining items
jessesuen commentedon Oct 7, 2024
Let's leave this open until we get a better handle/feedback if copy can fulfill the use case. But until then, we don't have to schedule this and can leave this open without milestone.
Chewie commentedon Oct 29, 2024
A use case where a generic patch via something equivalent to yq would be useful, and that copy doesn't cover, is a scenario in which the organization uses a CRD to represent the desired state (e.g OAM, knative, or a bespoke one created with crossplane), without helm or kustomize.
In this scenario, my gitops repo is just one document per stage, and I need to update the image reference in that CRD during the promotion step.
danielloader commentedon Oct 29, 2024
Exactly this, for me I just want to update a YAML config that launched images are derived from - not too dissimilar to knative.
https://www.shinyproxy.io/documentation/configuration/#overview
.specs[].container-imagein this scenariokrancour commentedon Oct 29, 2024
@Chewie @danielloader I think what the two of you need is simpler than what @jessesuen was going for with this issue. You just need a promotion step that can perform arbitrary updates on a YAML document. That could be added with very little effort today, although its usefulness would be somewhat limited to changing an arbitrary key in an arbitrary YAML file to some value derived from your Freight. Once the EL support we're working on for this release is complete, this would be far more flexible. If there isn't already an issue open for adding such a step in conjunction with the EL, I will open one, as it would actually be a great way to show off what sort of things can be accomplished with the EL.
jeffmccune commentedon Nov 24, 2024
@krancour
Could you help me find this issue? I'm looking into this to integrate Holos with Kargo as described in #2995
I found this from #527 which is also about integrating with CUE. Editing a yaml or json file with CEL would accomplish part of the integration, but I'm still looking for a way to run
holos render platformas a pipeline step to fully render the manifests.krancour commentedon Nov 24, 2024
It's been fully addressed in #2941
jmgilman commentedon Nov 25, 2024
So, just to be clear, we only have support for Kustomize, Helm, and plain YAML? Are we still missing tying into the rest of the ecosystem? All of my teams use intermediate tooling for generating the final YAML (CUE, KCL, etc.). It still looks like Kargo is incapable of being integrated into more complex deployments.
krancour commentedon Nov 25, 2024
@jmgilman we do not plan on directly integrating with tool after tool. We plan on enabling development of custom/third-party promotion steps to allow these integrations to be developed outside of Kargo proper.
Discussion in this thread has gotten a bit sidetracked. Everyone, please let's keep this issue about really low-level, generic sort of things and please open other issues for things that aren't as low-level as copying a file or performing an arbitrary update on a YAML or JSON file.
jmgilman commentedon Nov 25, 2024
@krancour I'm just really lost at where this feature is being tracked. I understand there's not going to be white-glove support for every tool out there, but there's been existing discussions around implementing plugins akin to Argo CD to be able to support these generators.
This issue briefly talked about it but was closed as a duplicate of the current issue. I also asked about this earlier in the thread but the answer really wasn't clear to me.
Is this plugin feature being tracked in this issue? Is there another one, yet? I'm just trying to have a place to track this plugin feature because our team would like to adopt Kargo when it can support our more advanced generation pipelines.
jeffmccune commentedon Nov 25, 2024
@krancour
To your point, I think @jmgilman and I need two things. First, perform an arbitrary update on a YAML or JSON file. The subject of this issue. Then, we need to execute a command other than what is built into Kargo to render manifests. Some other issue.
Is there an issue or discussion we can subscribe to and participate in regarding that second step, running the command after updating the YAML or JSON file? I opened up #2995 to try and find where the feature was being discussed as well.
krancour commentedon Nov 25, 2024
It's been on the roadmap for some time, but did not have an issue. I just opened #2996.
jessesuen commentedon Feb 28, 2025
Since filing this issue, we now have the ability to clone different branches of the same or even different git repo (e.g. a source commit, and to a destination branch). We also have
copy,yaml-parse,json-parse,yaml-updateandjson-updatesteps. I think I can achieve all the use cases I mentioned using these constructs. See the following example:https://github.com/akuity/kargo-examples/tree/main/03-features/03-yaml-parse-update
Additional work/use cases can be accomplished with newly proposed steps (e.g. hcl-update)
Closing as completed
jmgilman commentedon Mar 1, 2025
To be clear, I need to parse and promote CUE, which is why I was hoping for a language-agnostic way to handle promotions. Unfortunately, none of the proposed solutions work in my case.