-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
KEP 70 - Add/remove Stages to/from Project #70
Conversation
Signed-off-by: Johannes <johannes.braeuer@dynatrace.com>
Signed-off-by: Johannes <johannes.braeuer@dynatrace.com>
An alternative approach is to allow the editing of stages via the shipyard. This gives the user all possibilities to express how the sequences in the stages are interlinked, which is not possible with |
Signed-off-by: Johannes <johannes.braeuer@dynatrace.com>
When discussing this KEP with the team, we determined a list of open questions which should be answered in this KEP:
|
@agrimmer, excellent questions and many thanks for bringing those up. I will answer them in the order of (1), (3), and (2): (1) What should happen with events of a deleted stage? Should they be deleted? For example, what should happen if I delete a stage and then create a new one with the same name? Do we want to preserve the events?
(3) When creating a new stage, do we want to reuse configs from other/parent stages? Or do we want an empty config?
(2) Do we need to change the used Git model into a folder-based structure in order to allow dynamically adding/removing stages? This is a rather challenging question that required some research. Links to recent blog posts can be found here:
In order to derive a decision here, @agrimmer, @thisthat, @markuslackner and I (@johannes-b ) have created a comparison of the branch-based and folder-based approachs:
Since this is a drastic change in the way Keptn is using the Git repository, further discussions will happen to do this change. Please post your thoughts and comments here. |
Basically, I like the approach of flattening the repository structure and eventually deprecating the upstream repo in the future. In my opinion, a major case is not handled at the moment which led to the development of the promotion service and which might be prevented when switching the repository structure. This consists of two parts, the storage of the artefacts and the retrieval of them when deploying them. Storing artefacts Retrieving artefacts What’s the problem with the new approach? As far as I could get it, you plan to retrieve the artefacts based on commit ids in the future, and this might be the main part which will break this approach. Some questions:
As I wrote in the introduction, I totally support your approach and I would appreciate every efforts to remove the need to write to the upstream repository from a “user” perspective and deprecate the promotion service. But currently, I think above’s cases are unhandled and might be avoided in the worst case (as I’m currently not really sure how artefacts will be fetched then). |
Hello @thschue, many thanks for the great feedback and the discussion we had! 🙌 In a meeting, Thomas and I draw a picture to iterate over the questions raised in his previous comment.
Please consider the derived drawing that is be explained below:
Conclusion and status quoTL;DR:
Action Items
Already known tickets: |
The behaviour would have an impact on #67 and the gitops-prototype itself as illustrated here: As the artefacts could be pushed in one batch, there would be no more need to manipulate the upstream repository. Therefore, It would be possible to use a generic GitOps toolkit (as the ArgoCD or flux controllers) for the manipulation of the keptn resources. The delivery of the artefacts could be done via the keptn CLI (push), or with a simple controller (pull), which fetches the configuration and delivers it to the keptn API. Currently, also the promotion service which is doing the version handling at the moment, could be removed completely if keptn itself is able to provide artifacts for a specific version of a service in a stage. The drawing also shows, that the keptn API gets the center of all operations again, and there is no need for a user to access the upstream anymore. |
I see this listed as a disadvantage of branch based but don't see a corresponding advantage in the folder column: |
Hey @agardnerIT, I would say that the folder-based approach definitely moves us closer to solving #79. |
Thank you for the write-up @johannes-b and @thschue. Could you please provide more details on what type of configuration you envision to be stored in the
Would this replace the |
Hey @agardnerIT thanks for highlighting: The reasoning behind that disadvantage: This limitation falls back to the problem that the branch-based approach requires a lock on the Git repo to derive the correct I agree with @thisthat that going with the folder-based approach brings us closer to supporting the use case mentioned here: #79 since we eliminated the risk of reading configuration from another sequence (which is the main motivation for queuing sequences right now). Hey @thisthat I agree that it needs clarification whether Keptn specific config is managed this way too. Currently, much focus is on deployment specific config. I added your thought to the above list of Action Items. Hence, we need to follow up on that one. |
Hello @thisthat! We mostly thought about the configurations held in files in the repository at the moment (like helm-charts, sli.yaml, slo.yaml, job.config). The Webhook Configuration is more a keptn configuration that should be held in a Custom Resource or managed via the bridge in the long term (but in the beginning, it can stay in the Artifact store). We also discussed the replacement of the shipyard, which should be the ultimate goal, in my opinion. But we also agreed that this should not happen within the scope of this enhancement proposal. We should keep the internal keptn structure like Projects, Stages, Sequences in the Database (or as CRDs in Kubernetes in the Future). I see no advantage in getting the project structure from the Artifact Store (aka Upstream Repository), and I also think this change could bring us a lot of flexibility. For example, our new structure would allow the definition of simple configurations in all stages (base directory), and a customer can override specific configurations (like SLIs) per stage. If we add a new stage and there are no overrides, it's possible to deploy this service without adding any Artifacts per stage (but yes, the stage needs to be created). Adding and removing stages would be very easy in this case, as we would not have to take care of the Artifact store. The database would be the source of truth, and the Artifact store could contain specific information for this stage, but it wouldn't have to. Finally, the Artifact Store (whether we're using git/s3/local filesystem as backend) should only contain artifacts that the other keptn services might consume. It should be possible to configure Keptn via Custom Resources in the GitOps/Declarative approach or the CLI/Bridge in an imperative way (or applying the custom resources via the cli). |
In the developer meeting on the 21th of April, we agreed on splitting this KEP since it already covers three aspects:
I will proceed with moving (2) and (3) into a dedicated KEP. For the sake of comprehensibility, the started discussions for (2) and (3) will stay in this KEP.
|
This issue is mainly obsolete with the Lifecycle Controller, and I am no more really convinced that it makes sense to add this functionality to Keptn itself. Therefore, I propose to close this enhancement proposal (won't do). @johannes-b, @thisthat: What do you think about this? |
You could argue that this is still a missing capability. However, the Keptn lifecycle-toolkit is more flexible in this regard, and investments there are more future-proven. I'm fine closing this KEP. |
I think the Keptn Lifecycle Toolkit (KLT) is a better place for this enhancement since it targets the delivery of applications. I am also in favor of closing it with |
Closed in favor of the Keptn Lifecycle Toolkit (KLT). |
Add/remove Stages to/from Project
Success Criteria: Keptn supports adding and removing stages for existing projects.
Short abstract
In Keptn, a project stage (or just stage) defines a logical space, which has a dedicated purpose for an application in a continuous delivery process. Typically, a project has multiple stages that are ordered depending on the maturity level of the application. For instance, a project can consist of a
deployment
,hardening
, andproduction
stage whereas the maturity level grows from left to right. While adeployment
stage is used for feature development and initial testing, an application inproduction
is bullet-proofed and services production load.Due to modern Cloud-native development approaches, a static set of stages is not sufficient anymore. For instance, it should be possible to spin up a stage for chaos-testing an application and to remove it after the test was conducted. Another use case may focus on rolling out a multi/hybrid-Cloud strategy, which requires the need to add new stages for additional execution planes that are running on another Cloud provider.
Currently, it is not possible to add a new stage after the initial project setup. This KEP addresses this missing capability of adding/removing a stage on demand.
Depends on
Why
Target audience / Pain points / Related discussions
Pain points:
Use-case drivers:
Related discussions / PR
What
As mentioned above, stages are ordered from left to right creating a child-parent relationship. The following example shows a setup with four stages. While the stages
![image](https://user-images.githubusercontent.com/729071/152763695-1362fa90-dd41-4f33-93d6-f35954b1ab81.png)
dev
,hardening
, andproduction
are in sequential order, the stageproduction-remote
is parallel toproduction
and has thehardening
stage as parent.Currently, the linking of stages to create a child-parent relationship is based on sequences and their
triggeredOn
configuration. Based on the above example, a shipyard configures the relationship fromhardening
todev
as follows:To configure a child-parent-relationship between stages, both stages need to have the same sequence (
delivery
). Besides, the child stage has to have atriggeredOn
set on the event of the parent stage (dev.delivery.finished
).What it is?
⭐ Use Case: As a user, I can add a new stage to a project.
This use case comes in two ways:
chaos
in-betweendev
andhardening
for a dedicated testing purpose.quality-assurance
that is parallel to another one.Outcome:
![image](https://user-images.githubusercontent.com/729071/154692956-020cb941-c5d6-4192-91e0-e59312eaf4cc.png)
Both ways of adding a stage are supported by adjusting the shipyard of that project accordingly:
keptn upgrade project
or API call on PUT/project/{sockshop}
adds the new stages:⭐ Use Case: As a user, I can remove a stage from a project.
keptn upgrade project
or API call on PUT/project/{sockshop}
removes the stagesWhat it is? Summary:
/project/
endpoint allowing a user to add/remove a stage by updating the project.keptn update project sockshop --shipyard=new_shipyard.yaml
What it is not?
/project/{project}/stage/{stage}
> Creates a stage for theproject./project/{project}/stage/{stage}
> Deletes stage from project and updates the shipyard accordingly.Open Discussions