Orchestration Pipeline: Merge released code back into main branch

[michaelsauter]

Follow up from #364:

Upon successful deploy to prod merge release/ version branch back to Master (or what was configured in 'metadata.yml' of rm - this is the new starting point for a new Change imho.
I guess the best time to do - is before we go to Q - Q (and P) are holy, to if the merge does not succeed we can break early.

I think we are actually with two distinct issues here. For the sake of simplicity, we assume that the configured branch is master.

1. If a developer makes a change only in a release branch, the change would not be included in the next release branch unless explicitly merged to master before that next release branch is created. Only in rare cases this actually is desired: if the changed code only applies to code in the release branch, but does not apply to master. In general, we want the orchestration pipeline to integrate the changes in the release branch back into master. This should probably happen in assemble mode in D, for the reason that any code change on a release branch would need to be assembled again in D. Between Q and P, no code change is allowed. Note that merging the release branch into master might fail when there are merge conflicts. In that case, the user must solve them manually, push the merge commit to master, and then start assembly again.

2. Code that was on production should not be "lost" the next time we go to production. This should be covered by (1), but it would be good to verify that the release branch contains all code currently on Q/P when deploying a new release branch to Q/P. The check is crucial as one might need to integrate changes from one release branch to another - just integrating into master doesn't cut it. You might have a hot fix in a currently released branch when someone else already created a new release branch from master.

A couple of open topics:

• Right now, the orchestration pipeline does not "know" what is in Q/P, making (2) hard to implement. The closest we've got is going through the tags, but it's not enough as we don't know which version was the one prior to the one we are deploying at the moment. Therefore, the release manager needs to keep track of what has been deployed, so that subsequent deploys can look up this information.
• If there are commits done in the repos during deployment to Q and P, then we also need to re-integrate when deploying to Q and P ...
• When would re-integrate be done best? If we do it at the start of the pipeline, it is "safe" but we might block deploying. If we do it at the end of the pipeline it is "unsafe" but we don't block. If we only needed to re-integrate once (assembly in D) the issue would be smaller.

FYI @martsec @clemensutschig @metmajer Any comments? Ideas?

[clemensutschig]

Hey @michaelsauter - thanks for writing it down

Therefore, the release manager needs to keep track of what has been deployed, so that subsequent deploys can look up this information.

yes! (a map with change & commit & env would be great) ... as we commit and export on Q/P

If we do it at the end of the pipeline it is "unsafe" but we don't block. If we only needed to re-integrate once (assembly in D) the issue would be smaller.

On assembly .. non WIP

[martsec]

Hi @michaelsauter @clemensutschig , the issue gets even more complicated with the generation of the LeVA docs and the requirement that we might regenerate an already created document (e.g. SSDS or CSD) if we had to do a hot fix. Materializing some of the information in the release manager repo for example, is something that we might need to do already when enabling versioning.

What we are seeing implementing the feature is that if we have a change (e.g v. 2) that was already deployed into Q, but we need to do a Hotfix (v. 1.0.1) to the productive system, we would need to force the "restart" of the release 2.0, adding the hotfix changes to avoid adding again the bug. Restarting the release for the v2.0, we would have the changes from PROD and generate consistent documents.

IMO we should do the check at the start of the pipeline, providing fast feedback to the users. If the merging fails, we can create a bug in JIRA and fail the pipeline so the user knows that he needs to take action.

Keep in mind that restarting the release (something that we already support, at least via the Jira plugin) will mean that the users do a "Developer Preview/WIP" run before releasing officially into dev, allowing them to fail fast, and in the official release process.

We are currently discussing internally options that could provide more flexibility, but as far as I have seen, there might be no other option to restart the release process from the begging to avoid putting the applications into an inconsistent state.

FYI @metmajer

[michaelsauter]

@martsec I am not sure I fully understand your point.

the requirement that we might regenerate an already created document (e.g. SSDS or CSD) if we had to do a hot fix

A hot fix would be done on a release branch, so you'd just build the same technical version again, but using a difference change ID. In order to get the hot fix into PROD, you first need to assemble in DEV, then promote to QA, then to PROD. Which means you anyway generate new documents, right? To me it looks like this "feature" is built-in already?

we would need to force the "restart" of the release 2.0, adding the hotfix changes to avoid adding again the bug

That would be detected when going with 2.0 to PROD by my proposal with the check mentioned in (2). I don't see another way of "resetting". If 2.0 was successfully released to QA once, we cannot "undo" that - we can only prevent that the user can go with that outdated state to PROD. Right? One improvement might be that we tell the user somehow that a certain version has become stale e.g. by creating an issue in Jira how you mention.

So I am not sure if you agree with my proposal or think that it is not enough :) If the latter is true - what case am I missing?

[martsec]

Which means you anyway generate new documents, right? To me it looks like this "feature" is built-in already?

The issue here is that some documents are already signed in the document management system and thus, simply replacing them won't suffice. So when you have a release in QA, some of the documents you have generated are "productive" and valid. And you need to start with them for a given document (they need to be self-contained, including its history)
In short, the history of a document might not be the same as the history of the code/product.

I'm working on a way to explain our concerns more visually and clearly, so we can also use it to explain the solution.

That would be detected when going with 2.0 to PROD by my proposal with the check mentioned in (2).

Yes, we should check that we have the same changes. What I see though (and maybe it is the same you are saying), is that we should:

1. Block the release in progress
2. Merge the latest production code into the 2.0 branch
3. If the merge fails, tell the user that he needs to resolve the conflicts manually (creating a PR and a bug in JIRA maybe)
4. Start the release process into DEV again for the "new 2.0", then QA and PROD

Don't get me wrong, I like your proposal, but recently I have discovered the limitations we have with the documentation, that can make a couple of things change. Today I explored more this topic with the main stakeholders on the compliance part, and I plan to work on having a good solution that can meet the requirements for all parts.

[michaelsauter]

@martsec Cool. Looking forward to the visual explanation, that will help. I think regarding the merging, we are saying the same thing :)

[martsec]

Now that I think of it, this might be thought as a race condition to PROD between the documents and the software (code) of multiple versions.

[clemensutschig]

@martsec any drawing yet.. we want to build this.

[martsec]

I wanted to double check with the different stakeholders before sharing it.

The situation:

• We have a new version (2.0) of our application that we go through production. The Jira issues and the code use the current productive version (1.0) as a basis. We deploy it to QA. Our state will be
  • Valid (signed) documents: 2.0
  • Code in PROD: 1.0
  • Code in Q: 2.0
• Now we realize that we must fix something in production, or there is another change that has more importance and must happen first. Let's call it 1.1
• However, we must base our code on 1.0 but for most of our documents we must base them on the 2.0.
• Since in this 1.1 we do not want to bring the changes of 2.0, we state in the documents that those changes will not be valid for 1.1. We release 1.1
  . Our state will be
  • Valid documents: 1.1
  • Code in PROD: 1.1
  • Code in Q: 1.1
• Now, for the 2.0 change, we want to rebase/"restart" the code and the documents on the latest productive state. This means merging the 1.1 code back into the main branch and to the release one. And use the 1.1 documents as a base. Adding the 2.0 requirements/tests... on top of it.
• Proceed with the release (let's say 2.0-b) to prod as usual.

Why this happens:

• We have two states: one for documents and one for the code running
• We need to maintain document history consistent once they are signed
• Documents and code are not promoted to PROD/valid in a "single transaction"
• We face a race condition with another release

The diagram tries to illustrate the situation. Time goes down and it is a simplification that does not cover DEV and QA environments nor the status of the master branch. The self relations indicate the status of a given element.

Another way to represent it is thinking about the links to the previous versions:

• Standard change (2.0) start the process
  • History for Code and Version in Jira: 1.0 <-- 2.0 (up to QA)
  • Document History: 1.0 <-- 2.0
• Emergency/concurrent change
  • History for Code and Version in Jira: 1.0 <-- 1.1
  • Document History: 1.0 <-- 2.0 <-- 1.1
• "Restart" of the 2.0 version (2.0-b)
  • History for Code and Version in Jira: 1.0 <-- 1.1 <-- 2.0-b
  • Document History: 1.0 <-- 2.0 <-- 1.1 <-- 2.0-b

In my opinion @clemensutschig it is better if I can explain it in a 15-30 min call so I can explain thinks I might have missed in the explanation.

Anyway, I think the implementation of this is not really difficult, since we can manage the state of the documents that are in valid in the document tracking issue in JIRA.

[clemensutschig]

This is pretty similar what @metmajer and I came up :)

we have three topics to cover

1. we need? a "second" dev & qa for 1.1 - to NOT touch current 2.0 developments and transport only this to P then - that implies
2. once in P - we need to "rebase" the release status object for 2.0 on 1.1 instead of 1.0
3. we need to merge those changes into the version 2 branch(es)

I want to follow this up with QA today - so we have one aligned view .. - done .. they are happy!°