Interactions between the Git workflow and Builds actions

[FALLAI-Denis]

Hi,

This discussion was initiated following the discussion started on request #224, which was unrelated to the original request.

I tried to illustrate my vision of the organization necessary for the Builds and the relations they have with the Git workflow (in my case a workflow close to Gitflow).

Feature Management Timeline

The developer creates a feature branch from the develop branch.
He carries out his development and can trigger User Builds throughout his work.
These User Builds use a feature specific context, and in particular use debug compilation options, and COPYBOOKs through a concatenation of feature / develop / release / master levels, (the COPYBOOKs libraries only contain differences from the promotion path of Git branches ).

When its development is finished and tested, it requests a merge in the develop branch, (it does not process this merge itself).

At the appropriate time, the Release Manager triggers the merge of the feature branch into the develop branch.
This action triggers a Project Build through Jenkins mechanisms.
This Project Build use a specific context, and in particular use COPYBOOKs through a concatenation of develop / release / master levels.
Integration tests are performed with this Project Build.

The release manager creates or enriches a release branch.
These actions (create and successive merge in release branch) trigger a Project Build through Jenkins mechanisms.
These Project Builds use a specific context, and in particular use COPYBOOKs through a concatenation of release / master levels.
Release candidate tests are performed with this Project Build.

When the release is complete and tested, the release branch is merged into the master branch (and also into the develop branch to take into account any corrections made to the release).
This action trigger a Project Build through Jenkins mechanisms.
This Project Build use a specific context, and in particular use production compilation options, and COPYBOOKs at master level only.
The result of the Project Build is deposited in the Artifactory artifacts manager as a Package to be deployed.
This Package is used to successively feed the business acceptance, pre-production and finally production environments.
The package is also used to upgrade other environments used for Hotfix management.

There may be a significant delay on the one hand between the moment of the merger of the master branch which triggers the project construction and the creation of the package for Artifactory, and on the other hand the actual deployment in production, which complicates the process quick fix described below, because you need to manage patches for the business acceptance / pre-production / production environments.

Hotfix Management Timeline

When an incident is detected on components that have been made available for business acceptance / pre-production / production (built during the merge to the master branch), a rapid correction must be initiated.

The developer creates a hotfix branch from the master branch, not necessarily on the last commit of the master branch, , which is a problem because we don't necessarily have the build environment in the exact image of the environment in which the incident was identified...
He carries out his development and can trigger User Builds throughout his work.
These User Builds use a hotfix specific context, and in particular use debug compilation options, and COPYBOOKs through a concatenation of hotfix / master levels, (the COPYBOOKs libraries only contain differences from the promotion path of Git branches ).

When its development is finished and tested, he pushes his hotfix branch to the Git Repositories server.
This action trigger a Project Build through Jenkins mechanisms.
This Project Build use a specific context, and in particular use production compilation options, and COPYBOOKs at hotfix and master levels.
The result of the Project Build is deposited in the Artifactory artifacts manager as a Package to be deployed.
Note that this Artifactory package is not built from the master branch, because the master branch may be ahead of the environment in which the incident was identified.
This Package is used to feed the production environment.
The package is also used to upgrade other environments used for Feature management.

Additional actions can be carried out if necessary:

• report of the correction on the master (without construction, already packaged) and develop (without construction) branches if the correction is permanent, application of the Artifactory Package to all environments at develop and release level, business recipe, ... (enslavement of environments)
• otherwise creation of a feature branch to reintegrate the correction into the release management workflow, as described above.

Problems identified

At the level of User Build, we see that the context between a feature User Build and a hotfix User Build is not the same, in particular the same COPYBOOK libraries are not used. However, the configuration of the user build process by IBM DBB is unique at the level of the Workspace / Git Repository, just as the configuration of COPYBOOKs searches through zapp.yaml are unique, (configuration files are shared and merged between Git branches, like any other element of the Workspace / Git Repository).

Between a User Build at the local hotfix branch level, and a Project Build at the server hotfix branch level, we don't use the same context either: the User Build uses debug compilation options, while the Project Build uses production compilation options, and this Project Build should feed into the Artifactory package manager.

The project builds of the different levels do not use the same contexts of PDS libraries and compilation options either.

We must also find a mechanism for moving COPYBOOKs from one level to another.
The COPYBOOKs libraries cannot be full (cannot contain all the COPYBOOKs), they should be managed in differential mode and contain only level-specific COPYBOOKs versions, because of the double path between "feature" and "hotfix" and the sharing at "master" level which must be available for both paths. When a COPYBOOK is referenced at master level by the "feature" path, it must be seen from the "hotfix" path, and vice versa.

We also use different sets of COPYBOOK libraries to manage exposure levels between Applications:

• "Public" COPYBOOKs: usable by all Applications
• "Protected" COPYBOOKs: usable by the Applications defined in the same Domain or Product
• "Private" COPYBOOKs: can only be used by the proprietary Application.

In summary, each build is dependent on its Git branch context:

• for the choice of compilation options (debug or production)
• for the use of COPYBOOKs (versions of COPYBOOKs)

We also have a problem building patches after the build and packaging triggered when merging the release branch into the master branch, due to the fact that multiple environments exist in the test and recipe circuit downstream of this build/packaging, and the significant delay between building the package and its actual deployment in production.