Implementation of a solution compatible with IBM Z Open Editor for the management of compilation options and the search for COPY/INCLUDE

[FALLAI-Denis]

Hi,

IBM Dependency Based Build, or more precisely the zAppBuild component, uses its own mechanisms to manage compilation options and manage the search for COPY/INCLUDE at build time.

IBM Z Open Editor, or more precisely its Language Server components, uses its own mechanisms to manage the compilation options and to manage the search for COPY/INCLUDE when editing the source code.

We want a unified solution that is compatible with IBM Dependency Based Build and IBM Z Open Editor, and also open to customization for the customer site needs of these two IBM-sourced solutions.

IBM/zopeneditor-about#280

[FALLAI-Denis]

Hi,

Resolved by implementing a proprietary solution.

[dennis-behm]

Hi @FALLAI-Denis, I am re-opening this issue, because I believe, that this is a important one. I am in communication with the zopeneditor and idz team to discuss how we can better align these search path configurations for edit and build.

I am curious about your proprietary solution. Have you modified zAppBuild to consider ZAPP?

[FALLAI-Denis]

Hi @dennis-behm

Your needs

• manage several production systems: the same source (COBOL program) can be built for several systems
• develop several versions in parallel
• maintain the version(s) delivered in production at the same time as developing new versions
• associate the right version of COPYBOOKs according to the development context (according the Git branch used for programs' Git Repository)
• use a similar COPYBOOK search mechanisms for IBM Z Open Editor and for IBM Dependecy Based Build
• our build management system must be compatible between CA Endevor and IBM DBB (but this does not concern the search for COPYBOOKs which is different between the two contexts): we are doing a gradual migration from CA Endevor to Git and IBM DBB

The products used

• Bitbucket for managing Git repositories (all technologies)
• Artifactory for the management of Component Packages (all technologies)
• IBM Z Open Editor for editing mainframe source codes
• Jenkins for the orchestration of constructions (all technologies)
• IBM Dependency Based Build for builds and for package creation of mainframe components
• XLR for deployment orchestration (all technologies)
• a proprietary solution to carry out mainframe deployments

Our Solution

• the COPYBOOKs are managed in dedicated Git repositories, separate from the programs' Git repositories
• each Application uses its own Repositories for programs and for COPYBOOKs (no shared Repositories)
• a ".build" file is associated with each program to be built: it lists the target systems to be supplied and the construction procedure to be used for the program (our build.groovy analyze the ".build" files)
• a ".build" file is associated with each Git branch of the Program Repository: it gives the construction context in relation to a target system (our build.groovy analyze the ".build" files)
• the Git Repositories of COPYBOOKs to be used are declared in each ".build" file of the Git branch: same Git Repositories of COPYBOOKs but not the same branches of Git Repositories according to the Git branch of the Program Repository
• Git repositories of COPYBOOKs are automatically (by a proprietary VS Code extension, or by the Jenkins pipeline) imported into an ".imports" folder (declared in .gitignore) of the programs Git repository and positioned on the needed branch: opening VS Code and at each branch change of the Git Repository of programs, and at the launch of a Jenkins pipeline which must do construction
• Z Open Editor's zapp.yaml file is configured to look for COPYBOOKs in the ".imports" folder (and other locations)
• the zAppbuild procedures discover on their own the imported Git repositories of COPYBOOKs (they are in the "scope" of the workspace): the impactBuild alone discovers the COPYBOOKs modified since the previous build and triggers the construction of the programs which use the modified COPYBOOKs, (tested but not yet seen how the impactBuild making this discovery)

Our operation for IBM DBB

• for builds, each Git branch of the Program Repositories lives independently of the others: we manage Packages by branch (release branch --> snapshot Package , develop branch --> release-candidate Package , master branch --> release Package)
• we only do impactBuilds, except for the first time case where we do a fullBuild by scan to create the first Package of the branch (we are in an Endevor migration process)
• a new Package (time-stamped) is created each time the branch is built, and is the result of the previous Package enriched by the impactBuild (it is also necessary to manage the deleted sources and remove them from the new Package)

To do

• use the IBM DBB call API integrated into IBM Z Open Editor: not directly but through a proprietary VS Code extension to hide from the developer the switch from Endevor to IBM DBB --> the developer will always use the same interface to trigger a User Build

---

Our solution is similar to managing Git submodules, but we were advised against using Git submodules.

So we haven't changed how IBM Z Open Editor works, but just configured the zapp.yaml file so that it searches for COPYBOOKs in the Git repositories import folder for COPYBOOKs.
There is also a search in PDS on z/OS which correspond to the products (Db2, CICS...) and to the COPYBOOKs of our development framework, which do not have to be versioned and therefore are not dependent on the branches Git from Program Repositories.

We also didn't change the way zAppbuild works for the COPYBOOKs search part and COPYBOOKs impact detection (which was a great relief for us not having to code it ourselves).

We have modified the way zAppbuild works so that it uses our ".build" files associated with the sources to build and the context of the Git branch used for the build:

• it is the presence of a ".build" file associated with the source which conditions the construction of the source (if it is absent, no construction is triggered, even if the source has been modified)
• in the same way if a ".build" file associated with a source is modified, it triggers the construction of the source even if this source has not been modified (change of the construction procedure for example)
• we can trigger several builds if a program targets several systems depending on the content of the ".build" file associated with the source.

We use the git sparse-checkout command to filter the contents of Git repositories of imported COPYBOOKs, to meet a software architecture rule: an Application A can only access the COPYBOOKs of an Application B according to a defined sharing level by Application B and which is based on a hierarchical Urbanism Plan.

[FALLAI-Denis]

Implementation samples

Git repository organization:

.build file associated to a source file: S9TST1.cbl program must be built for a target named SIRIS and an change code T00001.

Folder of .build files associated to each Git branch:

.build file associated to change code T00001 also feature/T00001-userid branch: build target informations and imported Git repositories containing copybooks

.build file associated to master branch: build target informations and imported Git repositories containing copybooks

zapp.yaml: search for copybooks in /.imports folder and all subfolders