CBS script enhancements to manage packages lifecycle and ability to fetch external dependencies #299

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

dennis-behm merged 127 commits into IBM:main from dennis-behm:feature/297-enhance-cbs-scripts-to-pull-configured-application-dependencies

Apr 7, 2025

Member

dennis-behm commented Jan 31, 2025 •

edited

Loading

This is extending the existing common backend scripts dbbBuild.sh and zBuilder.sh to automatically fetch external dependencies from an artifact repository (instead of any concatenation).

It updates the structure how tar files are store in the Artifact repository.

wazideploy-generate.sh script to enhanced to re-compute the url of the package, to be able to download the the package with the Wazi Deploy provided mechanics.

High-level capabilities for fetch capability:

Read build dependency configuration defined in the Application Descriptor.
Load external dependencies based on naming conventions of the CBS from an artifact repository (Nexus, Artifactory)
Implement a central cache of external packages
Expand the external dependencies in the build workspace for consumption by the build framework

Deployment:

Enable wazideploy-generate to automatically pull the tar from the Artifact repository

Additional changes:

Common backend scripts no longer carry configuration parameters to point to existing scripts that sit in the same repository. It references the scripts within this repo.
ArtifactRepositoryHelpers methods upload and download are enhanced to return their return code

dennis-behm added 8 commits

January 30, 2025 10:47


          Fetch build dependencies based on AD

c8ca5db

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          Missing import of Utilities

82a3809

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          Align fetchDependencies script with Application Descriptor

e9eab81

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          initial documentation

8de7794

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          initial documentation

f584cfc

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          Consolidate fetch script for both dbbBuild.sh and zBuilder.sh

9e98a02

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          Set execute bits for zBuilder.sh

7b49b4e

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          Update release info and add page break

ca23c00

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

M-DLB reviewed

View reviewed changes

Member

M-DLB left a comment

A few comments.

Templates/Common-Backend-Scripts/README.md Outdated Show resolved Hide resolved

Templates/Common-Backend-Scripts/README.md Outdated Show resolved Hide resolved

Templates/Common-Backend-Scripts/dbbBuild.sh Outdated Show resolved Hide resolved

Templates/Common-Backend-Scripts/pipelineBackend.config Outdated

    
              # Path to main build script build.groovy.

              zAppBuild="$HOME/dbb-zappbuild"

              # DBEHM: MyCustomization for Path to build.groovy.

Member

M-DLB Feb 5, 2025

Would it make sense to remove this line and the next one?

Member Author

dennis-behm Feb 5, 2025

This will be just for testing for the moment

Templates/Common-Backend-Scripts/pipelineBackend.config Outdated

    
              # (Optional) Additional property files to customize the build framework

              zAppBuildPropFiles=""            

              zAppBuildPropFiles="/var/dbb/dbb-zappbuild-config/build.properties,/var/dbb/dbb-zappbuild-config/datasets.properties,/var/dbb/dbb-zappbuild-config/buildFolder.properties"

Member

M-DLB Feb 5, 2025

Would it make sense to remove this line?

Templates/Common-Backend-Scripts/utilities/fetchBuildDependencies.groovy Outdated

    
              importFolder.mkdirs()

              // setup package cache

              cacheFolder = new File("/tmp/.pkg")

Member

M-DLB Feb 5, 2025

Can we externalize this cache folder location with a property in the pipelineBackend.config?

Member Author

dennis-behm Feb 5, 2025

I have updated the pipelineBackend.config with new settings and implemented the logic for the fetch

Templates/Common-Backend-Scripts/utilities/fetchBuildDependencies.groovy Outdated Show resolved Hide resolved

Templates/Common-Backend-Scripts/utilities/fetchBuildDependencies.groovy Outdated

    
              				println("** Downloading application package '$artifactUrl' from Artifact Repository into ${cacheFolder}/${artifactRelPath}.")

              				def rc = artifactRepositoryHelpers.download(artifactUrl, tarFile as String, user, password, true)

              				println "download complete $rc" // TODO: Error handling in helper

Member

M-DLB Feb 5, 2025

Could we have a better info message, and a handling of rc != 0?

Templates/Common-Backend-Scripts/utilities/fetchBuildDependencies.groovy Outdated

    
              	}

              }

              // Do we actually need this or can this just be obtained from the dependencies?

Member

M-DLB Feb 5, 2025

The Baseline section of the AD file is meant to describe "dependency" on itself for a given application. It can serve two purposes: the first one is downloading the packages from earlier version that can be used as a base for artifacts that are not rebuilt (i.e., private object decks, generated include files from BMS, ...). The second one is to detect the baselineReference tag during an impact build.

I think the next section is necessary. Maybe it can be optimized to be turned into a method, to minimize the common code with the previous section.

Member

M-DLB Mar 21, 2025

I still disagree to use the dependencies section to load our own baseline, we should use the baseline section of the AD file to retrieve the baseline archive from the artifactory repository.

Templates/Common-Backend-Scripts/utilities/fetchBuildDependencies.groovy Outdated

    
              	yamlBuilder externalDependencies

              	externalDependencyFile = new File(props.externalDependenciesFilePath)

              	externalDependencyFile.withWriter("IBM-1047") { extDepWriter ->

Member

M-DLB Feb 5, 2025

For the sake of clarity, tagging is missing, even if it might not be useful.

dennis-behm and others added 21 commits

February 5, 2025 16:47


          Doc updates from code review

98e9de0

Co-authored-by: Mathieu Dalbin <mathieu.dalbin@fr.ibm.com>


          Configure Cache directory

b87255e

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          Configure Cache directory

9a1f583

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          Configure Cache directory

e923fd2

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          Configure Cache directory

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          Add dbbCommunityRepoRoot

4a32af5

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          Read Application Descriptor via utils

27587cf

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          Read Application Descriptor via utils

7e1ac6d

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          Read Application Descriptor via utils

88f2df3

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          Update Packaging process

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          Update Packaging notes

d4bd7dd

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          packageUtils

4aba150

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          Add eyecatcher to build frameworks

43797c7

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          why on earth again

4b25d17

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

bla

2d9da76

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          add buildIdentifier and releaseIdentifier

333c0f7

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          Compute publishing parameters for packaging

2eea9b1

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          fix duplicate variable usage

068cc8b

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          introduce packageBuildIdentifier

fd398d2

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          Refactored PackageBuildOutputs to return url

d09cdac

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          reference PackagingScript

86a5b61

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

dennis-behm mentioned this pull request

Reference cleanup scripts in generateCleanupCommand.sh #307

Merged


          README updates

b1bf88e

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

lauren-li reviewed

View reviewed changes

Templates/Common-Backend-Scripts/README.md Outdated Show resolved Hide resolved

Templates/Common-Backend-Scripts/README.md Outdated Show resolved Hide resolved

Templates/Common-Backend-Scripts/README.md Outdated Show resolved Hide resolved

Templates/Common-Backend-Scripts/README.md Outdated Show resolved Hide resolved

Templates/Common-Backend-Scripts/README.md Outdated Show resolved Hide resolved

dennis-behm and others added 5 commits

March 28, 2025 14:15


          Apply suggestions from code review

858591b

Co-authored-by: Lauren Li <45975633+lauren-li@users.noreply.github.com>


          numbering in readme

15504fe

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          Switch to UTF-8 output

1391d58

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          Fix path in ucdPackaging

deeb9bb

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          Remove configuration

e4e41f2

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

M-DLB reviewed

View reviewed changes

Member

M-DLB left a comment

A couple of changes. I think we're close, but we could simplify it a bit more, especially with not printing "packageUrl=" when calling computePackageUrl. By the way can we rename it to computeArchiveUrl? :)
I've tried to remove occurrences of Package as much as I can and i know I missed a few, if you can have a look, please.

Pipeline/PackageBuildOutputs/PackageBuildOutputs.groovy Outdated Show resolved Hide resolved

Pipeline/PackageBuildOutputs/utilities/applicationDescriptorUtils.groovy

    
                      dependency.name = applicationDependency

                      dependency.version = version

                      dependency.type = type

              		dependency.type = type

Member

M-DLB Apr 1, 2025

There is a mix a tabs and spaces in the indentation.

Member Author

dennis-behm Apr 2, 2025

Format applied

Pipeline/PackageBuildOutputs/utilities/applicationDescriptorUtils.groovy

    
               * If an existing baseline for a given branch already exists, the method replaces it

               */

              def addBaseline(ApplicationDescriptor applicationDescriptor, String branch, String baseline) {

Member

M-DLB Apr 1, 2025

Why is this method dropped?

Member Author

dennis-behm Apr 1, 2025

I think this is derived from the MM. We don't need it here - @M-DLB do you want me to add it back in? Then we can share the entire utils interchangeability?

Member

M-DLB Apr 1, 2025

Yes please. For easy management, we should have the exact same version than in the MM. You then might want to open a PR on the MM, to update it as well there :)

Templates/Common-Backend-Scripts/packageBuildOutputs.sh Outdated Show resolved Hide resolved

Templates/Common-Backend-Scripts/packageBuildOutputs.sh Outdated Show resolved Hide resolved

Templates/Common-Backend-Scripts/utilities/packagingUtilities.sh Outdated Show resolved Hide resolved

Templates/Common-Backend-Scripts/wazideploy-deploy.sh Outdated Show resolved Hide resolved

Templates/Common-Backend-Scripts/wazideploy-deploy.sh Outdated Show resolved Hide resolved

Templates/Common-Backend-Scripts/wazideploy-deploy.sh Show resolved Hide resolved

Templates/Common-Backend-Scripts/wazideploy-generate.sh

    
                  if [[ ! ${PackageInputFile:0:1} == "/" ]] ; then 

                      checkWorkspace

                      PackageInputFile="$(getLogDir)/${PackageInputFile}"

                  if [ ! ${PackageInputFile:0:1} == "/" ] && [ -z "${usePackageUrl}" ]; then

Member

M-DLB Apr 1, 2025

ArchiveInputFile? I know WD uses packageInputFile...
No need to change...

Member Author

dennis-behm Apr 2, 2025

With introducing the Archive - we already start with a new terminology. We should apply the Wazi Deploy terms here.

dennis-behm and others added 12 commits

April 1, 2025 17:15


          Apply suggestions from code review

275a526

Co-authored-by: Mathieu Dalbin <mathieu.dalbin@fr.ibm.com>


          Use Archive instead of Package

51a0da2

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          Fix renamed object names

9de0df9

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          sync with latest AD schema

eaca963

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          code formatting

5da0453

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          Updating messages for more consistency

f03c412

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          rename to archiveCache

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          disable new feature

6134ce3

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          rename to just url

be55b7a

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          manage missing AD definitions

5d73f63

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          fix syntax error

28b88c4

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>


          fix syntax error

837e847

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>

M-DLB approved these changes

View reviewed changes

Member

M-DLB left a comment

Thanks @dennis-behm for these great enhancements!

dennis-behm merged commit 28eb9a5 into IBM:main

1 check failed

dennis-behm added a commit to dennis-behm/dbb that referenced this pull request


          CBS script enhancements to manage packages lifecycle and ability to f…

5532c1b

…etch external dependencies (IBM#299)

* (CBS) packageBuildOutputs.sh to follow implemented naming conventions how archives are published to Artifact repository
* (CBS) dbbBuild.sh and zBuilder.sh to process build dependency configuration defined in the Application Descriptor
* (CBS) Download external dependencies based on naming conventions from the artifact repository (Nexus, Artifactory) using the ArtifactRepositoryHelpers
* (CBS) Expanding the external dependencies in the build workspace for consumption by the build framework
* (CBS) Central cache of external archives
* (CBS) Enable wazideploy-generate to automatically download the tar from the Artifact repository
* (CBS) Scripts no longer carry configuration parameters to point to existing scripts. It references the scripts directly within this repo.

---------

Signed-off-by: Dennis Behm <dennis.behm@de.ibm.com>
Co-authored-by: Mathieu Dalbin <mathieu.dalbin@fr.ibm.com>
Co-authored-by: Lauren Li <45975633+lauren-li@users.noreply.github.com>

This was referenced Apr 9, 2025

Align Application Descriptor Schema with latest structural updates IBM/dbb-git-migration-modeler#29

Closed

Enhance CBS scripts to pull configured application dependencies #297

Closed

Update pipeline templates to use updated package lifecycle #311

Open

This was referenced Apr 16, 2025

Update application descriptor with latest structural updates IBM/dbb-git-migration-modeler#30

Merged

Enable upload of object decks from build workspace into target libraries IBM/dbb-zappbuild#609

Closed

dennis-behm mentioned this pull request

Resolve and upload object decks from build workspace into target libraries IBM/dbb-zappbuild#610

Merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet