Backport changes needed to have linux-pkg run on 6.0/migration

[ghost]

Context:

https://jira.delphix.com/browse/TOOL-7953 tracks the removal of all Jenkins jobs from non devops-gate repositories. This includes the build-package job. The build-package job can only be removed from branches that support the new linux-pkg build system 6.0/migration is the last in-use branch that does not support linux-pkg.

Problem:

6.0/migration does not support the linux-pkg build system

Solution:

Backport support to this branch

Testing

1. Start the ops container
2. Run generate-jobs and modify the jobs.json blob to contain a link to my fork (http://github.com/jtamagnan-delphix/linux-pkg.git)
3. Run http://collins.d.delphix.com:20460/job/devops-gate/job/master/job/upgrade-verify/job/master/job/post-push/1/ job to generate an artifact that would be used later on
4. Start the bulid-package job:

• set BUILD_POLICY=missing
• set LINUX_PKG_GIT_URL=http://github.com/jtamagnan-delphix/linux-pkg.git

http://collins.d.delphix.com:20460/job/devops-gate/job/master/job/linux-pkg/job/6.0/job/migration/job/build-packages/job/pre-push/9/console

5. Once the build-packages job finished I was able to kick off appliance-build-orchestrator-pre-push:

• set UPSTREAM_PRODUCT_BRANCH=6.0/migration
• set COMBINED_PACKAGES_S3_URL=s3://dev-de-images/builds/jenkins-ops.jtamagnan/devops-gate/master/linux-pkg/6.0/migration/combine-packages/pre-push/3

http://selfservice.jenkins.delphix.com/job/devops-gate/job/master/job/appliance-build-orchestrator-pre-push/5187/console

Notes about the appliance-build-orchestrator-pre-push job:

• 6.0/migration is not supported by the blackbox-self-service job, it kept trying to pick the dlpx-6.0.6.2 dlpx-qa-gate branch which does not exist. I rebuilt the blackbox-self-service job with the correct dlpx-6.0.6.0 branch: http://selfservice.jenkins.delphix.com/job/devops-gate/job/master/job/blackbox-self-service/90612/console
• The split-precommit-unittest job fails because there is no dlpx-internal-unittest-6.0/migration image. This doesn't seem related to my change but instead it seems like we've opted to not generate that image
• The final testing sub-job dxostest is running correctly as expected http://selfservice.jenkins.delphix.com/job/devops-gate/job/master/job/integration-tests/job/split-precommit-dxostest/job/pre-push/258/console

Implementation:

I backported four diffs that were sent to me by Prakash:

• 10a9b24 TOOL-11245 leverage linux-pkg to build delphix-virtualization package (TOOL-11245 leverage linux-pkg to build delphix-virtualization package #145)
• 8449592 TOOL-11245 leverage linux-pkg to build delphix-virtualization package (TOOL-11245 leverage linux-pkg to build delphix-virtualization package #144)
• 8ad7f2c TOOL-11187 leverage linux-pkg to build delphix-masking package (TOOL-11187 leverage linux-pkg to build delphix-masking package #141)
  • While applying this diff I noticed that 6.0/migration was missing misc-debs. I did not add it
• c0f287d TOOL-11187 leverage linux-pkg to build delphix-masking package (TOOL-11187 leverage linux-pkg to build delphix-masking package #139)

I also found that I needed to backport:

• 219037a DLPX-73610 pin unzip package to version 6.0-21ubuntu1 (DLPX-73610 pin unzip package to version 6.0-21ubuntu1 #132)

Related diffs:

Pipeline-shared change to mark 6.0/migration as a linux-pkg branch: http://reviews.delphix.com/r/68916/

Deployment Plan:

For the moment my plan will be to:

• Push this change
• Push the pipeline shared change
• [at this point the changes will take effect others diffs will be cleanup]
• Push the change to remove the build-package job usage from the devops-gate
• Remove the build-package job from the dms-core-gate and dlpx-app-gate

[ghost]

Requesting reviews from @pzakha and @prakashsurya

[prakashsurya]

Changes seem reasonable, it's just pulling things from other branches.

I don't know how the 6.0/migration branch is maintained, so I can't say if this is OK to land or not; will have to defer to others on that.

I don't think you can land the changes to main.pkgs at the same time you land the changes to add those packages to the linux-pkg ecosystem. I think you need to first add the packages to linux-pkg, then go a build the packages via Jenkins, then add the package names to main.pkgs.

[pzakha]

LGTM, thanks for landing this.

I don't think you can land the changes to main.pkgs at the same time you land the changes to add those packages to the linux-pkg ecosystem. I think you need to first add the packages to linux-pkg, then go a build the packages via Jenkins, then add the package names to main.pkgs.

@prakashsurya that's right, you shouldn't be doing this on any active branch as once this lands the combine-packages job will be failing until generate-jobs is run and the new packages are built and published, however given that the 6.0/migration branch is on standby and the are no pending builds it's okay to do it in this case.