add the posibility to upgrade while using a local repository #1099

PeterMocary · 2023-07-12T20:41:01Z

Upgrade with a local repository required to host the repository locally for it to be visible from target user-space container during the upgrade. The added local_repos actor ensures that the local repository will be visible from the container by adjusting its baseurl (and possibly the mirrorlist as well, since they are interchangable) path - simply by prefixing a host root mount bind /installroot to it. The local_repos_inhibit actor is no longer needed, thus was removed.

Jira: OAMG-3324
BZ: BZ-1841198 / RHEL-3273 / OAMG-3324
Test MR: 277

Tested upgrade from rhel7 to rhel8:

Created local repository with sed package downloaded from target BaseOS repository and added exclude=sed to the BaseOS repository. That way only the local repository provides the sed package.
Using the ISO of rhel 8.8 to make sure that the changes don't affect local repositories provided by the ISO. This time I created a custom package for the testing and a local repository with higher version of the package for the upgrade.

Note:
It would be possible to remove dependency on TMPTargetRepositoriesFacts model simply by adjusting every file in the container /etc/yum.repos.d directory when needed. However, the missinggpgkeysinhibitor actor uses this model as well so it would not lead to the removal of the model all together, thus I used it.

github-actions · 2023-07-12T20:41:15Z

Thank you for contributing to the Leapp project!

Please note that every PR needs to comply with the Leapp Guidelines and must pass all tests in order to be mergeable.
If you want to request a review or rebuild a package in copr, you can use following commands as a comment:

review please @oamg/developers to notify leapp developers of the review request
/packit copr-build to submit a public copr build using packit

Packit will automatically schedule regression tests for this PR's build and latest upstream leapp build. If you need a different version of leapp from PR#42, use /packit test oamg/leapp#42

To launch regression testing public members of oamg organization can leave the following comment:

/rerun to schedule basic regression tests using this pr build and latest upstream leapp build as artifacts
/rerun 42 to schedule basic regression tests using this pr build and leapp*PR42* as artifacts
/rerun-sst to schedule sst tests using this pr build and latest upstream leapp build as artifacts
/rerun-sst 42 to schedule sst tests using this pr build and leapp*PR42* as artifacts

Please open ticket in case you experience technical problem with the CI. (RH internal only)

Note: In case there are problems with tests not being triggered automatically on new PR/commit or pending for a long time, please contact leapp-infra.

matejmatuska

Besides the comments I tested an upgrade with similar setup - sed package in local repo LOCAL_SED and excluded from target system BaseOS repo.

The package was excluded:

leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: Excludes in repo BASEOS: sed

And the package was found in the local repo:

leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator:  sed                         x86_64 4.5-5.el8                   LOCAL_SED 298 k

And upgraded:

leapp.workflow.RPMUpgrade.dnf_upgrade_transaction: Upgraded: sed-4.5-5.el8.x86_64

The upgrade finished succesfully.

I haven't tested with the ISO yet.

repos/system_upgrade/common/actors/localrepos/actor.py

repos/system_upgrade/common/actors/localrepos/libraries/localrepos.py

repos/system_upgrade/common/actors/localrepos/tests/test_localrepos.py

matejmatuska · 2023-07-21T14:12:06Z

I did an additional test with reposynced local BaseOS and AppStream repos.

Without this patch the upgrade is inhibited due to local repos as expected.

With this patch the upgrade is not inhibited and the packages are successfully downloaded in dnf_package_download actor, however in the upgrade_initramfs_generator the packages cannot be downloaded correctly at

leapp-repository/repos/system_upgrade/common/actors/initramfs/upgradeinitramfsgenerator/libraries/upgradeinitramfsgenerator.py

Line 155 in 5015311

def _install_initram_deps(packages):

because the /installroot bind mount isn't present.

PeterMocary · 2023-07-25T13:15:52Z

I added the /installroot mount bind to the transaction that installs initramdisk dependencies. I tested it with hostname and hwdata packages in a local repository since these packages have been among those that were not found prior to this mount bind patch.

Thank you @matejmatuska for finding the issue!

matejmatuska · 2023-07-31T11:18:00Z

I did an additional test with reposynced local BaseOS and AppStream repos.

Without this patch the upgrade is inhibited due to local repos as expected.

With this patch the upgrade is not inhibited and the packages are successfully downloaded in dnf_package_download actor, however in the upgrade_initramfs_generator the packages cannot be downloaded correctly at

leapp-repository/repos/system_upgrade/common/actors/initramfs/upgradeinitramfsgenerator/libraries/upgradeinitramfsgenerator.py

Line 155 in 5015311

def _install_initram_deps(packages):

because the /installroot bind mount isn't present.

I rerun the test with the new patches and the upgrade went smoothly this time.
@pirat89 Can you please check if there is anything to consider regarding the added bind mount?

repos/system_upgrade/common/actors/adjustlocalrepos/actor.py

repos/system_upgrade/common/libraries/dnfplugin.py

matejmatuska · 2023-08-03T15:01:05Z

I did an additional test with reposynced local BaseOS and AppStream repos.
Without this patch the upgrade is inhibited due to local repos as expected.
With this patch the upgrade is not inhibited and the packages are successfully downloaded in dnf_package_download actor, however in the upgrade_initramfs_generator the packages cannot be downloaded correctly at

leapp-repository/repos/system_upgrade/common/actors/initramfs/upgradeinitramfsgenerator/libraries/upgradeinitramfsgenerator.py

Line 155 in 5015311

def _install_initram_deps(packages):

because the /installroot bind mount isn't present.

I rerun the test with the new patches and the upgrade went smoothly this time. @pirat89 Can you please check if there is anything to consider regarding the added bind mount?

I tested this on 7 to 8 as well (as I failed to mention I did the test on 8 to 9 previously) and the upgrade went smoothly as well.

@PeterMocary after you do the changes that @pirat89 requested I think I can approve this :).

PeterMocary · 2023-08-14T15:30:00Z

Added the requested error log and rebased. @matejmatuska

pirat89 · 2023-08-16T09:46:40Z

depends on #1114

PeterMocary · 2023-08-16T13:41:36Z

Rebased to include the PR#1114.

pirat89 · 2023-08-25T09:07:57Z

@PeterMocary rebase please

matejmatuska · 2023-08-28T13:45:16Z

Peter and I have tested this multiple times with different configurations. All the requested changes were addressed, approving 🚀

fernflower · 2023-09-07T14:13:49Z

Please rebase, let's make the tests green again

fernflower · 2023-09-07T14:22:01Z

And just a small note - regression tests (not the sanity ones) do have a test that upgrade is inhibited with file:// based repositories (OAMG-3324).
So before merging the code it would be nice to check with the QE :)

pirat89 · 2023-09-08T13:59:49Z

@fernflower good point!

Upgrade with a local repository required to host the repository locally for it to be visible from target user-space container during the upgrade. The added actor ensures that the local repository will be visible from the container by adjusting the path to it simply by prefixing a host root mount bind '/installroot' to it. The local_repos_inhibit actor is no longer needed, thus was removed.

dkubek · 2023-10-03T12:29:20Z

/packit build

pirat89 · 2023-10-10T12:47:14Z

Synced with @mmacura311. We will need possibly another changes in tests. This PR should be merged together with update of tests.

pirat89 · 2023-11-03T15:30:56Z

The testing was postponed due to other high prio work. But it's expected to deal with it during the next week

pirat89 · 2023-11-10T14:56:03Z

Discussed with QE. The problematic test is going to be temporarily disabled so the merge is not blocked anymore. It'll be resolved on monday

pirat89 · 2023-11-14T10:45:22Z

Merging! Thanks @PeterMocary !! 🥳

## Packaging - Requires xfsprogs and e2fsprogs (oamg#1154) - Bump leapp-repository-dependencies to 10 (oamg#1154) ## Upgrade handling ### Fixes - Detect changes in openssl default configuration file and restore it to the default to the target default during the upgrade to reduce risk of potential issues (oamg#1131) - Do not try to download data files anymore when missing as the service is obsoleted since the data is part of installed packages (oamg#1120) - Drop the invalid `tuv` target channel (oamg#1130) - Fix handling of symlinks under /etc/pki when managing certificates (oamg#1135, oamg#1160, oamg#1166) - Fix semanage import issue (oamg#1164) - Fix the issue of going out of bounds in the isccfg parser (oamg#1124) - Fix traceback when saving the rhsm facts results and the /etc/rhsm/facts directory doesn’t exist yet (oamg#1132) - Handle the upgrade better when a proxy is configured in YUM/DNF configutations (oamg#1143) - Load all rpm repository substitutions that dnf knows about, not just releasever since repofiles may use the other substitutions too (oamg#1134) - Minor updates of generated reports (oamg#1151) - Print nice error msg when device and driver deprecation data is malformed (oamg#1168) - Report information about required manual steps after the upgrade when openssl-ibmca is installed (oamg#1131) - Update error messages and reports when installed upgrade data files are malformed or missing (oamg#1120) - [IPU 7 -> 8] Fix the upgrade of the RH Satellite server when tomcat is installed (oamg#1150) - [IPU 8 -> 9] Fix the upgrade from RHEL 8.9+ when the release is locked by subscription-manager (oamg#1136, oamg#1138) ### Enhancements - Update upgrade paths: (oamg#1146, oamg#1147, oamg#1175) - RHEL 7.9 -> 8.10, 8.8 (default: 8.10) - RHEL with SAPAHA 7.9 -> 8.10, 8.8 (default: 8.8) - RHEL w/o SAP HANA 8.8 -> 9.2 - RHEL w/o SAP HANA 8.10 -> 9.4 - Added possibility to define DNF configuration for the target system (oamg#1143) - Code cleaning: drop redundant and invalid NFS checks (oamg#1127) - Default to NO_RHSM mode when subscription-manager is not found (oamg#1133) - Detect customized configuration of dynamic linker (oamg#1118) - Detect possible unexpected RPM GPG keys has been installed during RPM transaction (oamg#1101) - Drop obsoleted upgrade paths that relates to releases: 8.6, 8.9, 9.0, 9.3 (oamg#1175) - Ignore Leapp related PES events (oamg#1153) - Introduce generic transition of systemd services states during the IPU (oamg#1060, oamg#1174) - Introduce possibility to upgrade with local repositories (oamg#1099) - Introduced some changes getting us closer to possibility of IPU for Centos (Stream) systems (oamg#1140) - Report the upgrade customisations and modifications of the upgrade tooling (oamg#1148) - Simplify handling of upgrades on systems using RHUI, reducing the maintenance burden for cloud providers (oamg#1057) - Update the leapp upgrade data files - bump data stream to "3.0" (oamg#1163, oamg#1165, oamg#1170) - [IPU 8 -> 9] Enable upgrades RHEL 8 -> 9 using RHUI on Alibaba cloud (oamg#1137, oamg#1165, oamg#1172) ## Additional changes interesting for devels - Introduced new functions returning a list of packages related to upgrade - see the rpms library (oamg#1156) - Make detection of installed signed packages distribution agnostic - covers RHEL & CentOS (oamg#876) - Model InstalledRedHatSignedRPM is deprecated, replaced by DistributionSignedRPM (oamg#876)

## Packaging - Requires xfsprogs and e2fsprogs (oamg#1154) - Bump leapp-repository-dependencies to 10 (oamg#1154) ## Upgrade handling ### Fixes - Detect changes in openssl default configuration file and restore it to the default to the target default during the upgrade to reduce risk of potential issues (oamg#1131) - Do not try to download data files anymore when missing as the service is obsoleted since the data is part of installed packages (oamg#1120) - Drop the invalid `tuv` target channel (oamg#1130) - Fix handling of symlinks under /etc/pki when managing certificates (oamg#1135, oamg#1160, oamg#1166) - Fix semanage import issue (oamg#1164) - Fix the issue of going out of bounds in the isccfg parser (oamg#1124) - Fix traceback when saving the rhsm facts results and the /etc/rhsm/facts directory doesn’t exist yet (oamg#1132) - Handle the upgrade better when a proxy is configured in YUM/DNF configutations (oamg#1143) - Load all rpm repository substitutions that dnf knows about, not just releasever since repofiles may use the other substitutions too (oamg#1134) - Minor updates of generated reports (oamg#1151) - Print nice error msg when device and driver deprecation data is malformed (oamg#1168) - Report information about required manual steps after the upgrade when openssl-ibmca is installed (oamg#1131) - Update error messages and reports when installed upgrade data files are malformed or missing (oamg#1120) - [IPU 7 -> 8] Fix the upgrade of the RH Satellite server when tomcat is installed (oamg#1150) - [IPU 8 -> 9] Fix the upgrade from RHEL 8.9+ when the release is locked by subscription-manager (oamg#1136, oamg#1138) ### Enhancements - Update upgrade paths: (oamg#1146, oamg#1147, oamg#1175) - RHEL 7.9 -> 8.10, 8.8 (default: 8.10) - RHEL with SAPAHA 7.9 -> 8.10, 8.8 (default: 8.8) - RHEL w/o SAP HANA 8.8 -> 9.2 - RHEL w/o SAP HANA 8.10 -> 9.4 - Added possibility to define DNF configuration for the target system (oamg#1143) - Code cleaning: drop redundant and invalid NFS checks (oamg#1127) - Default to NO_RHSM mode when subscription-manager is not found (oamg#1133) - Detect customized configuration of dynamic linker (oamg#1118) - Detect possible unexpected RPM GPG keys has been installed during RPM transaction (oamg#1101) - Drop obsoleted upgrade paths that relates to releases: 8.6, 8.9, 9.0, 9.3 (oamg#1175) - Ignore Leapp related PES events (oamg#1153) - Introduce generic transition of systemd services states during the IPU (oamg#1060, oamg#1174) - Introduce possibility to upgrade with local repositories (oamg#1099) - Introduced some changes getting us closer to possibility of IPU for Centos (Stream) systems (oamg#1140) - Report the upgrade customisations and modifications of the upgrade tooling (oamg#1148) - Simplify handling of upgrades on systems using RHUI, reducing the maintenance burden for cloud providers (oamg#1057) - Update the leapp upgrade data files - bump data stream to "3.0" (oamg#1163, oamg#1165, oamg#1170) - [IPU 8 -> 9] Enable upgrades RHEL 8 -> 9 using RHUI on Alibaba cloud (oamg#1137, oamg#1165, oamg#1172) - Unify breakpoints inside the upgrade initramfs for the easier troubleshooting (oamg#1157) ## Additional changes interesting for devels - Introduced new functions returning a list of packages related to upgrade - see the rpms library (oamg#1156) - Make detection of installed signed packages distribution agnostic - covers RHEL & CentOS (oamg#876) - Model InstalledRedHatSignedRPM is deprecated, replaced by DistributionSignedRPM (oamg#876)

## Packaging - Requires xfsprogs and e2fsprogs (#1154) - Bump leapp-repository-dependencies to 10 (#1154) ## Upgrade handling ### Fixes - Detect changes in openssl default configuration file and restore it to the default to the target default during the upgrade to reduce risk of potential issues (#1131) - Do not try to download data files anymore when missing as the service is obsoleted since the data is part of installed packages (#1120) - Drop the invalid `tuv` target channel (#1130) - Fix handling of symlinks under /etc/pki when managing certificates (#1135, #1160, #1166) - Fix semanage import issue (#1164) - Fix the issue of going out of bounds in the isccfg parser (#1124) - Fix traceback when saving the rhsm facts results and the /etc/rhsm/facts directory doesn’t exist yet (#1132) - Handle the upgrade better when a proxy is configured in YUM/DNF configutations (#1143) - Load all rpm repository substitutions that dnf knows about, not just releasever since repofiles may use the other substitutions too (#1134) - Minor updates of generated reports (#1151) - Print nice error msg when device and driver deprecation data is malformed (#1168) - Report information about required manual steps after the upgrade when openssl-ibmca is installed (#1131) - Update error messages and reports when installed upgrade data files are malformed or missing (#1120) - [IPU 7 -> 8] Fix the upgrade of the RH Satellite server when tomcat is installed (#1150) - [IPU 8 -> 9] Fix the upgrade from RHEL 8.9+ when the release is locked by subscription-manager (#1136, #1138) ### Enhancements - Update upgrade paths: (#1146, #1147, #1175) - RHEL 7.9 -> 8.10, 8.8 (default: 8.10) - RHEL with SAPAHA 7.9 -> 8.10, 8.8 (default: 8.8) - RHEL w/o SAP HANA 8.8 -> 9.2 - RHEL w/o SAP HANA 8.10 -> 9.4 - Added possibility to define DNF configuration for the target system (#1143) - Code cleaning: drop redundant and invalid NFS checks (#1127) - Default to NO_RHSM mode when subscription-manager is not found (#1133) - Detect customized configuration of dynamic linker (#1118) - Detect possible unexpected RPM GPG keys has been installed during RPM transaction (#1101) - Drop obsoleted upgrade paths that relates to releases: 8.6, 8.9, 9.0, 9.3 (#1175) - Ignore Leapp related PES events (#1153) - Introduce generic transition of systemd services states during the IPU (#1060, #1174) - Introduce possibility to upgrade with local repositories (#1099) - Introduced some changes getting us closer to possibility of IPU for Centos (Stream) systems (#1140) - Report the upgrade customisations and modifications of the upgrade tooling (#1148) - Simplify handling of upgrades on systems using RHUI, reducing the maintenance burden for cloud providers (#1057) - Update the leapp upgrade data files - bump data stream to "3.0" (#1163, #1165, #1170) - [IPU 8 -> 9] Enable upgrades RHEL 8 -> 9 using RHUI on Alibaba cloud (#1137, #1165, #1172) - Unify breakpoints inside the upgrade initramfs for the easier troubleshooting (#1157) ## Additional changes interesting for devels - Introduced new functions returning a list of packages related to upgrade - see the rpms library (#1156) - Make detection of installed signed packages distribution agnostic - covers RHEL & CentOS (#876) - Model InstalledRedHatSignedRPM is deprecated, replaced by DistributionSignedRPM (#876)

PeterMocary force-pushed the local-file-repository branch 3 times, most recently from f320f2a to 6a0d9f3 Compare July 13, 2023 14:14

PeterMocary marked this pull request as ready for review July 13, 2023 19:18

matejmatuska requested changes Jul 18, 2023

View reviewed changes

PeterMocary force-pushed the local-file-repository branch from 6a0d9f3 to ca51920 Compare July 20, 2023 09:26

pirat89 added this to the 8.10/9.4 milestone Aug 1, 2023

pirat89 reviewed Aug 3, 2023

View reviewed changes

repos/system_upgrade/common/actors/adjustlocalrepos/actor.py Outdated Show resolved Hide resolved

repos/system_upgrade/common/libraries/dnfplugin.py Show resolved Hide resolved

PeterMocary force-pushed the local-file-repository branch from c169b07 to ac9cb20 Compare August 14, 2023 15:27

PeterMocary force-pushed the local-file-repository branch from ac9cb20 to eb3cafa Compare August 16, 2023 13:10

PeterMocary force-pushed the local-file-repository branch 2 times, most recently from 1d60c92 to c459789 Compare August 28, 2023 11:24

matejmatuska approved these changes Aug 28, 2023

View reviewed changes

PeterMocary force-pushed the local-file-repository branch from c459789 to 617fa68 Compare September 13, 2023 13:54

fernflower approved these changes Sep 19, 2023

View reviewed changes

pirat89 mentioned this pull request Oct 10, 2023

make pylint happy #1129

Merged

pirat89 added the changelog-checked The merger/reviewer checked the changelog draft document and updated it when relevant label Nov 3, 2023

pirat89 merged commit e9f899c into oamg:master Nov 14, 2023
17 of 29 checks passed

pirat89 added the report Any reports have been added / removed / changed in the PR label Nov 15, 2023

pirat89 mentioned this pull request Feb 13, 2024

Release 0.20.0 #1177

Merged

pirat89 mentioned this pull request Apr 17, 2024

Package "X" from local repository "Y" has incorrect checksum #1203

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add the posibility to upgrade while using a local repository #1099

add the posibility to upgrade while using a local repository #1099

PeterMocary commented Jul 12, 2023 •

edited by pirat89

github-actions bot commented Jul 12, 2023

matejmatuska left a comment

matejmatuska commented Jul 21, 2023

PeterMocary commented Jul 25, 2023 •

edited

matejmatuska commented Jul 31, 2023

matejmatuska commented Aug 3, 2023

PeterMocary commented Aug 14, 2023

pirat89 commented Aug 16, 2023

PeterMocary commented Aug 16, 2023

pirat89 commented Aug 25, 2023

matejmatuska commented Aug 28, 2023

fernflower commented Sep 7, 2023

fernflower commented Sep 7, 2023

pirat89 commented Sep 8, 2023

dkubek commented Oct 3, 2023

pirat89 commented Oct 10, 2023

pirat89 commented Nov 3, 2023

pirat89 commented Nov 10, 2023 •

edited

pirat89 commented Nov 14, 2023

add the posibility to upgrade while using a local repository #1099

add the posibility to upgrade while using a local repository #1099

Conversation

PeterMocary commented Jul 12, 2023 • edited by pirat89

github-actions bot commented Jul 12, 2023

Thank you for contributing to the Leapp project!

matejmatuska left a comment

Choose a reason for hiding this comment

matejmatuska commented Jul 21, 2023

PeterMocary commented Jul 25, 2023 • edited

matejmatuska commented Jul 31, 2023

matejmatuska commented Aug 3, 2023

PeterMocary commented Aug 14, 2023

pirat89 commented Aug 16, 2023

PeterMocary commented Aug 16, 2023

pirat89 commented Aug 25, 2023

matejmatuska commented Aug 28, 2023

fernflower commented Sep 7, 2023

fernflower commented Sep 7, 2023

pirat89 commented Sep 8, 2023

dkubek commented Oct 3, 2023

pirat89 commented Oct 10, 2023

pirat89 commented Nov 3, 2023

pirat89 commented Nov 10, 2023 • edited

pirat89 commented Nov 14, 2023

PeterMocary commented Jul 12, 2023 •

edited by pirat89

PeterMocary commented Jul 25, 2023 •

edited

pirat89 commented Nov 10, 2023 •

edited