Proxy detection #914

PeterMocary · 2022-06-30T09:26:15Z

The actor scan_pkg_manager now scans the configuration file of package manager for proxy, which helps to identify the reason why the installation of packages in target_userspace_creator actor failed and provide better error message. The actor yum_config_scanner is merged into the scan_pkg_manager actor due to their similarity.

Jira ref: OAMG-6180
RHBZ: 2014009 (RFE)

github-actions · 2022-06-30T09:26:39Z

Thank you for contributing to the Leapp project!

Please note that every PR needs to comply with the Leapp Guidelines and must pass all tests in order to be mergable.
If you want to request a review or rebuild a package in copr, you can use following commands as a comment:

review please to notify leapp developers of review request
/packit copr-build to submit a public copr build using packit

To launch regression testing public members of oamg organization can leave the following comment:

/rerun to schedule basic regression tests using this pr build and leapp*master* as artifacts
/rerun 42 to schedule basic regression tests using this pr build and leapp*PR42* as artifacts
/rerun-all to schedule all tests (including sst) using this pr build and leapp*master* as artifacts
/rerun-all 42 to schedule all tests (including sst) using this pr build and leapp*PR42* as artifacts

Please open ticket in case you experience technical problem with the CI. (RH internal only)

Note: In case there are problems with tests not being triggered automatically on new PR/commit or pending for a long time, please consider rerunning the CI by commenting leapp-ci build (might require several comments). If the problem persists, contact leapp-infra.

MichalHe

Cool patch - I love that some actors were merged and you took the effort to change all code/docs properly. I have left you some comments, aiming at building your mindset around how to approach problems, and what to consider when implementing a patch. From my past experience, it really helps as a newcomer to use these reviews to really think about what you could do better next time and to improve.

repos/system_upgrade/common/actors/scanpkgmanager/libraries/scanpkgmanager.py

repos/system_upgrade/common/actors/scanpkgmanager/tests/test_scanpkgmanager.py

repos/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py

repos/system_upgrade/common/models/packagemanagerinfo.py

fernflower · 2022-07-11T08:39:58Z

/rerun

github-actions · 2022-07-11T08:45:23Z

Copr build succeeded: https://copr.fedorainfracloud.org/coprs/build/4622849

github-actions · 2022-07-11T09:04:36Z

Testing Farm request for RHEL-8.6-rhui/4622849 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2022-07-11T09:11:31Z

Testing Farm request for RHEL-7.9-rhui/4622849 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2022-07-11T09:22:36Z

Testing Farm request for RHEL-8.6.0-Nightly/4622849 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2022-07-11T09:40:13Z

Testing Farm request for RHEL-7.9-ZStream/4622849 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

fernflower

Nice refactoring!

repos/system_upgrade/common/actors/scanpkgmanager/libraries/scanpkgmanager.py

fernflower · 2022-07-13T14:50:00Z

/rerun

github-actions · 2022-07-13T14:55:44Z

Copr build succeeded: https://copr.fedorainfracloud.org/coprs/build/4632379

github-actions · 2022-07-13T15:14:55Z

Testing Farm request for RHEL-8.6-rhui/4632379 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2022-07-13T15:23:00Z

Testing Farm request for RHEL-7.9-rhui/4632379 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2022-07-13T15:31:59Z

Testing Farm request for RHEL-8.6.0-Nightly/4632379 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2022-07-13T15:59:22Z

Testing Farm request for RHEL-7.9-ZStream/4632379 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

repos/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py

MichalHe · 2022-07-14T15:59:04Z

Tested and works. However, proxies can be configured per repository, which could also be the source of the problem. However, this can be touched upon in a follow up task. Please fix the missing space, rewrite the commit message Scanpkgmanager actor now scans package manager config for proxy address leaving out redundant information implicit from commit history (now) e.g. Scanpkgmanager: scan for proxy configuration and then we are good to go.

MichalHe · 2022-07-18T09:14:21Z

/rerun

github-actions · 2022-07-18T09:19:17Z

Copr build succeeded: https://copr.fedorainfracloud.org/coprs/build/4640382

github-actions · 2022-07-18T09:38:34Z

Testing Farm request for RHEL-8.6-rhui/4640382 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2022-07-18T09:46:39Z

Testing Farm request for RHEL-7.9-rhui/4640382 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2022-07-18T09:57:36Z

Testing Farm request for RHEL-8.6.0-Nightly/4640382 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

MichalHe · 2022-07-18T10:26:10Z

I have started testing this patch, arriving at the following conclusions:

Performed tests

RHEL7

if a proxy is configured and the dnf command in the target_userspace_creator fails, the message contains a mention of the detected proxy configuration
if a proxy is not configured and the dnf command in the target_userspace_creator fails, the message does not contain a mention of the detected proxy configuration

I am suspending my testing of this patch, as agreed with @PeterMocary that the introduced changes should also include a scan for per-repository proxy configuration, so a follow up would not be needed then.

github-actions · 2022-07-18T11:36:53Z

Testing Farm request for RHEL-7.9-ZStream/4640382 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

MichalHe

I have tested the patch, works as expected. Code-wise OK, thus LGTM.

Let's wait on integration tests, and if they are OK I will merge the patch.

github-actions · 2022-08-01T15:12:55Z

Testing Farm request for RHEL-7.9-ZStream/4695242 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2022-08-01T15:17:06Z

Testing Farm request for RHEL-8.6-rhui/4695331 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2022-08-01T15:25:42Z

Testing Farm request for RHEL-7.9-rhui/4695331 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2022-08-01T15:34:24Z

Testing Farm request for RHEL-8.6.0-Nightly/4695331 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2022-08-01T15:57:28Z

Testing Farm request for RHEL-7.9-ZStream/4695331 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

pirat89

it was discussed that there is not enough capacity to cover the testing for this release already, so it's postponed for the next one. It can be merged right after the current release is created.

setting the required changes so it's not merged by mistake

pirat89 · 2022-09-12T10:58:12Z

@PeterMocary rebase please to resolve the conflicts

no reason for additional blocking

This new information enables targetuserspacecreator actor to inform user why the package installation might have failed

PeterMocary · 2022-09-12T13:57:23Z

@pirat89 Rebased.

PeterMocary · 2022-09-12T13:57:34Z

/rerun

github-actions · 2022-09-12T14:01:45Z

Copr build succeeded: https://copr.fedorainfracloud.org/coprs/build/4833243

github-actions · 2022-09-12T14:09:49Z

Testing Farm request for RHEL-8.6-rhui/4833243 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2022-09-12T15:16:10Z

Testing Farm request for RHEL-8.7.0-Nightly/4833243 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2022-09-12T15:26:05Z

Testing Farm request for RHEL-8.6.0-Nightly/4833243 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2022-09-12T15:37:15Z

Testing Farm request for RHEL-7.9-ZStream/4833243 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2022-09-12T15:38:32Z

Testing Farm request for RHEL-7.9-ZStream/4833243 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2022-09-12T16:12:44Z

Testing Farm request for RHEL-7.9-rhui/4833243 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

pirat89

merging \o

## Packaging - Requires cpio (oamg#979) - Requires python3-gobject-base, NetworkManager-libnm (oamg#969) - Bump leapp-repository-dependencies to 9 (oamg#969, oamg#979) ## Upgrade handling ### Fixes - Add leapp RHUI packages to an allowlist to drop confusing reports (oamg#995) - Check only mounted XFS partitions (oamg#1027) - Detect the kernel-core RPM instead of kernel to prevent an error during post-upgrade phases (oamg#1024) - Disable the amazon-id DNF plugin on AWS during the upgrade stage to omit error messages during the upgrade process caused by missing network connection (oamg#990) - Do not create new *pyc files when running leapp after the DNF upgrade transaction (oamg#1017) - Enable upgrades on s390x when /boot is part of rootfs (oamg#991) - Extend the allow list of RHUI clients by azure-sap-apps to omit confusing report (oamg#974) - Filter out PES events unrelated for the used upgrade path and handle overlapping events (oamg#1008) - Fix scan of ceph volumes on systems without ceph-osd (oamg#1011) - Fix scan of ceph volumes when ceph-osd container is not found (oamg#986) - Fix systemd symlinks that become incorrect during the IPU (oamg#972) - Fix the check of memory (RAM) limits (oamg#984) - Fix the upgrade of IBM Z machines configured with ZFCP (oamg#983) - Ignore external accounts in /etc/passwd (oamg#958) - Inhibit the upgrade when entries in /etc/fstab cause overshadowing during the upgrade (oamg#1009) - Prevent leapp failures caused by re-run of leapp in the upgrade initramfs after previous failure, which causes additional confusing error message hiding original bugs (oamg#996) - Prevent the upgrade with RHSM when a baseos and an appstream target repositories are not discovered (oamg#1001) - RHUI(Azure) handle correctly various SAP images (oamg#1037) - Rework the network configuration handling and parse the configuration data properly (oamg#969) - Set RHSM release for non-ga and non-beta channels (oamg#1033) - Use the "grub" library to find the GRUB device (oamg#989) - [IPU 7 -> 8] Detect corrupted grubenv file (oamg#1012) - [IPU 7 -> 8] Ensure that rsyncd stays enabled if it is enabled prior the upgrade(oamg#1043) - [IPU 7 -> 8] Ensure that satellite metapackages are installed after the upgrade (oamg#994) - [IPU 7 -> 8] Ensure the device_cio_free service stays enabled on s390x after the upgrade (oamg#977) - [IPU 7 -> 8] Fixed checks for RHEL SAP IPU 7.9 -> 8.6 (oamg#978) - [IPU 7 -> 8] Fixed migration of ntp to chrony when the ntp service is masked (oamg#966) - [IPU 7 -> 8] Prevent the traceback during migration of sendmail configuration files when the package is not installed (oamg#1041) - [IPU 7 -> 8] Satellite: reindex all related databases to fix issues due to new locales in RHEL 8 (oamg#1007, oamg#1018) - [IPU 7 -> 8] Use the correct domain name in SSSD reports (oamg#1040) - [IPU 8 -> 9] Added checks for RHEL SAP IPU 8.6 -> 9.0 (oamg#978) - [IPU 8 -> 9] CheckVDO: Ask user for the confirmation only on failures and undetermined devices (oamg#961) - [IPU 8 -> 9] Fix the kernel detection during initramfs creation for new kernel on RHEL 9.2+ (oamg#1048) - [IPU 8 -> 9] Fix the upgrade on Azure using RHUI for SAP Apps images (oamg#975) - [IPU 8 -> 9] Handle correctly firewalld version 0.8 (oamg#963) ### Enhancements - Set new upgrade paths (oamg#988): -- RHEL 7.9 -> 8.8, 8.6 (default: 8.8) -- RHEL 8.6 -> 9.0 -- RHEL 8.8 -> 9.2 - Check that used leapp data are valid and compatible with the installed leapp-repository (oamg#1003) - Detect a proxy configuration in YUM/DNF and adjust an error msg on issues caused by the configuration (oamg#914) - Detect and report systemd symlinks that are broken before the upgrade (oamg#972) - Drop obsoleted upgrade paths (oamg#1047) - Improve remediation instructions for packages in unknown repositories (oamg#1010) - Improve the error message to guide users when discovered more space is needed (oamg#956) - Introduce --nogpgcheck option to skip checking of RPM signatures (oamg#910) - Introduced an option to use an ISO file as a target RHEL version content source (oamg#979) - Introduced possibility to specify what systemd services should be enabled/disabled on the upgraded system (oamg#964) - Map the target repositories also based on the installed content (oamg#967) - Provide common information about systemd services (oamg#959) - Register subscribed systems automatically to Red Hat Insights unless --no-insights-register is used (oamg#1000) - Remove obsoleted GPG keys provided by RH after the upgrade to prevent errors (oamg#1022) - Run upgrade process with checking RPM signatures by default (oamg#910, oamg#993, oamg#1025) - Save breadcrumbs results as RHSM facts (oamg#1002) - Small improvements in various reports (oamg#1038, oamg#1039, oamg#1032) - [IPU 8 -> 9] Detect CIFS also when upgrading from RHEL8 to RHEL9 (PR1035) - [IPU 8 -> 9] Detect RoCE on IBM Z machines and check the configuration is safe for the upgrade (oamg#1030) - [IPU 8 -> 9] Enable upgrades of RHEL 8 for SAP HANA to RHEL 9 on ppc64le (oamg#1042) - [IPU 8 -> 9] Improve the handling of blocklisted certificates (oamg#992) ## Additional changes interesting for devels - Started work on bringing up networking inside the upgrade initramfs - currently available for testing and development purposes when LEAPP_DEVEL_INITRAM_NETWORK is set (oamg#960) - Add leapp debug tools to the upgrade initramfs - dracut upgrade module (oamg#997) - Enable disabling of DNF plugins in the dnfplugin library (oamg#990)

## Packaging - Requires cpio (#979) - Requires python3-gobject-base, NetworkManager-libnm (#969) - Bump leapp-repository-dependencies to 9 (#969, #979) ## Upgrade handling ### Fixes - Add leapp RHUI packages to an allowlist to drop confusing reports (#995) - Check only mounted XFS partitions (#1027) - Detect the kernel-core RPM instead of kernel to prevent an error during post-upgrade phases (#1024) - Disable the amazon-id DNF plugin on AWS during the upgrade stage to omit error messages during the upgrade process caused by missing network connection (#990) - Do not create new *pyc files when running leapp after the DNF upgrade transaction (#1017) - Enable upgrades on s390x when /boot is part of rootfs (#991) - Extend the allow list of RHUI clients by azure-sap-apps to omit confusing report (#974) - Filter out PES events unrelated for the used upgrade path and handle overlapping events (#1008) - Fix scan of ceph volumes on systems without ceph-osd (#1011) - Fix scan of ceph volumes when ceph-osd container is not found (#986) - Fix systemd symlinks that become incorrect during the IPU (#972) - Fix the check of memory (RAM) limits (#984) - Fix the upgrade of IBM Z machines configured with ZFCP (#983) - Ignore external accounts in /etc/passwd (#958) - Inhibit the upgrade when entries in /etc/fstab cause overshadowing during the upgrade (#1009) - Prevent leapp failures caused by re-run of leapp in the upgrade initramfs after previous failure, which causes additional confusing error message hiding original bugs (#996) - Prevent the upgrade with RHSM when a baseos and an appstream target repositories are not discovered (#1001) - RHUI(Azure) handle correctly various SAP images (#1037) - Rework the network configuration handling and parse the configuration data properly (#969) - Set RHSM release for non-ga and non-beta channels (#1033) - Use the "grub" library to find the GRUB device (#989) - [IPU 7 -> 8] Detect corrupted grubenv file (#1012) - [IPU 7 -> 8] Ensure that rsyncd stays enabled if it is enabled prior the upgrade(#1043) - [IPU 7 -> 8] Ensure that satellite metapackages are installed after the upgrade (#994) - [IPU 7 -> 8] Ensure the device_cio_free service stays enabled on s390x after the upgrade (#977) - [IPU 7 -> 8] Fixed checks for RHEL SAP IPU 7.9 -> 8.6 (#978) - [IPU 7 -> 8] Fixed migration of ntp to chrony when the ntp service is masked (#966) - [IPU 7 -> 8] Prevent the traceback during migration of sendmail configuration files when the package is not installed (#1041) - [IPU 7 -> 8] Satellite: reindex all related databases to fix issues due to new locales in RHEL 8 (#1007, #1018) - [IPU 7 -> 8] Use the correct domain name in SSSD reports (#1040) - [IPU 8 -> 9] Added checks for RHEL SAP IPU 8.6 -> 9.0 (#978) - [IPU 8 -> 9] CheckVDO: Ask user for the confirmation only on failures and undetermined devices (#961) - [IPU 8 -> 9] Fix the kernel detection during initramfs creation for new kernel on RHEL 9.2+ (#1048) - [IPU 8 -> 9] Fix the upgrade on Azure using RHUI for SAP Apps images (#975) - [IPU 8 -> 9] Handle correctly firewalld version 0.8 (#963) ### Enhancements - Set new upgrade paths (#988): -- RHEL 7.9 -> 8.8, 8.6 (default: 8.8) -- RHEL 8.6 -> 9.0 -- RHEL 8.8 -> 9.2 - Check that used leapp data are valid and compatible with the installed leapp-repository (#1003) - Detect a proxy configuration in YUM/DNF and adjust an error msg on issues caused by the configuration (#914) - Detect and report systemd symlinks that are broken before the upgrade (#972) - Drop obsoleted upgrade paths (#1047) - Improve remediation instructions for packages in unknown repositories (#1010) - Improve the error message to guide users when discovered more space is needed (#956) - Introduce --nogpgcheck option to skip checking of RPM signatures (#910) - Introduced an option to use an ISO file as a target RHEL version content source (#979) - Introduced possibility to specify what systemd services should be enabled/disabled on the upgraded system (#964) - Map the target repositories also based on the installed content (#967) - Provide common information about systemd services (#959) - Register subscribed systems automatically to Red Hat Insights unless --no-insights-register is used (#1000) - Remove obsoleted GPG keys provided by RH after the upgrade to prevent errors (#1022) - Run upgrade process with checking RPM signatures by default (#910, #993, #1025) - Save breadcrumbs results as RHSM facts (#1002) - Small improvements in various reports (#1038, #1039, #1032) - [IPU 8 -> 9] Detect CIFS also when upgrading from RHEL8 to RHEL9 (PR1035) - [IPU 8 -> 9] Detect RoCE on IBM Z machines and check the configuration is safe for the upgrade (#1030) - [IPU 8 -> 9] Enable upgrades of RHEL 8 for SAP HANA to RHEL 9 on ppc64le (#1042) - [IPU 8 -> 9] Improve the handling of blocklisted certificates (#992) ## Additional changes interesting for devels - Started work on bringing up networking inside the upgrade initramfs - currently available for testing and development purposes when LEAPP_DEVEL_INITRAM_NETWORK is set (#960) - Add leapp debug tools to the upgrade initramfs - dracut upgrade module (#997) - Enable disabling of DNF plugins in the dnfplugin library (#990)

MichalHe reviewed Jun 30, 2022

View reviewed changes

PeterMocary force-pushed the proxy-error branch from a3dfbb0 to 3376b19 Compare July 8, 2022 10:38

fernflower reviewed Jul 11, 2022

View reviewed changes

repos/system_upgrade/common/actors/scanpkgmanager/libraries/scanpkgmanager.py Outdated Show resolved Hide resolved

PeterMocary force-pushed the proxy-error branch from 3376b19 to c513d62 Compare July 11, 2022 15:35

Rezney assigned MichalHe Jul 14, 2022

pirat89 added this to the 7.9/8.7 milestone Jul 14, 2022

MichalHe reviewed Jul 14, 2022

View reviewed changes

repos/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py Outdated Show resolved Hide resolved

PeterMocary force-pushed the proxy-error branch from c513d62 to f851132 Compare July 18, 2022 08:04

PeterMocary force-pushed the proxy-error branch from f851132 to fe37fa2 Compare July 18, 2022 16:52

MichalHe previously approved these changes Aug 1, 2022

View reviewed changes

Rezney removed this from the 7.9/8.7 milestone Aug 15, 2022

pirat89 added this to the 8.8/9.2 milestone Aug 16, 2022

pirat89 previously requested changes Aug 16, 2022

View reviewed changes

PeterMocary added 2 commits September 12, 2022 13:31

Scanpkgmanager: detect proxy configuration

371f08c

This new information enables targetuserspacecreator actor to inform user why the package installation might have failed

Merge of the yumconfigscanner actor into the scanpkgmanager actor

237b857

PeterMocary dismissed MichalHe’s stale review via 237b857 September 12, 2022 12:21

PeterMocary force-pushed the proxy-error branch from 3d98969 to 237b857 Compare September 12, 2022 12:21

pirat89 approved these changes Sep 29, 2022

View reviewed changes

pirat89 merged commit b4c3de4 into oamg:master Sep 29, 2022

pirat89 added the changelog-checked The merger/reviewer checked the changelog draft document and updated it when relevant label Jan 23, 2023

pirat89 mentioned this pull request Feb 21, 2023

# Release 0.18.0 #1049

Merged

Proxy detection #914

Proxy detection #914

Conversation

PeterMocary commented Jun 30, 2022 • edited by MichalHe

github-actions bot commented Jun 30, 2022

Thank you for contributing to the Leapp project!

MichalHe left a comment

Choose a reason for hiding this comment

fernflower commented Jul 11, 2022

github-actions bot commented Jul 11, 2022

github-actions bot commented Jul 11, 2022

github-actions bot commented Jul 11, 2022

github-actions bot commented Jul 11, 2022

github-actions bot commented Jul 11, 2022

fernflower left a comment

Choose a reason for hiding this comment

fernflower commented Jul 13, 2022

github-actions bot commented Jul 13, 2022

github-actions bot commented Jul 13, 2022

github-actions bot commented Jul 13, 2022

github-actions bot commented Jul 13, 2022

github-actions bot commented Jul 13, 2022

MichalHe commented Jul 14, 2022 • edited

MichalHe commented Jul 18, 2022

github-actions bot commented Jul 18, 2022

github-actions bot commented Jul 18, 2022

github-actions bot commented Jul 18, 2022

github-actions bot commented Jul 18, 2022

MichalHe commented Jul 18, 2022

Performed tests

RHEL7

github-actions bot commented Jul 18, 2022

MichalHe left a comment

Choose a reason for hiding this comment

github-actions bot commented Aug 1, 2022

github-actions bot commented Aug 1, 2022

github-actions bot commented Aug 1, 2022

github-actions bot commented Aug 1, 2022

github-actions bot commented Aug 1, 2022

pirat89 left a comment

Choose a reason for hiding this comment

setting the required changes so it's not merged by mistake

pirat89 commented Sep 12, 2022

PeterMocary commented Sep 12, 2022

PeterMocary commented Sep 12, 2022

github-actions bot commented Sep 12, 2022

github-actions bot commented Sep 12, 2022

github-actions bot commented Sep 12, 2022

github-actions bot commented Sep 12, 2022

github-actions bot commented Sep 12, 2022

github-actions bot commented Sep 12, 2022

github-actions bot commented Sep 12, 2022

pirat89 left a comment

Choose a reason for hiding this comment

PeterMocary commented Jun 30, 2022 •

edited by MichalHe

MichalHe commented Jul 14, 2022 •

edited