Rework network configuration handling #969

lkundrak · 2022-09-26T11:32:46Z

This splits off the ifcfg & NetworkManager profile reading parts from CheckIfCfg and CheckNetworkDeprecations into separate scanners; allowing the ifcfg code to be shared.

There's a bugfix [1] at the end, depending on the IfCfg parsing consolidation.

[1] Make IfCfgScanner accept simple quoted values https://bugzilla.redhat.com/show_bug.cgi?id=2111691

github-actions · 2022-09-26T11:32:59Z

Thank you for contributing to the Leapp project!

Please note that every PR needs to comply with the Leapp Guidelines and must pass all tests in order to be mergeable.
If you want to request a review or rebuild a package in copr, you can use following commands as a comment:

review please to notify leapp developers of review request
/packit copr-build to submit a public copr build using packit

To launch regression testing public members of oamg organization can leave the following comment:

/rerun to schedule basic regression tests using this pr build and leapp*master* as artifacts
/rerun 42 to schedule basic regression tests using this pr build and leapp*PR42* as artifacts
/rerun-all to schedule all tests (including sst) using this pr build and leapp*master* as artifacts
/rerun-all 42 to schedule all tests (including sst) using this pr build and leapp*PR42* as artifacts

Please open ticket in case you experience technical problem with the CI. (RH internal only)

Note: In case there are problems with tests not being triggered automatically on new PR/commit or pending for a long time, please consider rerunning the CI by commenting leapp-ci build (might require several comments). If the problem persists, contact leapp-infra.

lkundrak · 2022-09-26T13:20:21Z

review please

leapp-bot · 2022-09-27T09:00:08Z

This PR has been linked in issue tracker (#OAMG-7676).

pirat89 · 2022-09-27T11:25:19Z

@lkundrak hi, is there a BZ for this? It seems to me familiar but not sure. if there is, please add the link into the description. we will get to the review today/tomorrow (putting on my todo)

lkundrak · 2022-09-30T09:10:07Z

@lkundrak hi, is there a BZ for this? It seems to me familiar but not sure.

Only for the last change. Instead of applying it in two different places, I ended up moving the code into a scanner actor, as (I think) you advised before.

if there is, please add the link into the description.

Done.

pirat89 · 2022-10-04T19:59:51Z

@lkundrak thanks! I am sorry for the postponed review (urgent looong bug investigation). It seems we get to review in the end of this week.

repos/system_upgrade/el8toel9/actors/checkifcfg/actor.py

repos/system_upgrade/el8toel9/actors/networkdeprecations/actor.py

repos/system_upgrade/el8toel9/actors/networkdeprecations/tests/unit_test_networkdeprecations.py

lkundrak · 2022-11-03T14:15:45Z

Pushed new version for review.

The major change is the omission of the secrets. Easy for ifcfg (where secrets are in a separate file), but for keyfiles we need to utilize libnm via GObject introspection. That won't be there on python2 platforms (no biggie, we're concerned about el8 and later only) and on UBI (used for testing, not NetworkManager there at all).

pirat89 · 2023-01-18T16:22:25Z

review in progress

pirat89 · 2023-01-18T16:22:31Z

/rerun

github-actions · 2023-01-18T16:27:06Z

Copr build succeeded: https://copr.fedorainfracloud.org/coprs/build/5244681

github-actions · 2023-01-18T16:58:22Z

Testing Farm request for RHEL-7.9-rhui/5244681 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2023-01-18T16:59:28Z

Testing Farm request for RHEL-8.6-rhui/5244681 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2023-01-18T18:52:43Z

Testing Farm request for RHEL-7.9-ZStream/5244681 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2023-01-18T18:53:47Z

Testing Farm request for RHEL-7.9-ZStream/5244681 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

...el8toel9/actors/networkmanagerconnectionscanner/libraries/networkmanagerconnectionscanner.py

This represents the legacy network configuration stored in /etc/sysconfig/network-scripts in form of ifcfg-* files (along with associated keys-, rules-, routes-, etc. files).

This scans the legacy network configuration in /etc/sysconfig/network-scripts and produces an IfCfg for each ifcfg-* file encountered (along with associated keys-, rules-, routes-, etc. files).

This represents the NetworkManager connection profiles in form of keyfiles at /etc/NetworkManager/system-connections.

This scans the NetworkManager connection profiles in form of keyfiles at /etc/NetworkManager/system-connections and produces a NetworkManagerConnection whenever for each one. This doesn't need the NetworkManager daemon to be actually running, but needs GObject introspection to be available. The reason for that is that libnm is used (via Gir) to strip the secrets. Add requirement for NetworkManager-libnm python3-gobject-base packages. Both are available for all architectures on RHEL 8 and 9. Currently require them only on RHEL 8 as they are not used in the code anywhere for RHEL 9 and they seem to be used only for upgrade RHEL 8 to RHEL 9. Bump leapp-repository-dependencies to 9

The NetworkManagerConnectionScanner requires GObject introspection and libnm to be installed. The RHEL8 UBI doesn't contain NetworkManager at all and the Python 2 platforms are too old to support GObject introspection. Let's add the packages to the f34 base so that we test the the scanner somewhere.

This actor used to scan the NetworkManager keyfiles and icfg files itself. No more!

This actor used to scan the NetworkManager ifcfg files itself. Now it uses IfCfg messages, sharing the scanning code with CheckNetworkDeprecations.

Turns out people actually use those. https://bugzilla.redhat.com/show_bug.cgi?id=2111691

pirat89

merging \o/ thanks for the contribution!

## Packaging - Requires cpio (oamg#979) - Requires python3-gobject-base, NetworkManager-libnm (oamg#969) - Bump leapp-repository-dependencies to 9 (oamg#969, oamg#979) ## Upgrade handling ### Fixes - Add leapp RHUI packages to an allowlist to drop confusing reports (oamg#995) - Check only mounted XFS partitions (oamg#1027) - Detect the kernel-core RPM instead of kernel to prevent an error during post-upgrade phases (oamg#1024) - Disable the amazon-id DNF plugin on AWS during the upgrade stage to omit error messages during the upgrade process caused by missing network connection (oamg#990) - Do not create new *pyc files when running leapp after the DNF upgrade transaction (oamg#1017) - Enable upgrades on s390x when /boot is part of rootfs (oamg#991) - Extend the allow list of RHUI clients by azure-sap-apps to omit confusing report (oamg#974) - Filter out PES events unrelated for the used upgrade path and handle overlapping events (oamg#1008) - Fix scan of ceph volumes on systems without ceph-osd (oamg#1011) - Fix scan of ceph volumes when ceph-osd container is not found (oamg#986) - Fix systemd symlinks that become incorrect during the IPU (oamg#972) - Fix the check of memory (RAM) limits (oamg#984) - Fix the upgrade of IBM Z machines configured with ZFCP (oamg#983) - Ignore external accounts in /etc/passwd (oamg#958) - Inhibit the upgrade when entries in /etc/fstab cause overshadowing during the upgrade (oamg#1009) - Prevent leapp failures caused by re-run of leapp in the upgrade initramfs after previous failure, which causes additional confusing error message hiding original bugs (oamg#996) - Prevent the upgrade with RHSM when a baseos and an appstream target repositories are not discovered (oamg#1001) - RHUI(Azure) handle correctly various SAP images (oamg#1037) - Rework the network configuration handling and parse the configuration data properly (oamg#969) - Set RHSM release for non-ga and non-beta channels (oamg#1033) - Use the "grub" library to find the GRUB device (oamg#989) - [IPU 7 -> 8] Detect corrupted grubenv file (oamg#1012) - [IPU 7 -> 8] Ensure that rsyncd stays enabled if it is enabled prior the upgrade(oamg#1043) - [IPU 7 -> 8] Ensure that satellite metapackages are installed after the upgrade (oamg#994) - [IPU 7 -> 8] Ensure the device_cio_free service stays enabled on s390x after the upgrade (oamg#977) - [IPU 7 -> 8] Fixed checks for RHEL SAP IPU 7.9 -> 8.6 (oamg#978) - [IPU 7 -> 8] Fixed migration of ntp to chrony when the ntp service is masked (oamg#966) - [IPU 7 -> 8] Prevent the traceback during migration of sendmail configuration files when the package is not installed (oamg#1041) - [IPU 7 -> 8] Satellite: reindex all related databases to fix issues due to new locales in RHEL 8 (oamg#1007, oamg#1018) - [IPU 7 -> 8] Use the correct domain name in SSSD reports (oamg#1040) - [IPU 8 -> 9] Added checks for RHEL SAP IPU 8.6 -> 9.0 (oamg#978) - [IPU 8 -> 9] CheckVDO: Ask user for the confirmation only on failures and undetermined devices (oamg#961) - [IPU 8 -> 9] Fix the kernel detection during initramfs creation for new kernel on RHEL 9.2+ (oamg#1048) - [IPU 8 -> 9] Fix the upgrade on Azure using RHUI for SAP Apps images (oamg#975) - [IPU 8 -> 9] Handle correctly firewalld version 0.8 (oamg#963) ### Enhancements - Set new upgrade paths (oamg#988): -- RHEL 7.9 -> 8.8, 8.6 (default: 8.8) -- RHEL 8.6 -> 9.0 -- RHEL 8.8 -> 9.2 - Check that used leapp data are valid and compatible with the installed leapp-repository (oamg#1003) - Detect a proxy configuration in YUM/DNF and adjust an error msg on issues caused by the configuration (oamg#914) - Detect and report systemd symlinks that are broken before the upgrade (oamg#972) - Drop obsoleted upgrade paths (oamg#1047) - Improve remediation instructions for packages in unknown repositories (oamg#1010) - Improve the error message to guide users when discovered more space is needed (oamg#956) - Introduce --nogpgcheck option to skip checking of RPM signatures (oamg#910) - Introduced an option to use an ISO file as a target RHEL version content source (oamg#979) - Introduced possibility to specify what systemd services should be enabled/disabled on the upgraded system (oamg#964) - Map the target repositories also based on the installed content (oamg#967) - Provide common information about systemd services (oamg#959) - Register subscribed systems automatically to Red Hat Insights unless --no-insights-register is used (oamg#1000) - Remove obsoleted GPG keys provided by RH after the upgrade to prevent errors (oamg#1022) - Run upgrade process with checking RPM signatures by default (oamg#910, oamg#993, oamg#1025) - Save breadcrumbs results as RHSM facts (oamg#1002) - Small improvements in various reports (oamg#1038, oamg#1039, oamg#1032) - [IPU 8 -> 9] Detect CIFS also when upgrading from RHEL8 to RHEL9 (PR1035) - [IPU 8 -> 9] Detect RoCE on IBM Z machines and check the configuration is safe for the upgrade (oamg#1030) - [IPU 8 -> 9] Enable upgrades of RHEL 8 for SAP HANA to RHEL 9 on ppc64le (oamg#1042) - [IPU 8 -> 9] Improve the handling of blocklisted certificates (oamg#992) ## Additional changes interesting for devels - Started work on bringing up networking inside the upgrade initramfs - currently available for testing and development purposes when LEAPP_DEVEL_INITRAM_NETWORK is set (oamg#960) - Add leapp debug tools to the upgrade initramfs - dracut upgrade module (oamg#997) - Enable disabling of DNF plugins in the dnfplugin library (oamg#990)

## Packaging - Requires cpio (#979) - Requires python3-gobject-base, NetworkManager-libnm (#969) - Bump leapp-repository-dependencies to 9 (#969, #979) ## Upgrade handling ### Fixes - Add leapp RHUI packages to an allowlist to drop confusing reports (#995) - Check only mounted XFS partitions (#1027) - Detect the kernel-core RPM instead of kernel to prevent an error during post-upgrade phases (#1024) - Disable the amazon-id DNF plugin on AWS during the upgrade stage to omit error messages during the upgrade process caused by missing network connection (#990) - Do not create new *pyc files when running leapp after the DNF upgrade transaction (#1017) - Enable upgrades on s390x when /boot is part of rootfs (#991) - Extend the allow list of RHUI clients by azure-sap-apps to omit confusing report (#974) - Filter out PES events unrelated for the used upgrade path and handle overlapping events (#1008) - Fix scan of ceph volumes on systems without ceph-osd (#1011) - Fix scan of ceph volumes when ceph-osd container is not found (#986) - Fix systemd symlinks that become incorrect during the IPU (#972) - Fix the check of memory (RAM) limits (#984) - Fix the upgrade of IBM Z machines configured with ZFCP (#983) - Ignore external accounts in /etc/passwd (#958) - Inhibit the upgrade when entries in /etc/fstab cause overshadowing during the upgrade (#1009) - Prevent leapp failures caused by re-run of leapp in the upgrade initramfs after previous failure, which causes additional confusing error message hiding original bugs (#996) - Prevent the upgrade with RHSM when a baseos and an appstream target repositories are not discovered (#1001) - RHUI(Azure) handle correctly various SAP images (#1037) - Rework the network configuration handling and parse the configuration data properly (#969) - Set RHSM release for non-ga and non-beta channels (#1033) - Use the "grub" library to find the GRUB device (#989) - [IPU 7 -> 8] Detect corrupted grubenv file (#1012) - [IPU 7 -> 8] Ensure that rsyncd stays enabled if it is enabled prior the upgrade(#1043) - [IPU 7 -> 8] Ensure that satellite metapackages are installed after the upgrade (#994) - [IPU 7 -> 8] Ensure the device_cio_free service stays enabled on s390x after the upgrade (#977) - [IPU 7 -> 8] Fixed checks for RHEL SAP IPU 7.9 -> 8.6 (#978) - [IPU 7 -> 8] Fixed migration of ntp to chrony when the ntp service is masked (#966) - [IPU 7 -> 8] Prevent the traceback during migration of sendmail configuration files when the package is not installed (#1041) - [IPU 7 -> 8] Satellite: reindex all related databases to fix issues due to new locales in RHEL 8 (#1007, #1018) - [IPU 7 -> 8] Use the correct domain name in SSSD reports (#1040) - [IPU 8 -> 9] Added checks for RHEL SAP IPU 8.6 -> 9.0 (#978) - [IPU 8 -> 9] CheckVDO: Ask user for the confirmation only on failures and undetermined devices (#961) - [IPU 8 -> 9] Fix the kernel detection during initramfs creation for new kernel on RHEL 9.2+ (#1048) - [IPU 8 -> 9] Fix the upgrade on Azure using RHUI for SAP Apps images (#975) - [IPU 8 -> 9] Handle correctly firewalld version 0.8 (#963) ### Enhancements - Set new upgrade paths (#988): -- RHEL 7.9 -> 8.8, 8.6 (default: 8.8) -- RHEL 8.6 -> 9.0 -- RHEL 8.8 -> 9.2 - Check that used leapp data are valid and compatible with the installed leapp-repository (#1003) - Detect a proxy configuration in YUM/DNF and adjust an error msg on issues caused by the configuration (#914) - Detect and report systemd symlinks that are broken before the upgrade (#972) - Drop obsoleted upgrade paths (#1047) - Improve remediation instructions for packages in unknown repositories (#1010) - Improve the error message to guide users when discovered more space is needed (#956) - Introduce --nogpgcheck option to skip checking of RPM signatures (#910) - Introduced an option to use an ISO file as a target RHEL version content source (#979) - Introduced possibility to specify what systemd services should be enabled/disabled on the upgraded system (#964) - Map the target repositories also based on the installed content (#967) - Provide common information about systemd services (#959) - Register subscribed systems automatically to Red Hat Insights unless --no-insights-register is used (#1000) - Remove obsoleted GPG keys provided by RH after the upgrade to prevent errors (#1022) - Run upgrade process with checking RPM signatures by default (#910, #993, #1025) - Save breadcrumbs results as RHSM facts (#1002) - Small improvements in various reports (#1038, #1039, #1032) - [IPU 8 -> 9] Detect CIFS also when upgrading from RHEL8 to RHEL9 (PR1035) - [IPU 8 -> 9] Detect RoCE on IBM Z machines and check the configuration is safe for the upgrade (#1030) - [IPU 8 -> 9] Enable upgrades of RHEL 8 for SAP HANA to RHEL 9 on ppc64le (#1042) - [IPU 8 -> 9] Improve the handling of blocklisted certificates (#992) ## Additional changes interesting for devels - Started work on bringing up networking inside the upgrade initramfs - currently available for testing and development purposes when LEAPP_DEVEL_INITRAM_NETWORK is set (#960) - Add leapp debug tools to the upgrade initramfs - dracut upgrade module (#997) - Enable disabling of DNF plugins in the dnfplugin library (#990)

lkundrak force-pushed the lr/ifcfg-nm-rework branch 3 times, most recently from 82e40fa to f38e2bf Compare September 26, 2022 12:57

pirat89 self-assigned this Oct 17, 2022

pirat89 requested changes Oct 17, 2022

View reviewed changes

repos/system_upgrade/el8toel9/actors/checkifcfg/actor.py Show resolved Hide resolved

repos/system_upgrade/el8toel9/actors/networkdeprecations/actor.py Outdated Show resolved Hide resolved

pirat89 reviewed Oct 19, 2022

View reviewed changes

repos/system_upgrade/el8toel9/actors/networkdeprecations/tests/unit_test_networkdeprecations.py Outdated Show resolved Hide resolved

pirat89 added the bug Something isn't working label Oct 26, 2022

pirat89 added this to the 8.8/9.2 milestone Oct 26, 2022

lkundrak force-pushed the lr/ifcfg-nm-rework branch 4 times, most recently from cea5d38 to b9f2a49 Compare November 3, 2022 14:07

lkundrak force-pushed the lr/ifcfg-nm-rework branch from b9f2a49 to c35b14e Compare November 3, 2022 14:29

lkundrak requested a review from pirat89 November 4, 2022 13:04

lkundrak force-pushed the lr/ifcfg-nm-rework branch from c35b14e to 0546b79 Compare January 12, 2023 15:39

pirat89 reviewed Jan 20, 2023

View reviewed changes

...el8toel9/actors/networkmanagerconnectionscanner/libraries/networkmanagerconnectionscanner.py Show resolved Hide resolved

lkundrak added 8 commits January 24, 2023 12:35

Add IfCfg model

f724cba

This represents the legacy network configuration stored in /etc/sysconfig/network-scripts in form of ifcfg-* files (along with associated keys-, rules-, routes-, etc. files).

Add IfCfgScanner actor

308500d

This scans the legacy network configuration in /etc/sysconfig/network-scripts and produces an IfCfg for each ifcfg-* file encountered (along with associated keys-, rules-, routes-, etc. files).

Add NetworkManagerConnection model

d6daaa9

This represents the NetworkManager connection profiles in form of keyfiles at /etc/NetworkManager/system-connections.

Make CheckNetworkDeprecations consume IfCfg and NetworkManagerConnection

86ed131

This actor used to scan the NetworkManager keyfiles and icfg files itself. No more!

Make CheckIfCfg consume IfCfg

7cfa924

This actor used to scan the NetworkManager ifcfg files itself. Now it uses IfCfg messages, sharing the scanning code with CheckNetworkDeprecations.

Make IfCfgScanner accept simple quoted values

bf78568

Turns out people actually use those. https://bugzilla.redhat.com/show_bug.cgi?id=2111691

pirat89 added the changelog-checked The merger/reviewer checked the changelog draft document and updated it when relevant label Jan 24, 2023

lkundrak force-pushed the lr/ifcfg-nm-rework branch 2 times, most recently from 1031276 to bf78568 Compare January 26, 2023 12:45

pirat89 approved these changes Jan 27, 2023

View reviewed changes

pirat89 merged commit aed1a9c into oamg:master Jan 27, 2023

pirat89 mentioned this pull request Feb 21, 2023

# Release 0.18.0 #1049

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rework network configuration handling #969

Rework network configuration handling #969

lkundrak commented Sep 26, 2022 •

edited

github-actions bot commented Sep 26, 2022

lkundrak commented Sep 26, 2022

leapp-bot commented Sep 27, 2022

pirat89 commented Sep 27, 2022

lkundrak commented Sep 30, 2022

pirat89 commented Oct 4, 2022

lkundrak commented Nov 3, 2022

pirat89 commented Jan 18, 2023

pirat89 commented Jan 18, 2023

github-actions bot commented Jan 18, 2023

github-actions bot commented Jan 18, 2023

github-actions bot commented Jan 18, 2023

github-actions bot commented Jan 18, 2023

github-actions bot commented Jan 18, 2023

pirat89 left a comment

Rework network configuration handling #969

Rework network configuration handling #969

Conversation

lkundrak commented Sep 26, 2022 • edited

github-actions bot commented Sep 26, 2022

Thank you for contributing to the Leapp project!

lkundrak commented Sep 26, 2022

leapp-bot commented Sep 27, 2022

pirat89 commented Sep 27, 2022

lkundrak commented Sep 30, 2022

pirat89 commented Oct 4, 2022

lkundrak commented Nov 3, 2022

pirat89 commented Jan 18, 2023

pirat89 commented Jan 18, 2023

github-actions bot commented Jan 18, 2023

github-actions bot commented Jan 18, 2023

github-actions bot commented Jan 18, 2023

github-actions bot commented Jan 18, 2023

github-actions bot commented Jan 18, 2023

pirat89 left a comment

Choose a reason for hiding this comment

lkundrak commented Sep 26, 2022 •

edited