Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix false positive non-utf symlinks reported #1052

Merged
merged 4 commits into from
Apr 26, 2023

Conversation

fernflower
Copy link
Member

@fernflower fernflower commented Mar 8, 2023

Because of botched up check for python2 valid utf symlinks were reported as non-utf ones.

OAMG-8629

Relates on oamg/leapp#821

@github-actions
Copy link

github-actions bot commented Mar 8, 2023

Thank you for contributing to the Leapp project!

Please note that every PR needs to comply with the Leapp Guidelines and must pass all tests in order to be mergeable.
If you want to request a review or rebuild a package in copr, you can use following commands as a comment:

  • review please to notify leapp developers of review request
  • /packit copr-build to submit a public copr build using packit

To launch regression testing public members of oamg organization can leave the following comment:

  • /rerun to schedule basic regression tests using this pr build and leapp*master* as artifacts
  • /rerun 42 to schedule basic regression tests using this pr build and leapp*PR42* as artifacts
  • /rerun-sst to schedule sst tests using this pr build and leapp*master* as artifacts
  • /rerun-sst 42 to schedule sst tests using this pr build and leapp*PR42* as artifacts

Please open ticket in case you experience technical problem with the CI. (RH internal only)

Note: In case there are problems with tests not being triggered automatically on new PR/commit or pending for a long time, please consider rerunning the CI by commenting leapp-ci build (might require several comments). If the problem persists, contact leapp-infra.

@fernflower fernflower force-pushed the fixfalsepositivenonutf8 branch 2 times, most recently from 0b5306d to 0c114db Compare March 8, 2023 12:43
@fernflower
Copy link
Member Author

/rerun

@github-actions
Copy link

github-actions bot commented Mar 8, 2023

Copr build succeeded: https://copr.fedorainfracloud.org/coprs/build/5614910

@github-actions
Copy link

github-actions bot commented Mar 8, 2023

Testing Farm request for RHEL-8.6-rhui/5614910 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

@github-actions
Copy link

github-actions bot commented Mar 8, 2023

Testing Farm request for RHEL-7.9-rhui/5614910 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

@github-actions
Copy link

github-actions bot commented Mar 8, 2023

Testing Farm request for RHEL-8.6.0-Nightly/5614910 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

@github-actions
Copy link

github-actions bot commented Mar 8, 2023

Testing Farm request for RHEL-8.8.0-Nightly/5614910 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

@github-actions
Copy link

github-actions bot commented Mar 8, 2023

Testing Farm request for RHEL-7.9-ZStream/5614910 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

@pirat89
Copy link
Member

pirat89 commented Mar 9, 2023

@fernflower As the previous tests has not caught the problem before, is the particular test that discovered the issue now part of the executed integration tests for the PR? If so, do you know the name? Just tried to search for "root" i could not find anything related. From my understanding, such a test exists, so if it's not executed now, possibly it could be added to the set? Otherwise it would be nice to prepare unit-test so it's covered. But missing tests is not blocker from my POV for the merge right now. Can be tested manually still. The fix seems reasonable to me, haven't tested it yet.

@pirat89 pirat89 added the bug Something isn't working label Mar 9, 2023
@fernflower
Copy link
Member Author

@pirat89 Yeah, there is a test, it's MR199 to the integration tests in gitlab. We just decided not to merge the MR before the leapp change gets in not to cause extra failures.

@fernflower
Copy link
Member Author

/rerun

@pirat89 pirat89 added this to the 8.9/9.3 milestone Mar 14, 2023
@github-actions
Copy link

Copr build succeeded: https://copr.fedorainfracloud.org/coprs/build/5637134

@github-actions
Copy link

Testing Farm request for RHEL-8.6-rhui/5637134 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

@github-actions
Copy link

Testing Farm request for RHEL-7.9-rhui/5637134 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

@github-actions
Copy link

Testing Farm request for RHEL-8.6.0-Nightly/5637134 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

@github-actions
Copy link

Testing Farm request for RHEL-8.8.0-Nightly/5637134 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

@github-actions
Copy link

Testing Farm request for RHEL-7.9-ZStream/5637134 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

@github-actions
Copy link

Testing Farm request for RHEL-7.9-ZStream/5637134 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

@fernflower
Copy link
Member Author

/packit build

Copy link
Member

@pirat89 pirat89 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the current implementation leads to traceback on rhel8:

# mkdir /root/břeťa
# ln -s /root/břeťa/ /bobošík
# leapp preupgrade --debug
...
2023-03-23 13:27:17.925 INFO     PID: 26891 leapp.workflow: Workflow finished due to the until-phase flag
2023-03-23 13:27:17.932 INFO     PID: 26891 leapp: Answerfile will be created at /var/log/leapp/answerfile
Traceback (most recent call last):
  File "/usr/bin/leapp", line 9, in <module>
    load_entry_point('leapp==0.15.1', 'console_scripts', 'leapp')()
  File "/usr/lib/python2.7/site-packages/leapp/cli/__init__.py", line 45, in main
    cli.command.execute('leapp version {}'.format(VERSION))
  File "/usr/lib/python2.7/site-packages/leapp/utils/clicmd.py", line 111, in execute
    args.func(args)
  File "/usr/lib/python2.7/site-packages/leapp/utils/clicmd.py", line 133, in called
    self.target(args)
  File "/usr/lib/python2.7/site-packages/leapp/cli/commands/upgrade/breadcrumbs.py", line 163, in wrapper
    return f(*args, breadcrumbs=breadcrumbs, **kwargs)
  File "/usr/lib/python2.7/site-packages/leapp/cli/commands/preupgrade/__init__.py", line 78, in preupgrade
    util.generate_report_files(context, report_schema)
  File "/usr/lib/python2.7/site-packages/leapp/cli/commands/upgrade/util.py", line 148, in generate_report_files
    generate_report_file(messages, context, report_txt, report_schema)
  File "/usr/lib/python2.7/site-packages/leapp/utils/report.py", line 121, in generate_report_file
    f.write('Remediation: {}\n'.format(remediation))
  File "/usr/lib/python2.7/site-packages/leapp/reporting/__init__.py", line 283, in __repr__
    return "\n".join([repr(r) for r in self._remediations])
  File "/usr/lib/python2.7/site-packages/leapp/reporting/__init__.py", line 248, in __repr__
    return "[{}] {}".format(self._value['type'], ' '.join(self._value['context']))
UnicodeEncodeError: 'ascii' codec can't encode character u'\u0159' in position 20: ordinal not in range(128)

Because of botched up check for python2 valid utf symlinks were reported
as non-utf ones.

OAMG-8629
Also introduce tests for the nonutf symlinks
@github-actions
Copy link

Copr build succeeded: https://copr.fedorainfracloud.org/coprs/build/5782399

@github-actions
Copy link

Testing Farm request for RHEL-7.9-rhui/5782399 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

@github-actions
Copy link

Testing Farm request for RHEL-8.6-rhui/5782399 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

@github-actions
Copy link

Testing Farm request for RHEL-8.8.0-Nightly/5782399 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

@github-actions
Copy link

Testing Farm request for RHEL-8.6.0-Nightly/5782399 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

@github-actions
Copy link

Testing Farm request for RHEL-7.9-ZStream/5782399 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

@github-actions
Copy link

Testing Farm request for RHEL-7.9-ZStream/5782399 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

@fernflower
Copy link
Member Author

/packit build

@fernflower
Copy link
Member Author

/packit test

@fernflower
Copy link
Member Author

/rerun 821

@github-actions
Copy link

Copr build succeeded: https://copr.fedorainfracloud.org/coprs/build/5830593

@github-actions
Copy link

Copr build succeeded: https://copr.fedorainfracloud.org/coprs/build/5830974

@github-actions
Copy link

Testing Farm request for RHEL-8.6-rhui/5830974;5830593 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

@github-actions
Copy link

Testing Farm request for RHEL-7.9-rhui/5830974;5830593 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

@github-actions
Copy link

Testing Farm request for RHEL-8.8.0-Nightly/5830974;5830593 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

@github-actions
Copy link

Testing Farm request for RHEL-8.6.0-Nightly/5830974;5830593 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

@github-actions
Copy link

Testing Farm request for RHEL-7.9-ZStream/5830974;5830593 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

@github-actions
Copy link

Testing Farm request for RHEL-7.9-ZStream/5830974;5830593 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

@fernflower fernflower requested a review from pirat89 April 25, 2023 09:18
@fernflower
Copy link
Member Author

/packit test oamg/leapp#821

Copy link
Member

@pirat89 pirat89 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix in framework has been merged in upstream. Manually tested, seems it works as it should. Merging \o/

Note: the failed tests seems to be unrelated to this fix. Checking the failures, it seems as issue in tests.

@pirat89 pirat89 added the changelog-checked The merger/reviewer checked the changelog draft document and updated it when relevant label Apr 26, 2023
@pirat89 pirat89 merged commit d1a7170 into oamg:master Apr 26, 2023
pirat89 added a commit to pirat89/leapp-repository that referenced this pull request Aug 23, 2023
## Packaging
- Requires leapp-framework 5.0

## Upgrade handling
### Fixes
- Add el8toel9 actor to handle directory -> symlink with ruby IRB. (oamg#1076)
- Do not try to update GRUB core on IBM Z systems (oamg#1117)
- Fix failing upgrades with devtmpfs file systems specified in FSTAB (oamg#1090)
- Fix the calculation of the required free space on each partitions/volume for the upgrade transactions (oamg#1097)
- Fix the generation of the report about hybrid images (oamg#1064)
- Handle correctly the installed certificates to allow upgrades with custom repositories using HTTPs with enabled SSL verification (oamg#1106)
- Minor improvements and fixes of various reports (oamg#1066, oamg#1067, oamg#1085)
- Update error messages about leapp data files to inform user how to obtain valid data files (oamg#1121)
- Update links in various reports (oamg#1062, oamg#1086)
- Update the repomap data to cover changed repoids in RHUI Azure (oamg#1087)
- [IPU 7 -> 8] Fix false positive report about invalid symlinks on RHEL 7 (oamg#1052)
- [IPU 8 -> 9] Inhibit the upgrade when unsupported x86-64 microarchitecture is detected (oamg#1059)

### Enhancements
- Include updated leapp data files in the RPM (oamg#1046, oamg#1092, oamg#1119)
- Update the set of supported upgrade paths (oamg#1077):
  - RHEL with SAP HANA 7.9 -> 8.6, 8.8 (default: 8.6)
  - RHEL with SAP HANA 8.8 -> 9.2
- Introduce new upgrade paths:
  - RHEL 7.9 -> 8.9 (default)
  - RHEL 8.9 -> 9.3
- Correctly update grub2 when /boot resides on multiple devices aggregated in RAID (oamg#1093, oamg#1115)
- Enable upgrades for machines using RHUI on AlibabaCloud (oamg#1088)
- Introduce possibility to add kernel drivers to initramfs (oamg#1081)
- Redesign handling of information about kernel (booted and target) in preparation for new changes in RHEL 9 (oamg#1107)
- Redesign source system overlay to use disk images backed by sparse files to optimize disk space consumption (oamg#1097, oamg#1103)
- Requires leapp-framework 5.0 (oamg#1061, oamg#1116)
- Use new leapp CLI API which provides better report summary output (oamg#1061, oamg#1116)
- [IPU 8 -> 9] Detect and report use of deprecated Xorg drivers (oamg#1078)
- [IPU 8 -> 9] Introduce IPU for systems with FIPS enabled (oamg#1053)

## Additional changes interesting for devels
- Deprecated `GrubInfo.orig_device_name` field in the `GrubInfo` model (replaced by `GrubInfo.orig_devices`) (oamg#1093)
- Deprecated `InstalledTargetKernelVersion` model (replaced by `InstalledTargetKernelInfo`) (oamg#1107)
- Deprecated `leapp.libraries.common.config.version.is_rhel_realtime` (check the type in msg `KernelInfo`, field `type`) (oamg#1107)
- Deprecated `leapp.libraries.common.grub.get_grub_device()` (replaced by `leapp.libraries.common.grub.get_grub_devices()`) (oamg#1093)
- Introduced new devel envar LEAPP_DEVEL_KEEP_DISK_IMGS=1 to skip the removal of the created disk images for OVL. That's sometimes handy for the debugging. (oamg#1097)
@pirat89 pirat89 mentioned this pull request Aug 23, 2023
Rezney pushed a commit that referenced this pull request Aug 23, 2023
## Packaging
- Requires leapp-framework 5.0

## Upgrade handling
### Fixes
- Add el8toel9 actor to handle directory -> symlink with ruby IRB. (#1076)
- Do not try to update GRUB core on IBM Z systems (#1117)
- Fix failing upgrades with devtmpfs file systems specified in FSTAB (#1090)
- Fix the calculation of the required free space on each partitions/volume for the upgrade transactions (#1097)
- Fix the generation of the report about hybrid images (#1064)
- Handle correctly the installed certificates to allow upgrades with custom repositories using HTTPs with enabled SSL verification (#1106)
- Minor improvements and fixes of various reports (#1066, #1067, #1085)
- Update error messages about leapp data files to inform user how to obtain valid data files (#1121)
- Update links in various reports (#1062, #1086)
- Update the repomap data to cover changed repoids in RHUI Azure (#1087)
- [IPU 7 -> 8] Fix false positive report about invalid symlinks on RHEL 7 (#1052)
- [IPU 8 -> 9] Inhibit the upgrade when unsupported x86-64 microarchitecture is detected (#1059)

### Enhancements
- Include updated leapp data files in the RPM (#1046, #1092, #1119)
- Update the set of supported upgrade paths (#1077):
  - RHEL with SAP HANA 7.9 -> 8.6, 8.8 (default: 8.6)
  - RHEL with SAP HANA 8.8 -> 9.2
- Introduce new upgrade paths:
  - RHEL 7.9 -> 8.9 (default)
  - RHEL 8.9 -> 9.3
- Correctly update grub2 when /boot resides on multiple devices aggregated in RAID (#1093, #1115)
- Enable upgrades for machines using RHUI on AlibabaCloud (#1088)
- Introduce possibility to add kernel drivers to initramfs (#1081)
- Redesign handling of information about kernel (booted and target) in preparation for new changes in RHEL 9 (#1107)
- Redesign source system overlay to use disk images backed by sparse files to optimize disk space consumption (#1097, #1103)
- Requires leapp-framework 5.0 (#1061, #1116)
- Use new leapp CLI API which provides better report summary output (#1061, #1116)
- [IPU 8 -> 9] Detect and report use of deprecated Xorg drivers (#1078)
- [IPU 8 -> 9] Introduce IPU for systems with FIPS enabled (#1053)

## Additional changes interesting for devels
- Deprecated `GrubInfo.orig_device_name` field in the `GrubInfo` model (replaced by `GrubInfo.orig_devices`) (#1093)
- Deprecated `InstalledTargetKernelVersion` model (replaced by `InstalledTargetKernelInfo`) (#1107)
- Deprecated `leapp.libraries.common.config.version.is_rhel_realtime` (check the type in msg `KernelInfo`, field `type`) (#1107)
- Deprecated `leapp.libraries.common.grub.get_grub_device()` (replaced by `leapp.libraries.common.grub.get_grub_devices()`) (#1093)
- Introduced new devel envar LEAPP_DEVEL_KEEP_DISK_IMGS=1 to skip the removal of the created disk images for OVL. That's sometimes handy for the debugging. (#1097)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working changelog-checked The merger/reviewer checked the changelog draft document and updated it when relevant
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants