leapp fails to create target userspace when there are dangling symlinks in /etc/pki

[evgeni]

Actual behavior
When calling leapp preupgrade on a system that has a dangling symlink in /etc/pki, LEAPP aborts with a backtrace, because shutil.copytree tries to resolve that symlink and fails if it can't.

Example traceback:

...

Process Process-417:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/site-packages/leapp/repository/actor_definition.py", line 72, in _do_run
    actor_instance.run(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/leapp/actors/__init__.py", line 335, in run
    self.process(*args)
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/actor.py", line 52, in process
    userspacegen.perform()
  File "/usr/lib/python2.7/site-packages/leapp/utils/deprecation.py", line 42, in process_wrapper
    return target_item(*args, **kwargs)
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py", line 681, in perform
    _create_target_userspace(context, indata.packages, indata.files, target_repoids)
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py", line 655, in _create_target_userspace
    _prep_repository_access(context, target_path)
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py", line 238, in _prep_repository_access
    context.copytree_from('/etc/pki', os.path.join(target_etc, 'pki'))
  File "/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/mounting.py", line 221, in copytree_from
    shutil.copytree(self.full_path(src), dst)
  File "/usr/lib64/python2.7/shutil.py", line 208, in copytree
    raise Error, errors
Error: [('/var/lib/leapp/scratch/mounts/root_/system_overlay/etc/pki/pulp/content/pulp-global-repo.ca', '/var/lib/leapp/el8userspace/etc/pki/pulp/content/pulp-global-repo.ca', "[Errno 2] No such file or directory: '/var/lib/leapp/scratch/mounts/root_/system_overlay/etc/pki/pulp/content/pulp-global-repo.ca'")]


=========================================================================================================
Actor target_userspace_creator unexpectedly terminated with exit code: 1 - Please check the above details
=========================================================================================================

Above, the problem is that the /etc/pki/pulp/content/pulp-global-repo.ca file is a dangling symlink:

# ls -alh /etc/pki/pulp/content/pulp-global-repo.ca 
lrwxrwxrwx. 1 root root 20 Oct 29  2020 /etc/pki/pulp/content/pulp-global-repo.ca -> /etc/pki/pulp/ca.crt
# ls -alh /etc/pki/pulp/ca.crt
ls: cannot access /etc/pki/pulp/ca.crt: No such file or directory

And LEAPP tries to copy all of /etc/pki (and /etc/rhsm) from the EL7 system to the EL8 target:

leapp-repository/repos/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py

Lines 231 to 232 in a098738

	context.copytree_from('/etc/pki', os.path.join(target_etc, 'pki'))
	context.copytree_from('/etc/rhsm', os.path.join(target_etc, 'rhsm'))

It does so by using shutil.copytree:

leapp-repository/repos/system_upgrade/common/libraries/mounting.py

Lines 213 to 221 in a098738

	def copytree_from(self, src, dst):
	"""
	Recursively copy an entire directory tree rooted at src. The destination directory,
	named by dst, must not already exist; it will be created as well as missing parent directories.
	The destination directory is considered to be on the current system root.
	The source directory is considered to be in the isolated environment.
	"""
	shutil.copytree(self.full_path(src), dst)

And shutil.copytree doesn't like to copy dangling symlinks (TIL!) by default:

shutil.copytree(src, dst, symlinks=False, ignore=None)
If symlinks is true, symbolic links in the source tree are represented as symbolic links in the new tree, but the metadata of the original links is NOT copied; if false or omitted, the contents and metadata of the linked files are copied to the new tree.

https://docs.python.org/2.7/library/shutil.html#shutil.copytree (same-ish also in Python 3: https://docs.python.org/3/library/shutil.html#shutil.copytree).

To Reproduce
Steps to reproduce the behavior

1. ln -s missing_target /etc/pki/broken_symlink
2. leapp preupgrade
3. See error

Expected behavior
Either copying the bad symlink is is or at least producing an actionable error instead of a backtrace.

System information (please complete the following information):

RHEL 7.9

leapp-0.13.0-1.el7_9.noarch
leapp-upgrade-el7toel8-0.15.0-6.el7_9.noarch

Additional context
This was originally reported in https://bugzilla.redhat.com/show_bug.cgi?id=2063910 as part of Satellite LEAPP upgrade testing, but the bug is not specific to Satellite setups.

[pirat89]

Thank you for the report, we are aware of the problem and it's on our roadmap, however we cannot tell when the problem will be addressed and what will be the solution. The first expactation for doing the upgrade is that the system is configured correctly - without any exceptions! From our POV, the system with dangling symlinks in /etc is kind of broken already - especially in case of /etc/pki.

I can say that printing just warning could possibly end with cryptic errors when a dangling symlinks are present in /etc on the original system (the current traceback is far away from the cryptic error from my undersanding, regarding possible problems I can imagine). From that point, I am more fan of the way to print better error msg tbh. But do not understand this as a decision - that's just my curent opinion on this problem and I react early to provide at least some insights.

[evgeni]

Yeah, I agree the system is not in a "clean" state, but honestly, which system would be after running RHEL 7 for 7 years? ;-)

For the Satellite case, we have added a cleanup step to our tooling (as that's also the one who omitted to clean it up in the first place).
For the generic case here, I do agree, printing a clear, user-actionable error here would be probably the safest thing to do.

[pirat89]

This has been covered by #1135