Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

process using the imgcreate module, or one of the libraries used by that process, leaked a reference to the filesystem. #258

Open
maitra opened this issue Jan 22, 2023 · 15 comments

Comments

@maitra
Copy link

maitra commented Jan 22, 2023

My livecd command gives the following error: here is the latter part of the error messages:

0	basesystem-11-14.fc37.noarch
0	anaconda-install-env-deps-37.12.6-1.fc37.x86_64
0	anaconda-37.12.6-1.fc37.x86_64
/usr/bin/mandb: warning: $PATH not set
Processing manual pages under /usr/share/man...
/usr/bin/mandb: can't update index cache /var/cache/man/135228: No such file or directory
Updating index cache for path `/usr/share/man/man3'. Wait.../usr/bin/mandb: warning: /usr/share/man/man3/TimeDate.3pm.gz: whatis parse for TimeDate(3pm) failed
Updating index cache for path `/usr/share/man/man1'. Wait.../usr/bin/mandb: warning: /usr/share/man/man1/lastb.1.gz: whatis parse for lastb(1) failed
/usr/bin/mandb: can't open /usr/share/man/man1/latex-dev.1: No such file or directory
/usr/bin/mandb: warning: /usr/share/man/man1/dvilualatex-dev.1.gz: bad symlink or ROFF `.so' request
Updating index cache for path `/usr/share/man/man8'. Wait.../usr/bin/mandb: warning: /usr/share/man/man8/swapoff.8.gz: whatis parse for swapoff(8) failed
/usr/bin/mandb: warning: /usr/share/man/man8/cryptsetup-bitlkOpen.8.gz: whatis parse for cryptsetup-bitlkOpen(8) failed
/usr/bin/mandb: warning: /usr/share/man/man8/cryptsetup-create.8.gz: whatis parse for cryptsetup-create(8) failed
/usr/bin/mandb: warning: /usr/share/man/man8/cryptsetup-loopaesOpen.8.gz: whatis parse for cryptsetup-loopaesOpen(8) failed
/usr/bin/mandb: warning: /usr/share/man/man8/cryptsetup-luksErase.8.gz: whatis parse for cryptsetup-luksErase(8) failed
/usr/bin/mandb: warning: /usr/share/man/man8/cryptsetup-luksOpen.8.gz: whatis parse for cryptsetup-luksOpen(8) failed
/usr/bin/mandb: warning: /usr/share/man/man8/cryptsetup-plainOpen.8.gz: whatis parse for cryptsetup-plainOpen(8) failed
/usr/bin/mandb: warning: /usr/share/man/man8/cryptsetup-tcryptOpen.8.gz: whatis parse for cryptsetup-tcryptOpen(8) failed
Updating index cache for path `/usr/share/man/man6x'. Wait...done.
Checking for stray cats under /usr/share/man...
Checking for stray cats under /var/cache/man...
Processing manual pages under /usr/share/man/ro...
/usr/bin/mandb: can't update index cache /var/cache/man/ro/135228: No such file or directory
Updating index cache for path `/usr/share/man/ro/man8'. Wait...done.
Checking for stray cats under /usr/share/man/ro...
Checking for stray cats under /var/cache/man/ro...
Processing manual pages under /usr/share/man/de...
/usr/bin/mandb: can't update index cache /var/cache/man/de/135228: No such file or directory
Updating index cache for path `/usr/share/man/de/man8'. Wait...done.
Checking for stray cats under /usr/share/man/de...
Checking for stray cats under /var/cache/man/de...
Processing manual pages under /usr/share/man/it...
/usr/bin/mandb: can't update index cache /var/cache/man/it/135228: No such file or directory
Updating index cache for path `/usr/share/man/it/man8'. Wait...done.
Checking for stray cats under /usr/share/man/it...
Checking for stray cats under /var/cache/man/it...
Processing manual pages under /usr/share/man/hu...
/usr/bin/mandb: can't update index cache /var/cache/man/hu/135228: No such file or directory
Updating index cache for path `/usr/share/man/hu/man8'. Wait...done.
Checking for stray cats under /usr/share/man/hu...
Checking for stray cats under /var/cache/man/hu...
Processing manual pages under /usr/share/man/ko...
/usr/bin/mandb: can't update index cache /var/cache/man/ko/135228: No such file or directory
Updating index cache for path `/usr/share/man/ko/man8'. Wait.../usr/bin/mandb: warning: /usr/share/man/ko/man8/rpm.8.gz: whatis parse for rpm(8) failed
/usr/bin/mandb: warning: /usr/share/man/ko/man8/rpm2cpio.8.gz: whatis parse for rpm2cpio(8) failed
done.
Checking for stray cats under /usr/share/man/ko...
Checking for stray cats under /var/cache/man/ko...
Processing manual pages under /usr/share/man/id...
/usr/bin/mandb: can't update index cache /var/cache/man/id/135228: No such file or directory
Updating index cache for path `/usr/share/man/id/man8'. Wait...done.
Checking for stray cats under /usr/share/man/id...
Checking for stray cats under /var/cache/man/id...
Processing manual pages under /usr/share/man/da...
/usr/bin/mandb: can't update index cache /var/cache/man/da/135228: No such file or directory
Updating index cache for path `/usr/share/man/da/man8'. Wait...done.
Checking for stray cats under /usr/share/man/da...
Checking for stray cats under /var/cache/man/da...
Processing manual pages under /usr/share/man/ja...
/usr/bin/mandb: can't update index cache /var/cache/man/ja/135228: No such file or directory
Updating index cache for path `/usr/share/man/ja/man8'. Wait.../usr/bin/mandb: warning: /usr/share/man/ja/man8/rpm.8.gz: whatis parse for rpm(8) failed
/usr/bin/mandb: warning: /usr/share/man/ja/man8/rpm2cpio.8.gz: whatis parse for rpm2cpio(8) failed
/usr/bin/mandb: warning: /usr/share/man/ja/man8/rpmbuild.8.gz: whatis parse for rpmbuild(8) failed
/usr/bin/mandb: warning: /usr/share/man/ja/man8/rpmgraph.8.gz: whatis parse for rpmgraph(8) failed
done.
Checking for stray cats under /usr/share/man/ja...
Checking for stray cats under /var/cache/man/ja...
Processing manual pages under /usr/share/man/sv...
/usr/bin/mandb: can't update index cache /var/cache/man/sv/135228: No such file or directory
Updating index cache for path `/usr/share/man/sv/man8'. Wait...done.
Checking for stray cats under /usr/share/man/sv...
Checking for stray cats under /var/cache/man/sv...
Processing manual pages under /usr/share/man/sr...
/usr/bin/mandb: can't update index cache /var/cache/man/sr/135228: No such file or directory
Updating index cache for path `/usr/share/man/sr/man8'. Wait...done.
Checking for stray cats under /usr/share/man/sr...
Checking for stray cats under /var/cache/man/sr...
Processing manual pages under /usr/share/man/uk...
/usr/bin/mandb: can't update index cache /var/cache/man/uk/135228: No such file or directory
Updating index cache for path `/usr/share/man/uk/man8'. Wait...done.
Checking for stray cats under /usr/share/man/uk...
Checking for stray cats under /var/cache/man/uk...
Processing manual pages under /usr/share/man/sk...
/usr/bin/mandb: can't update index cache /var/cache/man/sk/135228: No such file or directory
Updating index cache for path `/usr/share/man/sk/man8'. Wait...done.
Checking for stray cats under /usr/share/man/sk...
Checking for stray cats under /var/cache/man/sk...
Processing manual pages under /usr/share/man/pt_BR...
/usr/bin/mandb: can't update index cache /var/cache/man/pt_BR/135228: No such file or directory
Updating index cache for path `/usr/share/man/pt_BR/man8'. Wait...done.
Checking for stray cats under /usr/share/man/pt_BR...
Checking for stray cats under /var/cache/man/pt_BR...
Processing manual pages under /usr/share/man/zh_TW...
/usr/bin/mandb: can't update index cache /var/cache/man/zh_TW/135228: No such file or directory
Updating index cache for path `/usr/share/man/zh_TW/man8'. Wait...done.
Checking for stray cats under /usr/share/man/zh_TW...
Checking for stray cats under /var/cache/man/zh_TW...
Processing manual pages under /usr/share/man/cs...
/usr/bin/mandb: can't update index cache /var/cache/man/cs/135228: No such file or directory
Updating index cache for path `/usr/share/man/cs/man8'. Wait...done.
Checking for stray cats under /usr/share/man/cs...
Checking for stray cats under /var/cache/man/cs...
Processing manual pages under /usr/share/man/es...
/usr/bin/mandb: can't update index cache /var/cache/man/es/135228: No such file or directory
Updating index cache for path `/usr/share/man/es/man8'. Wait...done.
Checking for stray cats under /usr/share/man/es...
Checking for stray cats under /var/cache/man/es...
Processing manual pages under /usr/share/man/pl...
/usr/bin/mandb: can't update index cache /var/cache/man/pl/135228: No such file or directory
Updating index cache for path `/usr/share/man/pl/man8'. Wait...done.
Checking for stray cats under /usr/share/man/pl...
Checking for stray cats under /var/cache/man/pl...
Processing manual pages under /usr/share/man/pt...
/usr/bin/mandb: can't update index cache /var/cache/man/pt/135228: No such file or directory
Updating index cache for path `/usr/share/man/pt/man8'. Wait...done.
Checking for stray cats under /usr/share/man/pt...
Checking for stray cats under /var/cache/man/pt...
Processing manual pages under /usr/share/man/zh_CN...
/usr/bin/mandb: can't update index cache /var/cache/man/zh_CN/135228: No such file or directory
Updating index cache for path `/usr/share/man/zh_CN/man8'. Wait...done.
Checking for stray cats under /usr/share/man/zh_CN...
Checking for stray cats under /var/cache/man/zh_CN...
Processing manual pages under /usr/share/man/tr...
/usr/bin/mandb: can't update index cache /var/cache/man/tr/135228: No such file or directory
Updating index cache for path `/usr/share/man/tr/man8'. Wait...done.
Checking for stray cats under /usr/share/man/tr...
Checking for stray cats under /var/cache/man/tr...
Processing manual pages under /usr/share/man/fr...
/usr/bin/mandb: can't update index cache /var/cache/man/fr/135228: No such file or directory
Updating index cache for path `/usr/share/man/fr/man8'. Wait...done.
Checking for stray cats under /usr/share/man/fr...
Checking for stray cats under /var/cache/man/fr...
Processing manual pages under /usr/share/man/nl...
/usr/bin/mandb: can't update index cache /var/cache/man/nl/135228: No such file or directory
Updating index cache for path `/usr/share/man/nl/man8'. Wait...done.
Checking for stray cats under /usr/share/man/nl...
Checking for stray cats under /var/cache/man/nl...
Processing manual pages under /usr/share/man/ca...
/usr/bin/mandb: can't update index cache /var/cache/man/ca/135228: No such file or directory
Updating index cache for path `/usr/share/man/ca/man5'. Wait...done.
Checking for stray cats under /usr/share/man/ca...
Checking for stray cats under /var/cache/man/ca...
Processing manual pages under /usr/share/man/ru...
/usr/bin/mandb: can't update index cache /var/cache/man/ru/135228: No such file or directory
Updating index cache for path `/usr/share/man/ru/man8'. Wait.../usr/bin/mandb: warning: /usr/share/man/ru/man8/rpm.8.gz: whatis parse for rpm(8) failed
/usr/bin/mandb: warning: /usr/share/man/ru/man8/rpm2cpio.8.gz: whatis parse for rpm2cpio(8) failed
done.
Checking for stray cats under /usr/share/man/ru...
Checking for stray cats under /var/cache/man/ru...
Processing manual pages under /usr/local/share/man...
/usr/bin/mandb: can't update index cache /var/cache/man/local/135228: No such file or directory
Updating index cache for path `/usr/local/share/man/man6x'. Wait...done.
Checking for stray cats under /usr/local/share/man...
Checking for stray cats under /var/cache/man/local...
125 man subdirectories contained newer manual pages.
3422 manual pages were added.
0 stray cats were added.
0 old database entries were purged.
error reading information on service network: No such file or directory
Error creating Live CD : Unable to unmount filesystem at /home/maitra/remix/tmp/imgcreate-ejc8qp73/install_root/sys. The process using the imgcreate module, or one of the libraries used by that process, leaked a reference to the filesystem. Please report a bug at <https://github.com/livecd-tools/livecd-tools/issues>.

Not sure what the bug is, or how to fix it. Happy to provide more information as needed.

@maitra maitra changed the title Error creating LiveCD on Fedora 37 process using the imgcreate module, or one of the libraries used by that process, leaked a reference to the filesystem. Jan 22, 2023
@sheynema
Copy link

I have the same issue and actively looking to what is happening. In a last %post step, I have added the call to lsof to see what are the files are left opened. Here is the list

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd-logind 838 root 6r REG 0,21 4096 17290 ./sys/devices/virtual/tty/tty0/active
gdm 1154 root 8r REG 0,21 4096 17290 ./sys/devices/virtual/tty/tty0/active

@maitra
Copy link
Author

maitra commented Jan 22, 2023

I have the same issue and actively looking to what is happening. In a last %post step, I have added the call to lsof to see what are the files are left opened. Here is the list

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME systemd-logind 838 root 6r REG 0,21 4096 17290 ./sys/devices/virtual/tty/tty0/active gdm 1154 root 8r REG 0,21 4096 17290 ./sys/devices/virtual/tty/tty0/active

Thanks, is there a workaround that you suggest?

@maitra
Copy link
Author

maitra commented Jan 24, 2023

I have verified that these issues show up even with the kickstart files for the official Fedora spins too. So, for instance, with Live Workstation, Live LXDE, Live XFCE: all of them end up at the same error. So, there is a major issue.

@sheynema
Copy link

sheynema commented Jan 24, 2023

I believe the issue is present for all OS creation with livecd-tools version 30 or better. The exact code change that produces the error is 9095966. I have written a work around in a code change in fs.py that is the following in order to have no error report by use of lazy unmounting the /sys. I do not feel it is dangerous for /sys to allow to continue since all previous livecd-tools in earlier Fedora did it that way.

--- a/imgcreate/fs.py
+++ b/imgcreate/fs.py
@@ -1036,6 +1036,8 @@
                 open(self.dest, 'a').close()
             else:
                 return
+
+        logging.info("BindChrootMount mounting %s into %s" % (self.src, self.dest))
         args = ['mount', '--bind', self.src, self.dest]
         rc = call(args)
         if rc != 0:
@@ -1065,10 +1067,15 @@
             self.mounted = False
             return
 
+        logging.info("BindChrootMount Unmounting directory %s" % self.dest)
         rc = call(['umount', self.dest])
         if rc != 0:
             call(['umount', '-l', self.dest])
-            raise MountError(umount_fail_fmt % self.dest)
+            if self.src == '/sys':
+                logging.info("Unable to unmount %s normally, source %s exception using lazy unmount" %
+                             (self.dest, self.src))
+            else:
+                raise MountError(umount_fail_fmt % self.dest)
         self.mounted = False
 
     def cleanup(self):

@maitra
Copy link
Author

maitra commented Jan 24, 2023

Thanks, interesting, i was able to spin my LiveCDs up until at least Fedora 34. (At least that is the latest spin whose copy I have: I don't recall if I made any more after that.) I did not run into this problem before then.

Lets hope that the issue is fixed. It is unclear to me how the Live ISOs that Fedora ships are made. Not using these kickstart files?

@sheynema
Copy link

My guess would be that they may use headless systems. The files opened as I reported are related to gnome and login.

@maitra
Copy link
Author

maitra commented Jan 24, 2023

Should we also open a bug report on bugzilla? There does not seem to be much going on here. The last time anything was closed here was over six months ago.

@sheynema
Copy link

Maybe. But it may be handled by the owner of this livecd-tools code. And when fixed, it may be requested to be pulled into Fedora update mechanism. For the moment, I am happy with my change to make my project progress and for sure it is valuable to have it officially fixed.

@maitra
Copy link
Author

maitra commented Jan 24, 2023

I believe the issue is present for all OS creation with livecd-tools version 30 or better. The exact code change that produces the error is 9095966. I have written a work around in a code change in fs.py that is the following in order to have no error report by use of lazy unmounting the /sys. I do not feel it is dangerous for /sys to allow to continue since all previous livecd-tools in earlier Fedora did it that way.

--- a/imgcreate/fs.py
+++ b/imgcreate/fs.py
@@ -1036,6 +1036,8 @@
                 open(self.dest, 'a').close()
             else:
                 return
+
+        logging.info("BindChrootMount mounting %s into %s" % (self.src, self.dest))
         args = ['mount', '--bind', self.src, self.dest]
         rc = call(args)
         if rc != 0:
@@ -1065,10 +1067,15 @@
             self.mounted = False
             return
 
+        logging.info("BindChrootMount Unmounting directory %s" % self.dest)
         rc = call(['umount', self.dest])
         if rc != 0:
             call(['umount', '-l', self.dest])
-            raise MountError(umount_fail_fmt % self.dest)
+            if self.src == '/sys':
+                logging.info("Unable to unmount %s normally, source %s exception using lazy unmount" %
+                             (self.dest, self.src))
+            else:
+                raise MountError(umount_fail_fmt % self.dest)
         self.mounted = False
 
     def cleanup(self):

Where do you patch this file. On the /usr/lib/python3.11/site-packages/imgcreate/fs.py of the local machine? I guess we can not do this using the kickstart file?

@sheynema
Copy link

Yes, the changed file is /usr/lib/python3.11/site-packages/imgcreate/fs.py. Be careful as if new new livecd-tools is released and update is going on you may loose this fix.

@maitra
Copy link
Author

maitra commented Feb 22, 2023

I believe the issue is present for all OS creation with livecd-tools version 30 or better. The exact code change that produces the error is 9095966. I have written a work around in a code change in fs.py that is the following in order to have no error report by use of lazy unmounting the /sys. I do not feel it is dangerous for /sys to allow to continue since all previous livecd-tools in earlier Fedora did it that way.

--- a/imgcreate/fs.py
+++ b/imgcreate/fs.py
@@ -1036,6 +1036,8 @@
                 open(self.dest, 'a').close()
             else:
                 return
+
+        logging.info("BindChrootMount mounting %s into %s" % (self.src, self.dest))
         args = ['mount', '--bind', self.src, self.dest]
         rc = call(args)
         if rc != 0:
@@ -1065,10 +1067,15 @@
             self.mounted = False
             return
 
+        logging.info("BindChrootMount Unmounting directory %s" % self.dest)
         rc = call(['umount', self.dest])
         if rc != 0:
             call(['umount', '-l', self.dest])
-            raise MountError(umount_fail_fmt % self.dest)
+            if self.src == '/sys':
+                logging.info("Unable to unmount %s normally, source %s exception using lazy unmount" %
+                             (self.dest, self.src))
+            else:
+                raise MountError(umount_fail_fmt % self.dest)
         self.mounted = False
 
     def cleanup(self):

Btw, I wanted to mention that this patch works for me too. I hope that it, or an appropriate modification, will be adopted. Thanks!

@brynnneilson
Copy link

brynnneilson commented Sep 1, 2023

I can confirm this patch worked for me too. I got the same "leaked a reference to the filesystem" error when trying to compile my own kickstart and also fedora-live-workstation.ks. For my own reference, here's my cheat sheet to get things working:

Move to directory to apply patch
cd /usr/lib/python3.11/site-packages/imgcreate/

Backup file
cp fs.py fs.py.bak

Create patch - add content of patch above
nano my.patch

Apply patch. When asked for file name, enter: fs.py
patch -p1 < my.patch

Then you should be able to compile your kickstart with something like:
livecd-creator -v -c /usr/share/spin-kickstarts/fedora-live-workstation.ks -f fedora-workstation-$releasev-x86_64 ;

@ekner
Copy link

ekner commented Oct 26, 2023

After some debugging it seems I've found the issue (the "leaking reference").

In imgcreate/fs.py, e.g. on row 1065, the code uses the function os.path.ismount to detect if there is a mount, and if there is, it will perform the unmount. However, as apparently also described in the documentation, os.path.ismount "is not able to reliably detect bind mounts on the same filesystem".

In my case it was that when the script were to unmount the /sys/fs/selinux bind mount, os.path.ismount incorrectly reported it as not mounted, and therefore the unmont was skipped. So when /sys was to be unmounted afterwards, it failed because of "Target is Busy".

Basically, os.path.ismount must be replaced with some other code. Below is a sample patch that replaces os.path.ismount with a simple call to mountpoint, which gets rid of the issue for me (which can be used for testing that we have the same issue).

echo '-- /usr/lib/python3.9/site-packages/imgcreate/fs.py
+++ /usr/lib/python3.9/site-packages/imgcreate/fs.py
@@ -35,6 +35,11 @@
 from imgcreate.util import *
 from imgcreate.errors import *
 
+def ismount(path):
+    res = subprocess.run(["mountpoint", str(path)], capture_output=True)
+    return res.returncode == 0
+os.path.ismount = ismount
+
 umount_fail_fmt = ("Unable to unmount filesystem at %s. The process using the "
                    "imgcreate module, or one of the libraries used by that "
                    "process, leaked a reference to the filesystem. Please "' | patch -d / -p0

@araujorm
Copy link

araujorm commented Feb 23, 2024

I confirm this bug, a major issue rendering the tool unusable without the aforementioned workarounds.

@FGrose
Copy link
Contributor

FGrose commented Feb 24, 2024

What happens if we remove the faulty test altogether for BindChrootMount()?

diff --git a/imgcreate/fs.py b/imgcreate/fs.py
index b144e4e..7c781f6 100644
--- a/imgcreate/fs.py
+++ b/imgcreate/fs.py
@@ -1050,7 +1050,7 @@ class BindChrootMount():
         self.mounted = True
 
     def remount(self, ops):
-        if not self.mounted or not os.path.ismount(self.dest):
+        if not self.mounted:
             self.mounted = False
             return
 
@@ -1062,7 +1062,7 @@ class BindChrootMount():
                              (remount_ops, self.src, self.dest))
 
     def unmount(self):
-        if not self.mounted or not os.path.ismount(self.dest):
+        if not self.mounted:
             self.mounted = False
             return

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants