0.8.3: Wrong paths in zfs-functions script

[mazouffre]

System information

Type	Version/Name
Distribution Name	CentOS 6
Distribution Version	6.10
Linux Kernel	4.14.x
Architecture	x86_64
ZFS Version	0.8.0-308_g94bcf6f
SPL Version	0.8.0-308_g94bcf6f

Describe the problem you're observing

With the lastest version of ZFS On Linux, the ZFS init scripts for CentOS 6 are not working anymore because /usr/local is used as prefix in zfs-functions:

cat /etc/zfs/zfs-functions | grep /local
ZFS="/usr/local/sbin/zfs"
ZED="/usr/local/sbin/zed"
ZPOOL="/usr/local/sbin/zpool"
ZPOOL_CACHE="/usr/local/etc/zfs/zpool.cache

The ZFS packages are generated with make rpm-utils . I don't know if it is related, but make rpm-utils is called after ./configure --enable-linux-builtin as we do an in-tree build.

Reverting this commit fixes the regression: 9323aad.

The log of configure:
config.log

The output of make:
make.zfs.rpm-utils.log

Before the commit, the script is generated by this:

if [ -e /etc/debian_version ]; then \
		NFS_SRV=nfs-kernel-server; \
	  else \
		NFS_SRV=nfs; \
	  fi; \
	  if [ -e /sbin/openrc-run ]; then \
		SHELL=/sbin/openrc-run; \
	  else \
		SHELL=/bin/sh; \
	  fi; \
	  /bin/sed -e 's,@bindir\@,/usr/bin,g' \
		 -e 's,@sbindir\@,/sbin,g' \
		 -e 's,@udevdir\@,/lib/udev,g' \
		 -e 's,@udevruledir\@,/lib/udev/rules.d,g' \
		 -e 's,@sysconfdir\@,/etc,g' \
		 -e 's,@initconfdir\@,/etc/sysconfig,g' \
		 -e 's,@initdir\@,/etc/init.d,g' \
		 -e 's,@runstatedir\@,/var/run,g' \
		 -e "s,@SHELL\@,$SHELL,g" \
		 -e "s,@NFS_SRV\@,$NFS_SRV,g" \
		 zfs-functions.in >'zfs-functions'; \
	  [ 'zfs-functions' = 'zfs-functions' -o 'zfs-functions' = 'zfs' ] || \
		chmod +x 'zfs-functions')

After:

make -C ../../etc/init.d zfs zfs-functions
make -C ../../etc/init.d zfs zfs-functions
...
(if [ -e /etc/debian_version ]; then \
		NFS_SRV=nfs-kernel-server; \
	  else \
		NFS_SRV=nfs; \
	  fi; \
	  if [ -e /sbin/openrc-run ]; then \
		SHELL=/sbin/openrc-run; \
	  else \
		SHELL=/bin/sh; \
	  fi; \
	  /bin/sed -e 's,@bindir\@,/usr/local/bin,g' \
		 -e 's,@sbindir\@,/usr/local/sbin,g' \
		 -e 's,@udevdir\@,/lib/udev,g' \
		 -e 's,@udevruledir\@,/lib/udev/rules.d,g' \
		 -e 's,@sysconfdir\@,/usr/local/etc,g' \
		 -e 's,@initconfdir\@,/etc/sysconfig,g' \
		 -e 's,@initdir\@,/usr/local/etc/init.d,g' \
		 -e 's,@runstatedir\@,/usr/local/var/run,g' \
		 -e "s,@SHELL\@,$SHELL,g" \
		 -e "s,@NFS_SRV\@,$NFS_SRV,g" \
		 zfs-functions.in >'zfs-functions'; \
	  [ 'zfs-functions' = 'zfs-functions' -o 'zfs-functions' = 'zfs' ] || \
		chmod +x 'zfs-functions')
(if [ -e /etc/debian_version ]; then \
		NFS_SRV=nfs-kernel-server; \
	  else \
		NFS_SRV=nfs; \
	  fi; \
	  if [ -e /sbin/openrc-run ]; then \
		SHELL=/sbin/openrc-run; \
	  else \
		SHELL=/bin/sh; \
	  fi; \
	  /bin/sed -e 's,@bindir\@,/usr/local/bin,g' \
		 -e 's,@sbindir\@,/usr/local/sbin,g' \
		 -e 's,@udevdir\@,/lib/udev,g' \
		 -e 's,@udevruledir\@,/lib/udev/rules.d,g' \
		 -e 's,@sysconfdir\@,/usr/local/etc,g' \
		 -e 's,@initconfdir\@,/etc/sysconfig,g' \
		 -e 's,@initdir\@,/usr/local/etc/init.d,g' \
		 -e 's,@runstatedir\@,/usr/local/var/run,g' \
		 -e "s,@SHELL\@,$SHELL,g" \
		 -e "s,@NFS_SRV\@,$NFS_SRV,g" \
		 zfs-functions.in >'zfs-functions'; \
	  [ 'zfs-functions' = 'zfs-functions' -o 'zfs-functions' = 'zfs' ] || \
		chmod +x 'zfs-functions')
...
	cp ../../etc/init.d/zfs-functions \
	    /tmp/zfs-build-toor-1aCjG3VQ/BUILDROOT/zfs-0.8.0-308_g94bcf6f.el6.x86_64/etc/zfs/

[rlaager]

The substitutions themselves seem fine. The question is, why is @bindir@ under /usr/local?

In config.log, line 3331, prefix is set to /usr/local. I wonder if that's related. From a quick glance, is ./configure getting called a second time for some sort of code coverage check?

[gyakovlev]

just hit this packaging 0.8.3 for gentoo and haven't figured out yet.
What I see is after make install there still a lot of compilation happens.
still digging.

[fdr]

So, after updating someone is in for a nasty surprise once they re-boot on CentOS 6. Should something be done about this, even if it's going back to 0.8.2 in the yum repository?

[rlaager]

@dmaziuk @fdr @mazouffre I'm not a RedHat guy. Can any of you explain how the RPMs are built? Are you running ./configure yourself? If so, are you passing --prefix=/usr, --prefix=/usr/local, or not passing --prefix at all?

Can you specifically test with something like:

./configure --prefix=/usr
make rpm-utils

@behlendorf Who builds the official RPMs? It sounds like they're broken too in the same way. Are they specifying --prefix=/usr?

[fdr]

Can any of you explain how the RPMs are built? Are you running ./configure yourself?

Huh? No. Just "yum install" from the zol repo as I have for years. I capture a fresh VM image once in a while and test it before putting anything important on it, and noticed reboots were busted thankfully before anything shipped.

I can try building the RPM tomorrow to see what happens, if it comes easily.

[dmaziuk]

@dmaziuk @fdr @mazouffre I'm not a RedHat guy. Can any of you explain how the RPMs are built? Are you running ./configure yourself? If so, are you passing --prefix=/usr, --prefix=/usr/local, or not passing --prefix at all?

No, that's the point of #9890: the problem is rpms in zfs-kmod repository built by ZoL team. Anyone who tracks that repo has dead systems, they just know it yet if they haven't rebooted.

The build problem is the bug to be fixed for next release, the "poison pill" rpms should've been removed from the repo right flipping now to limit the damage.

[behlendorf]

We're looking in to the problematic CentOS 6 packages and why they contain the wrong paths.

[dmaziuk]

I'm sure you are, but not pulling the rpms from the public repo in the meantime is the problem.

[eborisch]

The wrong (with '/usr/local') paths are in the /etc/zfs/zfs_functions file in zfs-0.8.3-1.el6.x86_64 from the DKMS ZoL epel/6 repository, so it's not just the zfs-kmod repo. A quick %s/\/usr\/local// in vim fixed it right up, but it is a bit of a shock to the system initially...

[eborisch]

Perhaps update the issue name to be CentOS 6 (0.8.3): Wrong paths... for people landing here looking for where their pools went.

[rlaager]

@behlendorf There are no .spec files in the tree, only .spec.in files. Therefore, you must run ./configure to even have a .spec file in the first place. Likewise, you must run ./configure to have a Makefile so you can make rpm-utils. That invocation of ./configure drives the building of zfs-functions. This ends up inside the SOURCES/zfs-0.8.3.tar.gz and is never rebuilt. Those built files should not be added to the tarball by make dist. That's the underlying bug.

I've submitted PR #9955 to fix this.

[tonyhutter]

Sorry this bug has been such a headache for everyone. I've pushed out a new zfs-0.8.3-2.el6.x86.rpm for both kmod and dkms with my fix included (#9959). There's a new 0.8.3-2 source RPM as well. All you should have to do is yum update and it will pull in the single new zfs package. Please post if you have any issues with it.

In the meantime, we'll have to figure out if we want to pull @rlaager's #9955 fix or my #9959 one into master.

[fdr]

Thanks for expediting that. I'll give it a try soon.

[eborisch]

Updated zfs-0.8.3-2.el6 package works for me. Thanks for the quick turn-around.

[fdr]

Late to the punch, but yeah, it works.

[dmaziuk]

mee too: I'm just getting around to updating the rest of our systems, upgrade from 0.8.2 now works w/o problems.

[fixapc]

Currently have the same issue with this in 2022.