skip TEST-02-UNITTESTS on ubuntu CIs??

[mrc0mmand]

systemd version the issue has been seen with

latest master

Used distribution

Ubuntu/Debian

Since #22013 was merged, a couple of issues were spotted which cause the test to fail (under autopkgtest) and also might cause the test to give inaccurate results.

# 1 As discussed in #22053 the test fails under autopkgtest (Ubuntu CI and possibly Debian soon). I'll copy my findings verbatim from the mentioned PR:

There are multiple things going "wrong":

1. debuild strips empty directories
2. debuild (or something along the way) doesn't like dangling symlinks

Now, the install_subdir() in meson explicitly ignores .gitatrributes, so the respective installed test-fstab-generator/*.expected/*.requires/ directory ends up with only a dangling symlink. And it looks like debuild doesn't like dangling symlinks so it strips them as well, leading to test-{01..11}*.expected/*.requires/ directories being empty and subsequently stripped too. All this ends up in the systemd-tests*.deb package, which is then used when running the integration tests.

The systemd-tests package then looks like this:

D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-01-dev-nfs.input
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-02-dhcp.input
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-03-dhcp6.input
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-04-nfs.input
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-05-nfs4.input
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-06-ipv4.input
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-07-ipv6.input
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-08-implicit-nfs.input
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-09-cifs.input
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-10-iscsi.input
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-11-live.input
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-12-dev-sdx.expected/initrd-root-device.target.d/50-root-device.conf
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-12-dev-sdx.expected/initrd-root-fs.target.requires/sysroot.mount
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-12-dev-sdx.expected/initrd-usr-fs.target.requires/sysroot.mount
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-12-dev-sdx.expected/sysroot.mount
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-12-dev-sdx.expected/systemd-fsck-root.service
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-12-dev-sdx.input
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-13-label.expected/initrd-root-device.target.d/50-root-device.conf
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-13-label.expected/initrd-root-fs.target.requires/sysroot.mount
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-13-label.expected/initrd-usr-fs.target.requires/sysroot.mount
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-13-label.expected/sysroot.mount
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-13-label.expected/systemd-fsck-root.service
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-13-label.input
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-14-uuid.expected/initrd-root-device.target.d/50-root-device.conf
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-14-uuid.expected/initrd-root-fs.target.requires/sysroot.mount
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-14-uuid.expected/initrd-usr-fs.target.requires/sysroot.mount
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-14-uuid.expected/sysroot.mount
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-14-uuid.expected/systemd-fsck-root.service
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-14-uuid.input
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-15-partuuid.expected/initrd-root-device.target.d/50-root-device.conf
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-15-partuuid.expected/initrd-root-fs.target.requires/sysroot.mount
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-15-partuuid.expected/initrd-usr-fs.target.requires/sysroot.mount
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-15-partuuid.expected/sysroot.mount
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-15-partuuid.expected/systemd-fsck-root.service
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-15-partuuid.input
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-16-tmpfs.expected/initrd-root-fs.target.requires/sysroot.mount
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-16-tmpfs.expected/initrd-usr-fs.target.requires/sysroot.mount
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-16-tmpfs.expected/sysroot.mount
D: Installing /usr/lib/systemd/tests/testdata/test-fstab-generator/test-16-tmpfs.input

causing the test to fail, for obvious reasons.

So far I don't know enough about Debian packaging to propose what should be the next steps though...

# 2 The symlinks are currently copied as files by meson

Meson already complains during ninja install:

...
Installing /home/fsumsal/repos/@systemd/systemd/test/test-fstab-generator/test-15-partuuid.expected/initrd-usr-fs.target.requires/sysroot.mount to /home/fsumsal/repos/@systemd/systemd/test_install/usr/lib/syst
emd/tests/testdata/test-fstab-generator/test-15-partuuid.expected/initrd-usr-fs.target.requires                                                                                                                  
Warning: trying to copy a symlink that points to a file. This will copy the file,                                                                                                                                
but this will be changed in a future version of Meson to copy the symlink as is. Please update your                                                                                                              
build definitions so that it will not break when the change happens.                                                                                                                                             
Installing /home/fsumsal/repos/@systemd/systemd/test/test-fstab-generator/test-16-tmpfs.expected/sysroot.mount to /home/fsumsal/repos/@systemd/systemd/test_install/usr/lib/systemd/tests/testdata/test-fstab-gen
erator/test-16-tmpfs.expected                                                                                                                                                                                    
Installing /home/fsumsal/repos/@systemd/systemd/test/test-fstab-generator/test-16-tmpfs.expected/initrd-root-fs.target.requires/sysroot.mount to /home/fsumsal/repos/@systemd/systemd/test_install/usr/lib/system
d/tests/testdata/test-fstab-generator/test-16-tmpfs.expected/initrd-root-fs.target.requires                                                                                                                      
Warning: trying to copy a symlink that points to a file. This will copy the file,                                                                                                                                
but this will be changed in a future version of Meson to copy the symlink as is. Please update your                                                                                                              
build definitions so that it will not break when the change happens.                                                                                                                                             
Installing /home/fsumsal/repos/@systemd/systemd/test/test-fstab-generator/test-16-tmpfs.expected/initrd-usr-fs.target.requires/sysroot.mount to /home/fsumsal/repos/@systemd/systemd/test_install/usr/lib/systemd
/tests/testdata/test-fstab-generator/test-16-tmpfs.expected/initrd-usr-fs.target.requires                                                                                                                        
Warning: trying to copy a symlink that points to a file. This will copy the file,                                                                                                                                
but this will be changed in a future version of Meson to copy the symlink as is. Please update your                                                                                                              
build definitions so that it will not break when the change happens.      
...

And the respective trees indeed differ:

source tree:

$ ls -lR test/test-fstab-generator/test-15-partuuid.expected/
test/test-fstab-generator/test-15-partuuid.expected/:
total 8
drwxrwxr-x. 1 fsumsal fsumsal  38 Jan  9 00:45 initrd-root-device.target.d
drwxrwxr-x. 1 fsumsal fsumsal  26 Jan  9 00:45 initrd-root-fs.target.requires
drwxrwxr-x. 1 fsumsal fsumsal  26 Jan  9 00:45 initrd-usr-fs.target.requires
-rw-rw-r--. 1 fsumsal fsumsal 434 Jan  9 00:45 sysroot.mount
-rw-rw-r--. 1 fsumsal fsumsal 648 Jan  9 00:45 systemd-fsck-root.service

test/test-fstab-generator/test-15-partuuid.expected/initrd-root-device.target.d:
total 4
-rw-rw-r--. 1 fsumsal fsumsal 236 Jan  9 00:45 50-root-device.conf

test/test-fstab-generator/test-15-partuuid.expected/initrd-root-fs.target.requires:
total 4
lrwxrwxrwx. 1 fsumsal fsumsal 16 Jan  9 00:45 sysroot.mount -> ../sysroot.mount

test/test-fstab-generator/test-15-partuuid.expected/initrd-usr-fs.target.requires:
total 4
lrwxrwxrwx. 1 fsumsal fsumsal 16 Jan  9 00:45 sysroot.mount -> ../sysroot.mount

installed tree:

$ ls -lR test_install/usr/lib/systemd/tests/testdata/test-fstab-generator/test-15-partuuid.expected/
test_install/usr/lib/systemd/tests/testdata/test-fstab-generator/test-15-partuuid.expected/:
total 8
drwxr-xr-x. 1 fsumsal fsumsal  38 Jan  9 00:50 initrd-root-device.target.d
drwxr-xr-x. 1 fsumsal fsumsal  26 Jan  9 00:50 initrd-root-fs.target.requires
drwxr-xr-x. 1 fsumsal fsumsal  26 Jan  9 00:50 initrd-usr-fs.target.requires
-rw-r--r--. 1 fsumsal fsumsal 434 Jan  9 00:45 sysroot.mount
-rw-r--r--. 1 fsumsal fsumsal 648 Jan  9 00:45 systemd-fsck-root.service

test_install/usr/lib/systemd/tests/testdata/test-fstab-generator/test-15-partuuid.expected/initrd-root-device.target.d:
total 4
-rw-r--r--. 1 fsumsal fsumsal 236 Jan  9 00:45 50-root-device.conf

test_install/usr/lib/systemd/tests/testdata/test-fstab-generator/test-15-partuuid.expected/initrd-root-fs.target.requires:
total 4
-rw-r--r--. 1 fsumsal fsumsal 434 Jan  9 00:45 sysroot.mount

test_install/usr/lib/systemd/tests/testdata/test-fstab-generator/test-15-partuuid.expected/initrd-usr-fs.target.requires:
total 4
-rw-r--r--. 1 fsumsal fsumsal 434 Jan  9 00:45 sysroot.mount

Not sure how much of an issue this actually is, but the difference is there.

/cc @evverx @yuwata @bluca @mbiebl

[mrc0mmand]

I guess the easiest "solution" (which would help with both of the issues) would be to tar/zip/zstd/xz/... the test-fstab-generator test directory into git and just extract it in the test script, since we already require some of the archiving tools already for journal. But it doesn't sound too elegant.

[evverx]

Meson already complains during ninja install:

Looks like it's a meson issue: mesonbuild/meson#9233

[evverx]

On second thought to judge from mesonbuild/meson#4069 (comment) looks like meson works as expected:

I guess we should emulate the old behaviour in which a dangling symlink is copied as-is, but a real symlink is followed and the file itself is copied

[evverx]

The systemd-tests package then looks like this

As far as I know that package is responsible for the "unit-tests" test there and it passes on Ubuntu CI as far as I can see. It's the "upstream" test that's broken somehow in "ubuntu ci" mode where "NO_BUILD" is apparently passed to it. It works fine on Ubuntu with just make clean run.

[evverx]

I think the easiest solution would be not to install the test and its data and just run it as a normal unit test.

[evverx]

I guess the easiest "solution" (which would help with both of the issues) would be to tar/zip/zstd/xz/... the test-fstab-generator test directory into git and just extract it in the test script

I think it would make it hard to add new testcases because that archive would have to be unpacked and packed again to do that

[evverx]

Anyway it appears install_debian_systemd is broken somehow in the sense that's it's different from whatever has to be installed by the systemd-tests package

[evverx]

@mrc0mmand this issue is Ubuntu CI specific (even though it's technically in the systemd repository) so I'm not sure what to do about it. It should be fixed but I still think the test has to be blocked with a denylist file until it's fixed.

[mrc0mmand]

Well, another relatively easy workaround would be to rsync the test data directory in the install_debian_system function to copy over missing test files.

[evverx]

I think the idea behind install_debian_systemd was to replicate the structure of the debian packages using test-functions. I'm not sure rsyncing the source tree on top of all those files copied from the debian packages would be in line with that. I'd wait for @mbiebl and @bluca. It would be nice to also figure out why the tests are run in three different modes there and somehow make it less complicated.

[evverx]

@mrc0mmand I've just reopened #22053 just in case :-)