cockpit tests fail in mock with nspawn and default require_networking = False

[deryni]

We are building cockpit RPMs using mock. This worked with mock 1.3.4.

I upgraded to mock 1.4.2 today which enabled systemd-nspawn chroot-ing by default (it had been optional before).

This change appears to have broken 12 of the cockpit tests.

PASS: test-webserver 1 /web-server/table
**
cockpit-protocol:ERROR:src/common/test-webserver.c:57:setup: assertion failed: (inet != NULL)
./test-webserver terminated with SIGABRT
FAIL: test-webserver 2 /web-server/query-string
**
cockpit-protocol:ERROR:src/common/test-webserver.c:57:setup: assertion failed: (inet != NULL)
**
cockpit-protocol:ERROR:src/common/test-webserver.c:57:setup: assertion failed: (inet != NULL)
FAIL: test-webserver 3 /web-server/host-header
FAIL: test-webserver 4 /web-server/not-found
**
cockpit-protocol:ERROR:src/common/test-webserver.c:57:setup: assertion failed: (inet != NULL)
FAIL: test-webserver 5 /web-server/redirect-notls
**
cockpit-protocol:ERROR:src/common/test-webserver.c:57:setup: assertion failed: (inet != NULL)
FAIL: test-webserver 6 /web-server/no-redirect-localhost
**
cockpit-protocol:ERROR:src/common/test-webserver.c:57:setup: assertion failed: (inet != NULL)
FAIL: test-webserver 7 /web-server/no-redirect-exception
**
cockpit-protocol:ERROR:src/common/test-webserver.c:57:setup: assertion failed: (inet != NULL)
FAIL: test-webserver 8 /web-server/no-redirect-override
**
cockpit-protocol:ERROR:src/common/test-webserver.c:57:setup: assertion failed: (inet != NULL)
FAIL: test-webserver 9 /web-server/handle-resource
**
cockpit-protocol:ERROR:src/common/test-webserver.c:57:setup: assertion failed: (inet != NULL)
FAIL: test-webserver 10 /web-server/url-root
**
cockpit-protocol:ERROR:src/common/test-webserver.c:57:setup: assertion failed: (inet != NULL)
FAIL: test-webserver 11 /web-server/url-root-handlers
**
cockpit-protocol:ERROR:src/common/test-webserver.c:57:setup: assertion failed: (inet != NULL)
FAIL: test-webserver 12 /web-server/local-address-only
**
cockpit-protocol:ERROR:src/common/test-webserver.c:57:setup: assertion failed: (inet != NULL)
FAIL: test-webserver 13 /web-server/inet-address-only
PASS: test-webserver 14 /web-server/bad-address

I spoke to @petervo in #cockpit about this earlier:

petervo: do you have systemd-nspawn configured to add any sort of networking
deryni-work: I haven't configured it at all.
petervo: right, i'd have to investigate further what the exact commands would need to be
petervo: but it needs a non loopback interface to listen on
petervo: if you open a ticket on github, one of should be able to track down the right incantation to use to make it work

Ideally the container wouldn't have any external network connectivity when this was sorted out.

I poked at some of the systemd-nspawn options but short of setting up an actual bridge or veth with networking I didn't see anything obvious (possibly systemd-networkd in the container but I don't know anywhere near enough about nspawn/networkd/etc. to know if that would help or how I would make that happen offhand).

[stefwalter]

Could you provide a full reproducer set of commands? From a stock Fedora system, until you see the error?

[deryni]

I believe this set of steps will reproduce it (culled from my repro attempt which had a few other bits to it that weren't relevant).

1. Install fresh Fedora 25 machine.
2. dnf install mock
3. useradd -G mock mockuser
4. curl -L -O https://github.com/cockpit-project/cockpit/releases/download/146/cockpit-146.tar.xz
5. tar -xOf cockpit-146.tar.xz cockpit-146/tools/cockpit.spec > cockpit.spec
6. sed -i '/^Version:/s/0$/146/' cockpit.spec
7. mock -r fedora-26-x86_64 --buildsrpm --sources cockpit-146.tar.xz --spec cockpit.spec --resultdir srpm
8. mock -r fedora-26-x86_64 --resultdir rpm srpm/cockpit-146-1.fc26.src.rpm

[petervo]

@deryni, try

echo "config_opts['rpmbuild_networking'] = True" >>/etc/mock/site-defaults.cfg

Does doing that make it work?

[martinpitt]

This (most likely) got fixed as a by-product in PR #7594. Fedora and even RHEL 7.4 now enable nspawn by default in mock (on our test images too), and building seems to work fine there in master (and the latest release 150). Can you confirm this?

[martinpitt]

Closing as this seems fixed. Please yell if you still have trouble with recent versions.

[deryni]

No, this is still not working for me.
I wasn't seeing whatever tar errors were happening in the logs referenced from #7705. My errors were networking related.

The build referenced in the other issue does not build with networking disabled (the nspawn command line does not have --private-network on the command line).

That indicates, to me, that setting the rpmbuild_networking flag on my end should work. I was just hoping not to have to do that.

[deryni]

I do think, though, that some upcoming mock changes (1.4.6) might make this work without that flag but I need to wait for EPEL to package it to try that.

[martinpitt]

@deryni: I haven't tried on Fedora 25, but it does work fine on current Fedora 26/27 (mock 1.4.6) and RHEL 7.4 (mock 1.4.4). On all of these we use the default mock configuration, i. e. with rpmbuild_networking == False and use_nspawn == True. Can you be more specific about which cockpit release you are building on which OS?

[deryni]

@martinpitt If you look in your build logs do you see the --private-network flag being passed to systemd-nspawn when building? I don't see that being passed in the log referenced earlier (where the build fails for a different reason than I was seeing) and I do see it being passed in my build where the build continues to fail.

Unless I'm wildly misreading the mock code (for 1.4.4 here and 1.4.6 here) you will see that being passed if rpmbuild_networking is set to False in the config.

And actually... the log referenced above even shows that rpmbuild_networking = True is explicitly being set in the log (a few dozen lines up from the bottom of the log). (As do the few other logs I bothered to check in the https://fedorapeople.org/groups/cockpit/logs/ directory... not that I understand what those logs actually mean though.)

I'm building cockpit 150 currently (with only slight spec patches akin to what the CentOS package does) on a stock CentOS 7 machine (with a slightly customized mock config).

[martinpitt]

Ah, mea culpa - I only checked the mock configuration on our test images, which do have the default mock configuration. But indeed rpmbuild_networking gets enabled at runtime when building cockpit in our test VM.

As these tests require a non-loopback interface, pretty much the only thing that we can do there is to skip them if there is no network interface. But at least it's clear now why that happens, sorry for the noise!

[deryni]

Not a problem. Sorry I missed the config setting when looking at the logs in the first place. It would have saved us both a bit of time.

The tests require a non-loopback interface? So you think the changes in mock 1.4.6 won't help then? That's unfortunate.

Right now I'm still building cockpit with the old chroot instead but I should probably switch to nspawn with networking enabled instead.

[martinpitt]

Duplicate of #11312, fixed in PR #11314