systemd-resolved not available after network-online.target

[xnox]

Submission type

• Bug report
• Request for enhancement (RFE)

systemd version the issue has been seen with

232

Used distribution

Ubuntu

Problem

For releases using systemd-resolved (yakkety and zesty); the unit configuration does not require that the service be active before allowing systemd to reach 'network-online.target' which is a special target used to allow other units which require networking access to run.

In some cases, units which run After=network-online.target may encounter DNS failures if systemd-resolved is not yet completely active.

Solutions

Currently a distro patch has been applied to add Before=network-online.target to the Unit directives
for systemd-resolved.service.

Alternatively, maybe Install section needs to be changed to specify WantedBy=network-online.target rather than just the multi-user.target.

Imho, it is reasonable to expect DNS resolution to work once network-online.target has been reached.

In case of bug report: Steps to reproduce the problem

1. lxc launch ubuntu-daily:yakkety y1

2. lxc exec y1 -- journalctl -o short-precise
   --unit systemd-resolved --unit network-online.target

3. Check order of units; If 'Reached target Network is Online' is
   listed before 'Started Network Name Resolution', then DNS may not
   be up.

Further details at: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1673860

[mbiebl]

I don't think systemd-resolved is strictly related to network-online.target, so I fail to see the reasoning for that ordering.
It would probably make more sense to order it before nss-lookup.target (which is mapped to $named for LSB sysv init scripts)

[xnox]

True, it's not about systemd-resolved ordering w.r.t. to network-online.target; but rather about other units expecting network connectivity and assume working DNS resolution once network-online.target is reached. At the moment, those units order themself after network-online.target but that may not be sufficient as systemd-resolved can be started later than that (WantedBy=multi-user.target)

My understanding was that nss-lookup is a special target and is to be used for the LSB/SysV compat, not for systemd native things, no?!

[mbiebl]

Overloading the meaning of network-online.target this way feels wrong, especially if there is a dedicated target for name resolution.

https://www.freedesktop.org/software/systemd/man/systemd.special.html#nss-lookup.target

I don't read anywhere in the documentation that this target is supposed to be SysV/LSB compat only.

[poettering]

To some level I sympathize with this request I must say, and i figure when people order something after the network is supposed to be "online", then I figure most basic DNS should really work.

[mbiebl]

Could you elaborate then what the purpose of nss-lookup.target is?

[mbiebl]

So far we told DNS software like bind, to use

Wants=nss-lookup.target
Before=nss-lookup.target

https://codesearch.debian.net/search?q=Before%3Dnss-lookup.target

Why is systemd-resolved special in that regard?
Software which was traditionally requiring working DNS had a dependency on $named.
Again, I think overloading the meaning of network-online.target (or $network) is wrong imho. It's definition is vague enough as it is, pulling DNS into the mix makes it worse (and inconsistent to what's been used in the past)

[xnox]

@mbiebl nss-lookup.target is not part of any of the default transactions on legacy-free systems (no init scripts), so I am confused how ordering w.r.t. nss-lookup.target is expected to achieve anything.

I think another twist is that for example if one uses ifupdown or networkmanager to deliver network-online.target both usually configure dns resolution too; which is not the case for networkd+resolved combo, unless resolved is brought up earlier in the transaction.

There is nothing special about resolved. This is simply a request to start it as early as possible; rather than quite late in the mulit-user.target. Because there are performance benefits in having DNS resolution working earlier - prevents things from timing out resolutions; crashing; and restarting.

[mbiebl]

2017-03-29 17:39 GMT+02:00 Dimitri John Ledkov <notifications@github.com>:

There is nothing special about resolved. This is simply a request to start it as early as possible; rather than quite late in the mulit-user.target. Because there are performance benefits in having DNS resolution working earlier - prevents things from timing out resolutions; crashing; and restarting.

If you want to start it as early as possible, use DefaultDependencies=no and hook it up in sysinit.target. network-online.target is typically activated rather look in the boot sequence.

…

-- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth?

[Stebalien]

Is there any reason not to make systemd-resolved socket activatable? That avoid mucking around with manual dependencies entirely.

[poettering]

Why is systemd-resolved special in that regard?

resolved does two things:

1. it provides a local API via NSS and D-Bus
2. it provides kind of a local "DNS proxy" through its stub listener

nss-lookup.target is about the item 1 in the list above. When you ordered your service after that you know that you can at least talk to resolved, even if the network doesn't necessarily to permit all lookups to succeed just yet (but at least all the synthetic lookups will already work).

Ordering resolved before network-online.target has more to do with item 2 above. Note that the precise definition of "network-online.target" is not provided by us, we basically provide a whitespace that people can fill with meaning, and provide one possible implementation for it through networkd-wait-online.service, but other packages may implement it completely differently. REgardless how it is implemented though: I think it is safe to say that people would expect that DNS works for everything that it could possible work to — or to say this differently: the fact that resolved is involved or not should be mostly invisible, and if the networking works in general (and that is the vague assumption that network-online.target is about) then DNS should too, and thus the resolved DNS proxy should work, so that it doesn't break DNS when it would work if resolved was not used.

hence I think resolved should indeed use Before=network-online.target and Before=nss-lookup.target.

[poettering]

Is there any reason not to make systemd-resolved socket activatable? That avoid mucking around with manual dependencies entirely.

yeah, this would probably make sense. But it's hard now since resolved.conf has a config option for it...