Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

systemd-resolved has issues resolving Netflix hostnames when using libidn2 #6426

Closed
devurandom opened this issue Jul 22, 2017 · 9 comments

Comments

5 participants
@devurandom
Copy link

commented Jul 22, 2017

Submission type

  • Bug report

systemd version the issue has been seen with

234

Used distribution

Gentoo

In case of bug report: Expected behaviour you didn't see

ipv6_1-cxl0-c088.1.lhr004.ix.nflxvideo.net gets resolved to 37.77.187.142 or 2a00:86c0:5:5::142.

In case of bug report: Unexpected behaviour you saw

Primarily, Netflix does not work.

Further investigations, hinted at by error messages in the debug console of Chromium and in the network debugger of Firefox, revealed that getent hosts ipv6_1-cxl0-c088.1.lhr004.ix.nflxvideo.net has no output and returns error code 2, "One or more supplied key could not be found in the database."

Looking at the logs of systemd-resolved, I see the following, which might be related:

DNSSEC validation failed for question ix.nflxvideo.net IN SOA: failed-auxiliary
DNSSEC validation failed for question lhr004.ix.nflxvideo.net IN DS: failed-auxiliary
DNSSEC validation failed for question lhr004.ix.nflxvideo.net IN SOA: failed-auxiliary
DNSSEC validation failed for question 1.lhr004.ix.nflxvideo.net IN DS: failed-auxiliary
DNSSEC validation failed for question 1.lhr004.ix.nflxvideo.net IN SOA: failed-auxiliary
DNSSEC validation failed for question ipv61-cxl0-c088.1.lhr004.ix.nflxvideo.net IN DS: failed-auxiliary
DNSSEC validation failed for question ipv61-cxl0-c088.1.lhr004.ix.nflxvideo.net IN SOA: failed-auxiliary
DNSSEC validation failed for question ipv61-cxl0-c088.1.lhr004.ix.nflxvideo.net IN AAAA: failed-auxiliary
DNSSEC validation failed for question ipv61-cxl0-c088.1.lhr004.ix.nflxvideo.net IN A: failed-auxiliary

Please note that the hostname that appears in the logs is not ipv6_1-cxl0-c088.1.lhr004.ix.nflxvideo.net but ipv61-cxl0-c088.1.lhr004.ix.nflxvideo.net, which might explain why the DNS query fails.

In case of bug report: Steps to reproduce the problem

I enabled the use of systemd-resolved:

  1. /etc/resolv.conf is a symlink to /run/systemd/resolve/resolv.conf, which also makes NetworkManager feed systemd-resolved with the DNS server information.
  2. resolve is enabled as a resolver for hosts queries in libc's nsswitch:
# grep hosts: /etc/nsswitch.conf
hosts:       files mdns_minimal [NOTFOUND=return] mymachines resolve [!UNAVAIL=return] dns myhostname

My suspicion is that this is related to building systemd with libidn2.

Any help on how to debug this further would be appreciated.

# emerge --info systemd
Portage 2.3.6 (python 2.7.13-final-42, default/linux/amd64/13.0/desktop/plasma/systemd, gcc-6.3.0, glibc-2.24-r3, 4.12.1-gentoo x86_64)
=================================================================
                         System Settings
=================================================================
System uname: Linux-4.12.1-gentoo-x86_64-AMD_A10-7800_Radeon_R7,_12_Compute_Cores_4C+8G-with-gentoo-2.4.1
KiB Mem:    14352856 total,   4036764 free
KiB Swap:          0 total,         0 free
Timestamp of repository gentoo: Sat, 22 Jul 2017 08:45:01 +0000
sh bash 4.4_p12
ld GNU ld (Gentoo 2.28 p1.2) 2.28
ccache version 3.3.4 [disabled]
app-shells/bash:          4.4_p12::gentoo
dev-java/java-config:     2.2.0-r3::gentoo
dev-lang/perl:            5.24.2::gentoo
dev-lang/python:          2.7.13-r100::sage-on-gentoo, 3.4.6::gentoo, 3.5.3::gentoo
dev-util/ccache:          3.3.4::gentoo
dev-util/cmake:           3.9.0::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.4.1::gentoo
sys-apps/sandbox:         2.10-r4::gentoo
sys-devel/autoconf:       2.13::gentoo, 2.69-r3::gentoo
sys-devel/automake:       1.11.6-r2::gentoo, 1.13.4-r1::gentoo, 1.15.1::gentoo
sys-devel/binutils:       2.28-r2::gentoo
sys-devel/gcc:            5.4.0-r3::gentoo, 6.3.0::gentoo
sys-devel/gcc-config:     1.8-r1::gentoo
sys-devel/libtool:        2.4.6-r4::gentoo
sys-devel/make:           4.2.1-r1::gentoo
sys-kernel/linux-headers: 4.10::gentoo (virtual/os-headers)
sys-libs/glibc:           2.24-r3::gentoo
Repositories:

gentoo
    location: /var/cache/portage/gentoo
    sync-type: rsync
    sync-uri: rsync://rsync.de.gentoo.org/gentoo-portage
    priority: -1000

atom
    location: /var/lib/layman/atom
    sync-type: laymansync
    sync-uri: https://github.com/elprans/atom-overlay.git
    masters: gentoo
    priority: 50

flatpak-overlay
    location: /var/lib/layman/flatpak-overlay
    sync-type: laymansync
    sync-uri: git://github.com/fosero/flatpak-overlay.git
    masters: gentoo
    priority: 50

kde
    location: /var/lib/layman/kde
    sync-type: laymansync
    sync-uri: git://anongit.gentoo.org/proj/kde.git
    masters: gentoo
    priority: 50

sage-on-gentoo
    location: /var/lib/layman/sage-on-gentoo
    sync-type: laymansync
    sync-uri: git://github.com/cschwan/sage-on-gentoo.git
    masters: gentoo science
    priority: 50

science
    location: /var/lib/layman/science
    sync-type: laymansync
    sync-uri: git://anongit.gentoo.org/proj/sci.git
    masters: gentoo
    priority: 50

steam-overlay
    location: /var/lib/layman/steam-overlay
    sync-type: laymansync
    sync-uri: https://github.com/anyc/steam-overlay.git
    masters: gentoo
    priority: 50

local
    location: /var/cache/portage/local
    masters: gentoo
    priority: 100

ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-pipe -O2 -march=bdver3"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php7.1/ext-active/ /etc/php/cgi-php7.1/ext-active/ /etc/php/cli-php7.1/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-pipe -O2 -march=bdver3"
DISTDIR="/var/cache/portage/distfiles"
EMERGE_DEFAULT_OPTS="--keep-going --nospinner --verbose-conflicts"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs buildsyspkg cgroup compressdebug config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync network-sandbox news parallel-fetch parallel-install preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://ftp.spline.inf.fu-berlin.de/mirrors/gentoo/ http://ftp-stud.hs-esslingen.de/pub/Mirrors/gentoo/ http://distfiles.gentoo.org"
LANG="en_GB.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu"
MAKEOPTS="-j3"
PKGDIR="/var/cache/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="X a52 aac aacplus aacs acl acpi alsa amd64 appindicator appstream archive audit avahi bash-completion bdplus berkdb bluetooth bluray branding bs2b bzip2 cairo caps cdda cddb cdio cdr celt chromaprint cjk clang cli colord colorio conntrack cracklib crypt cups cxx d3d9 dbus declarative dirac djvu dri dts dvb dvd dvdr egl emboss encode epub exif fam fax fbcon ffmpeg fftw firefox fish-completion fits flac fontconfig fortran fribidi gdbm geoclue geolocation gif git glamor gles gmp google googledrive gpm gstreamer gtk gtk3 harfbuzz ibus iconv icu idn imlib inotify introspection ipv6 jemalloc jpeg jpeg2k kde kipi kwallet ladspa latex lcms ldap libidn2 libinput libnotify libproxy libsecret libsoxr lua_target_lua5-2 lv2 lz4 lzma lzo mad mercurial metis mjpeg mng mobi modemmanager modplug modules mp3 mp4 mpeg mplayer mtp multilib mysql ncurses netlink networkmanager nls nptl office ogg openal opencl opencv openexr opengl openmax openmp opus pam pango pcap pch pcre pcre2 pdf phonon plasma png policykit postscript ppds prison pulseaudio python qml qt3support qt4 qt5 raw readline rtmp samba scanner schroedinger sctp sdl seccomp semantic-desktop session sparse speech speex spell ssl startup-notification svg systemd tbb tcpd teamd telepathy tga theora threads tiff timezone truetype tslib udev udisks unicode unwind upnp upnp-av upower usb v4l v4l2 vaapi vdpau vorbis vpx vulkan wavpack wayland webp widgets x264 x265 xattr xcb xcomposite xinerama xkb xml xmp xrandr xscreensaver xv xvid xz zeroconf zlib" ABI_X86="64" ALSA_CARDS="hda-intel" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_EXPERIMENTAL_FEATURES="stage" CALLIGRA_FEATURES="words sheets karbon plan" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx f16c fma3 fma4 mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 xop" ELIBC="glibc" ENLIGHTENMENT_MODULES="*" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi-64" INPUT_DEVICES="joystick libinput" KERNEL="linux" L10N="de en en-GB ar fa tr ja ko zh zh-CN zh-TW" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="de en en_GB ar fa tr" LIRC_DEVICES="devinput" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6" POSTGRES_TARGETS="postgres9_5" PYTHON_SINGLE_TARGET="python3_4" PYTHON_TARGETS="python2_7 python3_4 python3_5 pypy pypy3" RUBY_TARGETS="ruby21 ruby22" STEAMGAMES="dirt_rally dont_starve portal source_engine te120 trine2 witcher2" USERLAND="GNU" VIDEO_CARDS="amdgpu" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON

=================================================================
                        Package Settings
=================================================================

sys-apps/systemd-234-r2::gentoo was built with the following:
USE="acl audit elfutils gcrypt gnuefi idn kmod libidn2 lz4 lzma pam (policykit) seccomp ssl sysv-utils xkb -apparmor -build -cryptsetup -curl -http -importd -nat -qrcode (-selinux) -test -vanilla" ABI_X86="32 (64) (-x32)"
CFLAGS="-pipe -O2 -march=bdver3 -g"
CXXFLAGS="-pipe -O2 -march=bdver3 -g"


net-dns/libidn2-2.0.2::gentoo was built with the following:                                
USE="-static-libs" ABI_X86="32 (64) (-x32)"                                                


net-dns/libidn-1.33::gentoo was built with the following:                                  
USE="nls -doc -emacs -java -mono -static-libs" ABI_X86="32 (64) (-x32)"
@riking

This comment has been minimized.

Copy link

commented Jul 22, 2017

DNSSEC is not a hard-fail for the domain: http://dnsviz.net/d/ipv6_1-cxl0-c088.1.lhr004.ix.nflxvideo.net/dnssec/

Underscores are allowed in domain names: https://tools.ietf.org/html/rfc2181#section-11

In particular, DNS servers must not refuse to serve a zone because it contains labels that might not be acceptable to some DNS client programs.

@devurandom

This comment has been minimized.

Copy link
Author

commented Jul 22, 2017

I'd like to add that drill any ... on the hostname succeeds to present a response, which is how I arrived at the IP addresses in the original post. I.e. my local DNS resolver/proxy/cache (a regular cable router of type AVM Fritz!Box) appears to be working correctly. Hence I am unsure what is causing this.

@devurandom

This comment has been minimized.

Copy link
Author

commented Jul 22, 2017

I just rebuilt systemd without libidn2 support and am now certain that the wrong behaviour is directly related to the -Dlibidn=false -Dlibidn2=true Meson flags:

  • If systemd is built without libidn2 support, name resolution works as expected.
  • If I build it again with libidn2 support, name resolution is broken again.

On a side-node, I also replaced /etc/resolv.conf, formerly a symlink to /run/systemd/resolve/resolv.conf, with a symlink to /lib/systemd/resolv.conf as recommended by the systemd-resolved.service man-page. This had no effect on the wrong behaviour, i.e. I cannot trigger or fix the bad behaviour by changing the symlink.

@keszybz

This comment has been minimized.

Copy link
Member

commented Jul 22, 2017

Yeah, if the name has underscores, behaviour will be busted with libidn2. Shortly before v234 was released, we "downgraded" libidn2 support to "experimental", primarily because of that (#6335). You can either revert to v233, or wait until #6420 is merged. (BTW., it would be great if you could test that that PR fixes the issue for you.)

@devurandom

This comment has been minimized.

Copy link
Author

commented Jul 22, 2017

@devurandom devurandom changed the title systemd-resolved has issues resolving Netflix hostnames systemd-resolved has issues resolving Netflix hostnames when using libidn2 Jul 22, 2017

@keszybz

This comment has been minimized.

Copy link
Member

commented Jul 22, 2017

keszybz@0926f34 should make netflix work for you, so that'd be the most important part. You could also install a patched libidn2 and test with that, especially non-ascii names with underscores. You could also test the whole PR, for the other changes.

@devurandom

This comment has been minimized.

Copy link
Author

commented Jul 22, 2017

I tried keszybz/systemd@0926f34 and it works. I did not apply the other patches, since they seemed not directly related to my issue.

I did not try the libidn2 patch, since that would require more modifications to the Gentoo ebuild for libidn2 to call autoreconf. If it would be of importance, I could test it, too, but I chose to skip over that for now.

I also requested that patch to be applied in Gentoo: https://bugs.gentoo.org/show_bug.cgi?id=625970

keszybz added a commit to keszybz/systemd that referenced this issue Jul 24, 2017

resolved: make sure idn2 conversions are roundtrippable
While working on the gateway→_gateway conversion, I noticed that
libidn2 strips the leading underscore in some names.
https://gitlab.com/libidn/libidn2/issues/30 was resolved in
https://gitlab.com/libidn/libidn2/commit/05d753ea69e2308cd02436d0511f4b844071dc79,
which disabled "STD3 ASCII rules" by default, i.e. disabled stripping
of underscores. So the situation is that with previously released libidn2
versions we would get incorrect behaviour, and once new libidn2 is released,
we should be OK.

Let's implement a simple test which checks that the name survives the
roundtrip, and if it doesn't, skip IDN resolution. Under old libidn2 this will
fail in more cases, and under new libidn2 in fewer, but should be the right
thing to do also under new libidn2.

Fixes systemd#6426.
@mirabilos

This comment has been minimized.

Copy link

commented Aug 4, 2017

Underscores are allowed in domain names: https://tools.ietf.org/html/rfc2181#section-11

@riking while underscores are allowed in DNS RRs such as SRV, they are not valid in hostnames.

Hostnames (FQDNs actually) consist of one or more labels (the first of which is the hostname/nodename) separated by a period, each of which follows the following RE: [0-9A-Za-z]([-0-9A-Za-z]{0,61}[0-9A-Za-z])?

The RFC 2181 section you linked specifically refers to https://tools.ietf.org/html/rfc1123#page-79 which, in turn, refers to https://tools.ietf.org/html/rfc1123#page-13 for the hostname and label format, which takes the original definition from RFC 952 [A-Za-z]([-0-9A-Za-z]*[0-9A-Za-z])? (for a label) and extends it by permitting leading digits and longer ones, coming to the regex from the previous paragraph. Compare the Domain production in https://tools.ietf.org/html/rfc5321#section-4.1.2 as well.

Someone said systemd/libidn/… just strips out the underscore, which is also wrong; if a hostname does not match the above regex it must throw an error (invalid hostname). I did not check this because I refuse to allow systemd on my systems, but I couldn’t stand someone being wrong on the internet ☺

Note further that an FQDN must match the first PCRE in the following list, but the second if it’s actually entered into DNS (due to length limitations on the whole string):

  • ^(?=.{1,255}$)[0-9A-Za-z]([-0-9A-Za-z]{0,61}[0-9A-Za-z])?(\.[0-9A-Za-z]([-0-9A-Za-z]{0,61}[0-9A-Za-z])?)*$
  • ^(?=.{1,253}$)[0-9A-Za-z]([-0-9A-Za-z]{0,61}[0-9A-Za-z])?(\.[0-9A-Za-z]([-0-9A-Za-z]{0,61}[0-9A-Za-z])?)*$
@poettering

This comment has been minimized.

Copy link
Member

commented Aug 9, 2017

@keszybz hmm, now that #6420 got merged, this bug can be closed, right?

@devurandom devurandom closed this Aug 9, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.