Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DHCPv6 client ignores packets with invalid bytes at the end #28183

Closed
dop251 opened this issue Jun 28, 2023 · 13 comments · Fixed by #28138
Closed

DHCPv6 client ignores packets with invalid bytes at the end #28183

dop251 opened this issue Jun 28, 2023 · 13 comments · Fixed by #28138
Labels
dhcp network not-our-bug RFE 🎁 Request for Enhancement, i.e. a feature request

Comments

@dop251
Copy link

dop251 commented Jun 28, 2023

systemd version the issue has been seen with

249.11

Used distribution

Ubuntu jammy

Linux kernel version used

5.15.0-1035-oracle

CPU architectures issue was seen on

aarch64

Component

systemd-networkd

Expected behaviour you didn't see

I'm running an OCI instance with dhcp6 client enabled. Starting from Jun 1 the IPv6 lease cannot be renewed. I'm not sure if I upgraded Ubuntu on that date or Oracle changed something on the server, still the current situation is such that I can see valid SOLICIT requests being sent, and valid REPLY messages are coming back, however they seem to be simply ignored by systemd-networkd. Nothing in the log file, even with debug enabled.

I used strace to confirm that the message is received by systemd-networkd process and indeed it is. dhclient works just fine in the same configuration, however it does not use rapid-commit.

Unexpected behaviour you saw

No response

Steps to reproduce the problem

No response

Additional program output to the terminal or log subsystem illustrating the issue

Jun 28 13:44:05 instance-20221231-0947 systemd-networkd[247378]: enp0s3: DHCPv6 client: Started in Managed mode
Jun 28 13:44:05 instance-20221231-0947 systemd-networkd[247378]: enp0s3: Acquiring DHCPv6 lease on NDisc request
Jun 28 13:44:05 instance-20221231-0947 systemd-networkd[247378]: enp0s3: Requesting route: dst: n/a, src: n/a, gw: fe80::200:17ff:feea:72a8, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 100
Jun 28 13:44:05 instance-20221231-0947 systemd-networkd[247378]: enp0s3: Requesting route: dst: 2603:c020:c008:9300::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 100
Jun 28 13:44:05 instance-20221231-0947 systemd-networkd[247378]: enp0s3: Setting NDisc routes.
Jun 28 13:44:05 instance-20221231-0947 systemd-networkd[247378]: enp0s3: State changed: configured -> configuring
Jun 28 13:44:05 instance-20221231-0947 systemd-networkd[247378]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/network1/link/_32 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=6 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Jun 28 13:44:05 instance-20221231-0947 systemd-networkd[247378]: enp0s3: link_check_ready(): dhcp4:yes ipv4ll:no dhcp6_addresses:no dhcp6_routes:no dhcp6_pd_addresses:no dhcp6_pd_routes:no ndisc_addresses:yes ndisc_routes:no
Jun 28 13:44:05 instance-20221231-0947 systemd-networkd[247378]: enp0s3: State changed: configuring -> configured
Jun 28 13:44:05 instance-20221231-0947 systemd-networkd[247378]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/network1/link/_32 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=7 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Jun 28 13:44:05 instance-20221231-0947 systemd-networkd[247378]: enp0s3: Configuring route: dst: n/a, src: n/a, gw: fe80::200:17ff:feea:72a8, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 100
Jun 28 13:44:05 instance-20221231-0947 systemd-networkd[247378]: enp0s3: Configuring route: dst: 2603:c020:c008:9300::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 100
Jun 28 13:44:05 instance-20221231-0947 systemd-networkd[247378]: enp0s3: DHCPv6 client: Sent SOLICIT
Jun 28 13:44:05 instance-20221231-0947 systemd-networkd[247378]: enp0s3: DHCPv6 client: Next retransmission in 1s
Jun 28 13:44:05 instance-20221231-0947 systemd-networkd[247378]: enp0s3: NDisc routes set.
Jun 28 13:44:05 instance-20221231-0947 systemd-networkd[247378]: enp0s3: Removing old NDisc information obtained from fe80::200:17ff:feea:72a8.
Jun 28 13:44:06 instance-20221231-0947 systemd-networkd[247378]: enp0s3: DHCPv6 client: Sent SOLICIT
Jun 28 13:44:06 instance-20221231-0947 systemd-networkd[247378]: enp0s3: DHCPv6 client: Next retransmission in 1s
Jun 28 13:44:08 instance-20221231-0947 systemd-networkd[247378]: enp0s3: DHCPv6 client: Sent SOLICIT
Jun 28 13:44:08 instance-20221231-0947 systemd-networkd[247378]: enp0s3: DHCPv6 client: Next retransmission in 3s
Jun 28 13:44:12 instance-20221231-0947 systemd-networkd[247378]: enp0s3: DHCPv6 client: Sent SOLICIT
Jun 28 13:44:12 instance-20221231-0947 systemd-networkd[247378]: enp0s3: DHCPv6 client: Next retransmission in 7s
Jun 28 13:44:16 instance-20221231-0947 systemd-networkd[247378]: enp0s3: NDISC: No RA received before link confirmation timeout
Jun 28 13:44:16 instance-20221231-0947 systemd-networkd[247378]: enp0s3: NDISC: Invoking callback for 'timeout' event.
@dop251 dop251 added the bug 🐛 Programming errors, that need preferential fixing label Jun 28, 2023
@dop251
Copy link
Author

dop251 commented Jun 28, 2023

Sample pcap file:
dhcp6.pcap.gz

@triatic
Copy link

triatic commented Jun 29, 2023

I have the same issue on OCI and can see extra logs on systemd version 252.5-2ubuntu3.

Jun 29 12:44:09 instance-2 systemd-networkd[1153]: enp0s6: DHCPv6 client: Sent Solicit
Jun 29 12:44:09 instance-2 systemd-networkd[1153]: enp0s6: DHCPv6 client: Next retransmission in 1min 52s
Jun 29 12:44:09 instance-2 systemd-networkd[1153]: enp0s6: DHCPv6 client: Failed to parse option header at offset 117 of total length 118: Bad message
Jun 29 12:44:09 instance-2 systemd-networkd[1153]: enp0s6: DHCPv6 client: Failed to process received Reply message, ignoring: Bad message

@yuwata
Copy link
Member

yuwata commented Jun 29, 2023

Already fixed by 50ee1fe.

@yuwata yuwata closed this as completed Jun 29, 2023
@yuwata yuwata added already-fixed dhcp and removed bug 🐛 Programming errors, that need preferential fixing labels Jun 29, 2023
@dop251
Copy link
Author

dop251 commented Jun 29, 2023

@yuwata, I don't think @triatic's case is the same as mine. You're suggesting that something is wrong with the response, but I don't see any issues.

Just to be sure I enabled rapid commit in dhclient (by setting send dhcp6.rapid-commit;) and it worked just fine (see the attached dhclient_rc.pcap.gz).

I still believe there is a bug in systemd-networkd.

@triatic
Copy link

triatic commented Jun 29, 2023

@dop251 why do you believe my issue is different to yours?

  • Both on Oracle Cloud
  • Both using systemd-networkd
  • Both lost DHCPv6 at around the same date

And yes, I can get an IP address with dhclient -6 too.

@yuwata
Copy link
Member

yuwata commented Jun 29, 2023

Ah, the issue should be a duplicate of the report in PR #28138. Please test the PR.

@yuwata yuwata reopened this Jun 29, 2023
@yuwata
Copy link
Member

yuwata commented Jun 29, 2023

BTW, if I understand correctly, the issue should be in the server side. Please report the issue to Oracle.

@dop251
Copy link
Author

dop251 commented Jun 29, 2023

Hm... Indeed, there is a trailing zero byte, but it's got nothing to do with rapid commit, I can see it in every message coming from the server, even when rapid commit is not used.

The PR looks like it should fix it, but I have no idea how to try it in Ubuntu 22.04.

@yuwata yuwata changed the title DHCPv6 client ignores rapid-commit responses DHCPv6 client ignores packets with invalid bytes at the end Jun 29, 2023
@dop251
Copy link
Author

dop251 commented Jun 29, 2023

I will report this to OCI support, but I have the feeling chances of them fixing it are even slimmer than getting this PR into Ubuntu 22.04 :)

@triatic sorry, I was thrown off by the lack of error messages that you're seeing, but now looking at the code I realised they had been added after 249.11. So yes, looks like you're facing the same issue.

yuwata pushed a commit to pkern/systemd that referenced this issue Jun 29, 2023
Oracle Cloud sends malformed DHCPv6 replies that have an invalid
byte at the end, which cannot be parsed as an option code.

networkd currently can cope with the invalid option (it is ignored),
but the whole packet is ignored altogether because of the additional
null at the end.

It's better to be liberal in what we accept and actually assign an
address, given that the reply contains a valid IA_NA.

Fixes systemd#28183.
@yuwata
Copy link
Member

yuwata commented Jun 29, 2023

Please request to Ubuntu to backport #28138.

@yuwata yuwata added RFE 🎁 Request for Enhancement, i.e. a feature request not-our-bug labels Jun 29, 2023
@triatic
Copy link

triatic commented Jun 29, 2023

@dop251 no problem! It is nice to see that the cause of the issue is coming to light. I spent a long time with Oracle Support discussing it, but in the end they advised me to configure the IP address statically and forget about DHCPv6 altogether. 🤷‍♂️

They posted the same advice on their forum: https://community.oracle.com/customerconnect/discussion/687610/oci-ipv6-address-not-assigned-to-oci-ubuntu-instance-by-dhcp

@dop251
Copy link
Author

dop251 commented Jul 25, 2023

Update: In a rather shocking development, Oracle have actually fixed the problem in response to my support request. Not sure if the fix is global or only applies to my instance, but you should give it a try, @triatic

@triatic
Copy link

triatic commented Jul 25, 2023

Thank you for the heads up @dop251 , I can confirm a fresh VM.Standard.A1.Flex instance with image Canonical-Ubuntu-22.04-aarch64-2023.06.30-0 comes up with an IPv6 address. 😮

Good news.

valentindavid pushed a commit to valentindavid/systemd that referenced this issue Aug 8, 2023
Oracle Cloud sends malformed DHCPv6 replies that have an invalid
byte at the end, which cannot be parsed as an option code.

networkd currently can cope with the invalid option (it is ignored),
but the whole packet is ignored altogether because of the additional
null at the end.

It's better to be liberal in what we accept and actually assign an
address, given that the reply contains a valid IA_NA.

Fixes systemd#28183.

(cherry picked from commit 81b7335)
(cherry picked from commit a11a001)
(cherry picked from commit 983f418)
nmeyerhans pushed a commit to nmeyerhans/systemd that referenced this issue Jan 21, 2024
Oracle Cloud sends malformed DHCPv6 replies that have an invalid
byte at the end, which cannot be parsed as an option code.

networkd currently can cope with the invalid option (it is ignored),
but the whole packet is ignored altogether because of the additional
null at the end.

It's better to be liberal in what we accept and actually assign an
address, given that the reply contains a valid IA_NA.

Fixes systemd#28183.

(cherry picked from commit 81b7335)
(cherry picked from commit a11a001)
yuwata pushed a commit to yuwata/systemd that referenced this issue Apr 26, 2024
Oracle Cloud sends malformed DHCPv6 replies that have an invalid
byte at the end, which cannot be parsed as an option code.

networkd currently can cope with the invalid option (it is ignored),
but the whole packet is ignored altogether because of the additional
null at the end.

It's better to be liberal in what we accept and actually assign an
address, given that the reply contains a valid IA_NA.

Fixes systemd#28183.

(cherry picked from commit 81b7335)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dhcp network not-our-bug RFE 🎁 Request for Enhancement, i.e. a feature request
Development

Successfully merging a pull request may close this issue.

3 participants