Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RoutingPolicyRule not applied on reconfigure #16784

Closed
vmspike opened this issue Aug 19, 2020 · 9 comments · Fixed by #16986
Closed

RoutingPolicyRule not applied on reconfigure #16784

vmspike opened this issue Aug 19, 2020 · 9 comments · Fixed by #16986
Labels
bug 🐛 Programming errors, that need preferential fixing network

Comments

@vmspike
Copy link

vmspike commented Aug 19, 2020 •
edited
Loading

systemd version the issue has been seen with

systemd 245 (245.6-1~bpo10+1)

Used distribution

Armbian

Expected behaviour you didn't see

ip rules have to be applied on reconfigure (as they do on systemctl restart systemd-networkd.service)

Unexpected behaviour you saw

ip rules not applied after networkctl reconfigure IFNAME

Steps to reproduce the problem
Create any .network config with RoutingPolicyRule, e.g.

[RoutingPolicyRule]
Table=42
Priority=42
FirewallMark=42

After boot or systemd-networkd.service restart, you see the rule.
Then either delete the rule by ip rule del pri 42 AND/OR add new [RoutingPolicyRule] to .network file - after networkctl reconfigure IFNAME OR networkctl reload you see it/them unaplied.
@yuwata yuwata added bug 🐛 Programming errors, that need preferential fixing network labels Aug 19, 2020
yuwata added a commit to yuwata/systemd that referenced this issue Aug 21, 2020
@yuwata
test-network: add a test case for reconfiguring routing policy rules
9dffe6f 
C.f. systemd#16784.
@yuwata yuwata mentioned this issue Aug 21, 2020
Merged
@yuwata
Copy link
Member

yuwata commented Aug 21, 2020

Hmm. I cannot reproduce the issue. Could you provide debugging logs of networkd? You can generate the debugging logs by creating the following drop-in config:

# /etc/systemd/system/systemd-networkd.conf.d/override.conf
[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

@vmspike
Copy link
Author

vmspike commented Aug 21, 2020 

[2020-08-21T07:47:24] 0 /root
root@01f1f150ec# date; systemctl restart systemd-networkd.service 
[2020-08-21T07:47:38]
Fri 21 Aug 2020 07:47:38 AM UTC
[2020-08-21T07:47:38] 0 /root
root@localhost# journalctl -u systemd-networkd --since '2020-08-21 07:47:00'
[2020-08-21T07:47:40]
-- Logs begin at Thu 2020-08-20 20:14:06 UTC, end at Fri 2020-08-21 07:47:40 UTC. --
Aug 21 07:47:38 localhost systemd[1]: Stopping Network Service...
Aug 21 07:47:38 localhost systemd[1]: systemd-networkd.service: Succeeded.
Aug 21 07:47:38 localhost systemd[1]: Stopped Network Service.
Aug 21 07:47:38 localhost systemd[1]: Starting Network Service...
Aug 21 07:47:38 localhost systemd-networkd[24962]: Failed to parse RPDB rule family, ignoring: AF_INETpriority=10111fwmark=1011/0
Aug 21 07:47:38 localhost systemd-networkd[24962]: Failed to parse RPDB rule family, ignoring: AF_INETpriority=10112oif=wl11
Aug 21 07:47:38 localhost systemd-networkd[24962]: Failed to parse RPDB rule family, ignoring: AF_INETpriority=10001fwmark=1000/0
Aug 21 07:47:38 localhost systemd-networkd[24962]: Failed to parse RPDB rule family, ignoring: AF_INETpriority=10113iif=wl11
Aug 21 07:47:38 localhost systemd-networkd[24962]: Failed to parse RPDB rule family, ignoring: AF_INETpriority=10002oif=en0
Aug 21 07:47:38 localhost systemd-networkd[24962]: /lib/systemd/network/42-lan0-slaves.network: No valid settings found in the [Match] section, ignoring file. To match all interfaces, add Name=
Aug 21 07:47:38 localhost systemd-networkd[24962]: lan0: netdev ready
Aug 21 07:47:38 localhost systemd-networkd[24962]: Enumeration completed
Aug 21 07:47:38 localhost systemd[1]: Started Network Service.
Aug 21 07:47:38 localhost systemd-networkd[24962]: lan0: netdev exists, using existing without changing its parameters
Aug 21 07:47:38 localhost systemd-networkd[24962]: wl11: Connected WiFi access point: somessid (00:00:00:00:00:00)
Aug 21 07:47:39 localhost systemd-networkd[24962]: wl11: DHCPv4 address 192.168.100.4/24 via 192.168.100.1
[2020-08-21T07:48:30] 0 /root
root@localhost# ip ru
[2020-08-21T07:48:33]
0:      from all lookup local 
10001:  from all fwmark 0x3e8 lookup 1000 
10002:  from all oif en0 lookup 1000 
10003:  from 192.168.100.42 lookup 1000 
10111:  from all fwmark 0x3f3 lookup 1011 
10112:  from all oif wl11 lookup 1011 
10113:  from all iif wl11 lookup 1011 
10114:  from 192.168.100.4 lookup 1011 
32766:  from all lookup main 
32767:  from all lookup default 
[2020-08-21T07:48:33] 0 /root
root@localhost# ip a
[2020-08-21T07:48:37]
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: en0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether b2:86:38:57:3a:d1 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.42/24 brd 192.168.100.255 scope global en0
       valid_lft forever preferred_lft forever
3: lan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 2e:81:04:78:87:97 brd ff:ff:ff:ff:ff:ff
    inet 192.168.40.1/24 brd 192.168.40.255 scope global lan0
       valid_lft forever preferred_lft forever
7: wl11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 12:01:f1:d2:11:eb brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.4/24 brd 192.168.100.255 scope global dynamic noprefixroute wl11
       valid_lft 86342sec preferred_lft 86342sec
[2020-08-21T07:48:37] 0 /root
root@localhost# vim /etc/systemd/network/43-wl11.network  # Added rule pri 10116 (see below)
[2020-08-21T07:48:59]
[2020-08-21T07:49:23] 0 /root
root@localhost# ip rule del pri 10113
[2020-08-21T07:49:34]
[2020-08-21T07:49:34] 0 /root
root@localhost# networkctl reconfigure wl11
[2020-08-21T07:49:58]
[2020-08-21T07:49:58] 0 /root
root@localhost# ip ru  # Both 10113 and 10116 absent
[2020-08-21T07:50:02]
0:      from all lookup local 
10001:  from all fwmark 0x3e8 lookup 1000 
10002:  from all oif en0 lookup 1000 
10003:  from 192.168.100.42 lookup 1000 
10111:  from all fwmark 0x3f3 lookup 1011 
10112:  from all oif wl11 lookup 1011 
10114:  from 192.168.100.4 lookup 1011 
32766:  from all lookup main 
32767:  from all lookup default 
[2020-08-21T07:50:02] 0 /root
root@localhost# date; systemctl restart systemd-networkd.service  # After restart all is fine, all rules present
[2020-08-21T07:50:17]
Fri 21 Aug 2020 07:50:17 AM UTC
[2020-08-21T07:50:17] 0 /root
root@localhost# ip ru
[2020-08-21T07:50:21]
0:      from all lookup local 
10001:  from all fwmark 0x3e8 lookup 1000 
10002:  from all oif en0 lookup 1000 
10003:  from 192.168.100.42 lookup 1000 
10111:  from all fwmark 0x3f3 lookup 1011 
10112:  from all oif wl11 lookup 1011 
10113:  from all iif wl11 lookup 1011 
10114:  from 192.168.100.4 lookup 1011 
10116:  from all uidrange 60011-60011 lookup 1011 
32766:  from all lookup main 
32767:  from all lookup default 
[2020-08-21T07:50:21] 0 /root
root@localhost#
[2020-08-21T07:52:30] 0 /root
root@localhost# journalctl -u systemd-networkd --since '2020-08-21 07:47:00' -o short
[2020-08-21T07:53:06]
<...see previous lines above...>
Aug 21 07:47:39 01f1f150ec systemd-networkd[24962]: wl11: DHCPv4 address 192.168.100.4/24 via 192.168.100.1
Aug 21 07:49:58 01f1f150ec systemd-networkd[24962]: wl11: Re-configuring with /etc/systemd/network/43-wl11.network
Aug 21 07:49:58 01f1f150ec systemd-networkd[24962]: wl11: DHCP lease lost
Aug 21 07:49:58 01f1f150ec systemd-networkd[24962]: wl11: DHCPv4 address 192.168.100.4/24 via 192.168.100.1
Aug 21 07:50:17 01f1f150ec systemd[1]: Stopping Network Service...
Aug 21 07:50:17 01f1f150ec systemd[1]: systemd-networkd.service: Succeeded.
Aug 21 07:50:17 01f1f150ec systemd[1]: Stopped Network Service.
Aug 21 07:50:17 01f1f150ec systemd[1]: Starting Network Service...
Aug 21 07:50:17 01f1f150ec systemd-networkd[25107]: Failed to parse RPDB rule family, ignoring: AF_INETpriority=10001fwmark=1000/0
Aug 21 07:50:17 01f1f150ec systemd-networkd[25107]: Failed to parse RPDB rule family, ignoring: AF_INETpriority=10111fwmark=1011/0
Aug 21 07:50:17 01f1f150ec systemd-networkd[25107]: Failed to parse RPDB rule family, ignoring: AF_INETpriority=10112oif=wl11
Aug 21 07:50:17 01f1f150ec systemd-networkd[25107]: Failed to parse RPDB rule family, ignoring: AF_INETpriority=10113iif=wl11
Aug 21 07:50:17 01f1f150ec systemd-networkd[25107]: Failed to parse RPDB rule family, ignoring: AF_INETpriority=10002oif=en0
Aug 21 07:50:17 01f1f150ec systemd-networkd[25107]: /lib/systemd/network/42-lan0-slaves.network: No valid settings found in the [Match] section, ignoring file. To match all interfaces, add Name=
Aug 21 07:50:17 01f1f150ec systemd-networkd[25107]: lan0: netdev ready
Aug 21 07:50:17 01f1f150ec systemd-networkd[25107]: Enumeration completed
Aug 21 07:50:17 01f1f150ec systemd[1]: Started Network Service.
Aug 21 07:50:17 01f1f150ec systemd-networkd[25107]: lan0: netdev exists, using existing without changing its parameters
Aug 21 07:50:17 01f1f150ec systemd-networkd[25107]: wl11: Connected WiFi access point: somessid (00:00:00:00:00:00)
Aug 21 07:50:18 01f1f150ec systemd-networkd[25107]: wl11: DHCPv4 address 192.168.100.4/24 via 192.168.100.1

Effective wl11 initial config:

[Match]
Name=wl11

[Network]
LinkLocalAddressing=no
IPv6AcceptRA=no
DHCP=ipv4

[DHCPv4]
RouteTable=1011
RouteMetric=1011
UseRoutes=yes
UseTimezone=no
UseMTU=yes
UseDNS=no
UseNTP=no
SendDecline=yes

# fwmark
[RoutingPolicyRule]
Table=1011
Family=ipv4
Priority=10111
FirewallMark=1011

# oif
[RoutingPolicyRule]
Table=1011
Family=ipv4
Priority=10112
OutgoingInterface=wl11

# iif
[RoutingPolicyRule]
Table=1011
Family=ipv4
Priority=10113
IncomingInterface=wl11

# user
#[RoutingPolicyRule]
#Table=1011
#Family=ipv4
#Priority=10116
#User=60011

# Default gateway in default table
[Route]
Table=default
Destination=0.0.0.0/0
Gateway=_dhcp
Metric=1011

@vmspike
Copy link
Author

vmspike commented Aug 21, 2020 •
edited
Loading

Sorry, it was not debug log. To override had to create override file in /etc/systemd/system/systemd-networkd.service.d/override.conf instead. See new debug log here

@yuwata
Copy link
Member

yuwata commented Aug 28, 2020

You use networkctl reconfigure INTERFACE, but reconfigure does not reload the edited .network file. Please use networkctl reload when you edit .network files.

With v246 and your .network file, I confirm that networkctl reload works as expected.

@yuwata yuwata added reviewed/needs-rework 🔨 PR has been reviewed and needs another round of reworks needs-reporter-feedback ❓ There's an unanswered question, the reporter needs to answer and removed bug 🐛 Programming errors, that need preferential fixing cant-reproduce reviewed/needs-rework 🔨 PR has been reviewed and needs another round of reworks labels Aug 28, 2020
@vmspike
Copy link
Author

vmspike commented Aug 28, 2020 •
edited
Loading

Ok, changes from .network file will not be applied on reconfigure but if some ip rule from "cached" .network file is absent it should be restored, right? With v245 I see it's still absent after either reconfigure or even reload, will check with v246 when have a chance.

@yuwata
Copy link
Member

yuwata commented Aug 28, 2020

Deleted rules should be restored by both reconfigure and reload at least with v246. Such a case is already tested. See 9dffe6f, and I confirm that localy. Though, I've not checked that with v245...

@vmspike
Copy link
Author

vmspike commented Aug 28, 2020

Interesting:
I've built and installed systemd 246 (246-394-gb23728e) (by make install since it's test host), tried to delete rule and reconfigure wl11 (embedded wireless device) - absent rule still unapplied as usual. After reboot wl11 disappear because kernel module was not loaded, after loading suitable module I've retried the same procedure and the rule is still absent. The same for both reconfigure or reload (even after .network file touching). Only systemctl restart systemd-networkd.service helps.

BUT when I try the same with en0 (embedded ethernet device: systemctl restart systemd-networkd.service -> del rule -> networkctl reconfigure en0) it works as expected - the rule got restored.
So the problem affect only wireless device somehow.
FYI: I use iwd as wireless backend.
Debug log on reconfigure wl11 here.

RFE: Would be nice if networkctl reload accept optional argument(s) as interface name(s) and/or .network/.netdev file(s).

@vmspike
Copy link
Author

vmspike commented Aug 28, 2020 •
edited
Loading

Hm, seems related not to wireless devices but to rule type.
E.g. when all rules deleted only one (10014) restored after reconfigure en1 or reload (after .network touching):

# systemctl restart systemd-networkd.service                          
# ip rule
0:      from all lookup local
10001:  from all fwmark 0x3e8 lookup 1000
10002:  from all oif en0 lookup 1000
10003:  from 192.168.100.42 lookup 1000
10011:  from all fwmark 0x3e9 lookup 1001
10012:  from all oif en1 lookup 1001
10013:  from all iif en1 lookup 1001
10014:  from 192.168.8.254 lookup 1001
32766:  from all lookup main
32767:  from all lookup default
# ip rule del pri 10011
# ip rule del pri 10012
# ip rule del pri 10013
# ip rule del pri 10014
# ip rule
0:      from all lookup local
10001:  from all fwmark 0x3e8 lookup 1000
10002:  from all oif en0 lookup 1000
10003:  from 192.168.100.42 lookup 1000
32766:  from all lookup main
32767:  from all lookup default
# networkctl reconfigure en1; sleep 2; ip rule
0:      from all lookup local
10001:  from all fwmark 0x3e8 lookup 1000
10002:  from all oif en0 lookup 1000
10003:  from 192.168.100.42 lookup 1000
10014:  from 192.168.8.254 lookup 1001
32766:  from all lookup main
32767:  from all lookup default

So iif, oif, and fwmark ignored, but the one with src ip address restored.

yuwata added a commit to yuwata/systemd that referenced this issue Sep 8, 2020
@yuwata
network: fix the default mask for FirewallMark=
2c4f42d 
C.f. torvalds/linux@b8964ed

Partially fixes systemd#16784.
yuwata added a commit to yuwata/systemd that referenced this issue Sep 8, 2020
@yuwata
network: also process RTM_NEWRULE or RTM_DELRULE message which does n…
35b220a 
…ot contain src and dst addresses

Fixes systemd#16784.
yuwata added a commit to yuwata/systemd that referenced this issue Sep 8, 2020
@yuwata
test-network: update tests for issue systemd#16784
0fa5c43
@yuwata yuwata mentioned this issue Sep 8, 2020
Merged
@yuwata
Copy link
Member

yuwata commented Sep 8, 2020 •
edited
Loading

Finally, I found the root of this issue. Fix is waiting in #16986.

@yuwata yuwata added bug 🐛 Programming errors, that need preferential fixing and removed needs-reporter-feedback ❓ There's an unanswered question, the reporter needs to answer labels Sep 8, 2020
yuwata added a commit to yuwata/systemd that referenced this issue Sep 8, 2020
@yuwata
network: fix the default mask for FirewallMark=
7ce0a33 
C.f. torvalds/linux@b8964ed

Partially fixes systemd#16784.
yuwata added a commit to yuwata/systemd that referenced this issue Sep 8, 2020
@yuwata
network: also process RTM_NEWRULE or RTM_DELRULE message which does n…
79df69a 
…ot contain src and dst addresses

Fixes systemd#16784.
yuwata added a commit to yuwata/systemd that referenced this issue Sep 8, 2020
@yuwata
test-network: update tests for issue systemd#16784
9c2272c
yuwata added a commit to yuwata/systemd that referenced this issue Sep 10, 2020
@yuwata
network: fix the default mask for FirewallMark=
bd1000b 
And always send FRA_FWMASK if FirewallMark= is set.

C.f. torvalds/linux@b8964ed

Partially fixes systemd#16784.
yuwata added a commit to yuwata/systemd that referenced this issue Sep 10, 2020
@yuwata
network: also process RTM_NEWRULE or RTM_DELRULE message which does n…
2102d33 
…ot contain src and dst addresses

Fixes systemd#16784.
yuwata added a commit to yuwata/systemd that referenced this issue Sep 10, 2020
@yuwata
test-network: update tests for issue systemd#16784
87adeab
ssahani pushed a commit to ssahani/systemd that referenced this issue Oct 5, 2020
@yuwata @ssahani
network: fix the default mask for FirewallMark=
40e74ea 
And always send FRA_FWMASK if FirewallMark= is set.

C.f. torvalds/linux@b8964ed

Partially fixes systemd#16784.
ssahani pushed a commit to ssahani/systemd that referenced this issue Oct 5, 2020
@yuwata @ssahani
network: also process RTM_NEWRULE or RTM_DELRULE message which does n…
9dc6b9d 
…ot contain src and dst addresses

Fixes systemd#16784.
ssahani pushed a commit to ssahani/systemd that referenced this issue Oct 5, 2020
@yuwata @ssahani
test-network: update tests for issue systemd#16784
94c6bc9
ssahani pushed a commit to ssahani/systemd that referenced this issue Oct 5, 2020
@yuwata @ssahani
network: fix the default mask for FirewallMark=
4ef3cd1 
And always send FRA_FWMASK if FirewallMark= is set.

C.f. torvalds/linux@b8964ed

Partially fixes systemd#16784.
ssahani pushed a commit to ssahani/systemd that referenced this issue Oct 5, 2020
@yuwata @ssahani
network: also process RTM_NEWRULE or RTM_DELRULE message which does n…
0a99835 
…ot contain src and dst addresses

Fixes systemd#16784.
ssahani pushed a commit to ssahani/systemd that referenced this issue Oct 5, 2020
@yuwata @ssahani
test-network: update tests for issue systemd#16784
e456bb0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug 🐛 Programming errors, that need preferential fixing network
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

network: fix routing policy rule issue yuwata/systemd
2 participants
@vmspike @yuwata