Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[18.7] "There were errors(s) loading the rules: no IP address found for openvpn:network" #2625

Closed
sjorge opened this issue Aug 11, 2018 · 33 comments
Assignees
Labels
cleanup Low impact changes
Milestone

Comments

@sjorge
Copy link
Contributor

sjorge commented Aug 11, 2018

After boot I get "There were errors(s) loading the rules: no IP address found for openvpn:network"

I did not get this during any of the RC (or I did not notice), I do not get this if I reload the rules once the box is booted. I assume something changed order wise where OpenVPN gets started later?

Not a super big problem for me as I just need to ssh in and reload before OpenVPN works. More of an annoyance.

@fichtner fichtner self-assigned this Aug 11, 2018
@fichtner fichtner added the cleanup Low impact changes label Aug 11, 2018
@fichtner
Copy link
Member

can you grep for me the openvpn:network rules in /tmp/rules.debug ? something similar reported in the forum. it's likely part of runtime vs. compile time pf.conf handling...

@fichtner fichtner added this to the 19.1 milestone Aug 11, 2018
@sjorge
Copy link
Contributor Author

sjorge commented Aug 11, 2018

sjorge@hydrogen:~ % grep openvpn:network /tmp/rules.debug
nat on openvpn inet6 proto udp from openvpn:network to $zeus_vlan40 port {123} -> openvpn port 1024:65535 # Redirect NTP to zeus (IPv6)
nat on openvpn inet proto udp from openvpn:network to $zeus_vlan40 port {123} -> openvpn port 1024:65535 # Redirect NTP to zeus (IPv4)
pass in quick on openvpn inet proto {tcp udp} from {(openvpn:network)} to $hydrogen_vlan10 port {53} keep state label "USER_RULE: Allow DNS" # 34134d2ce71d197e4847e0e0433dde17
pass in quick on openvpn inet6 proto {tcp udp} from {(openvpn:network)} to $hydrogen_vlan10 port {53} keep state label "USER_RULE: Allow DNS" # 34134d2ce71d197e4847e0e0433dde17
pass in quick on openvpn inet proto tcp from {(openvpn:network)} to {any} port {22} keep state label "USER_RULE: Allow SSH" # a9ce9537173cab2d43e38d2d9e2ea06f
pass in quick on openvpn inet6 proto tcp from {(openvpn:network)} to {any} port {22} keep state label "USER_RULE: Allow SSH" # a9ce9537173cab2d43e38d2d9e2ea06f
pass in quick on openvpn inet proto udp from {(openvpn:network)} to {(vtnet4:network)} port {623} keep state label "USER_RULE: Allow IPMI" # 481c06e52063e9d93d1189162954b2c1
pass in quick on openvpn inet6 proto udp from {(openvpn:network)} to {(vtnet4:network)} port {623} keep state label "USER_RULE: Allow IPMI" # 481c06e52063e9d93d1189162954b2c1
pass in quick on openvpn inet proto tcp from {(openvpn:network)} to {any} port {25} keep state label "USER_RULE: Allow SMTP" # bf9ca0ee5fe5fc7bc498072796b0c120
pass in quick on openvpn inet6 proto tcp from {(openvpn:network)} to {any} port {25} keep state label "USER_RULE: Allow SMTP" # bf9ca0ee5fe5fc7bc498072796b0c120
pass in quick on openvpn inet proto tcp from {(openvpn:network)} to {any} port {465} keep state label "USER_RULE: Allow SMTP/S" # 7d61e93edd0f3af2f75ffc5e2d598665
pass in quick on openvpn inet6 proto tcp from {(openvpn:network)} to {any} port {465} keep state label "USER_RULE: Allow SMTP/S" # 7d61e93edd0f3af2f75ffc5e2d598665
pass in quick on openvpn inet proto tcp from {(openvpn:network)} to {any} port {587} keep state label "USER_RULE: Allow submission" # 8c1bae83c5af45d6becd501c32f3a6c9
pass in quick on openvpn inet6 proto tcp from {(openvpn:network)} to {any} port {587} keep state label "USER_RULE: Allow submission" # 8c1bae83c5af45d6becd501c32f3a6c9
pass in quick on openvpn inet proto tcp from {(openvpn:network)} to {any} port {110} keep state label "USER_RULE: Allow POP3" # ae35cc40d9330080ec6014f75f9d8ce7
pass in quick on openvpn inet6 proto tcp from {(openvpn:network)} to {any} port {110} keep state label "USER_RULE: Allow POP3" # ae35cc40d9330080ec6014f75f9d8ce7
pass in quick on openvpn inet proto tcp from {(openvpn:network)} to {any} port {995} keep state label "USER_RULE: Allow POP3/S" # 5fbcc62b5d29f981239f445508b95c9d
pass in quick on openvpn inet6 proto tcp from {(openvpn:network)} to {any} port {995} keep state label "USER_RULE: Allow POP3/S" # 5fbcc62b5d29f981239f445508b95c9d
pass in quick on openvpn inet proto tcp from {(openvpn:network)} to {any} port {143} keep state label "USER_RULE: Allow IMAP" # 36342b4fbd340ee7003018b5f5e8c55a
pass in quick on openvpn inet6 proto tcp from {(openvpn:network)} to {any} port {143} keep state label "USER_RULE: Allow IMAP" # 36342b4fbd340ee7003018b5f5e8c55a
pass in quick on openvpn inet proto tcp from {(openvpn:network)} to {any} port {993} keep state label "USER_RULE: Allow IMAP/S" # c023213f49e26dd88685a07c80429b8a
pass in quick on openvpn inet6 proto tcp from {(openvpn:network)} to {any} port {993} keep state label "USER_RULE: Allow IMAP/S" # c023213f49e26dd88685a07c80429b8a
pass in quick on openvpn inet proto tcp from {(openvpn:network)} to {any} port {4190} keep state label "USER_RULE: Allow sieve" # fa193462087ebb3ad83c3e4d6aeda789
pass in quick on openvpn inet6 proto tcp from {(openvpn:network)} to {any} port {4190} keep state label "USER_RULE: Allow sieve" # fa193462087ebb3ad83c3e4d6aeda789
pass in quick on openvpn inet proto tcp from {(openvpn:network)} to {any} port $http_common keep state label "USER_RULE: Allow HTTP(/S)" # c74d470eedebfd923152616bbb86629e
pass in quick on openvpn inet6 proto tcp from {(openvpn:network)} to {any} port $http_common keep state label "USER_RULE: Allow HTTP(/S)" # c74d470eedebfd923152616bbb86629e
pass in quick on openvpn inet proto tcp from {(openvpn:network)} to {any} port {5223} keep state label "USER_RULE: Allow iOS Push Notifications (1/2)" # 011baa32d703d5fa59211bdd90e60523
pass in quick on openvpn inet6 proto tcp from {(openvpn:network)} to {any} port {5223} keep state label "USER_RULE: Allow iOS Push Notifications (1/2)" # 011baa32d703d5fa59211bdd90e60523
pass in quick on openvpn inet proto tcp from {(openvpn:network)} to {any} port {2195:2196} keep state label "USER_RULE: Allow iOS Push Notifications (2/2)" # 6e35f20d8afbfca7c221e69b34c8cb30
pass in quick on openvpn inet6 proto tcp from {(openvpn:network)} to {any} port {2195:2196} keep state label "USER_RULE: Allow iOS Push Notifications (2/2)" # 6e35f20d8afbfca7c221e69b34c8cb30
pass in quick on openvpn inet proto tcp from {(openvpn:network)} to {any} port {5228:5230} keep state label "USER_RULE: Allow Android GCM" # 7f611d2093650d4003b6310962e15ed0
pass in quick on openvpn inet6 proto tcp from {(openvpn:network)} to {any} port {5228:5230} keep state label "USER_RULE: Allow Android GCM" # 7f611d2093650d4003b6310962e15ed0
pass in quick on openvpn inet proto udp from {(openvpn:network)} port {16384:16472} to {any} port {3478} keep state set prio (5, 5) label "USER_RULE: Allow FaceTime (Setup)" # 1f44ed65042b3debd88af31a500f9b4e
pass in quick on openvpn inet6 proto udp from {(openvpn:network)} port {16384:16472} to {any} port {3478} keep state set prio (5, 5) label "USER_RULE: Allow FaceTime (Setup)" # 1f44ed65042b3debd88af31a500f9b4e
pass in quick on openvpn inet proto udp from {(openvpn:network)} port {16384:16472} to {any} port {16384:16472} keep state set prio (5, 5) label "USER_RULE: Allow FaceTime (VoIP)" # 45cd35020be6d38033d1a2af67ce61da
pass in quick on openvpn inet6 proto udp from {(openvpn:network)} port {16384:16472} to {any} port {16384:16472} keep state set prio (5, 5) label "USER_RULE: Allow FaceTime (VoIP)" # 45cd35020be6d38033d1a2af67ce61da
pass in quick on openvpn inet proto {tcp udp} from {(openvpn:network)} to {any} port {444} keep state label "USER_RULE: Allow SNPP" # adccbb50a4719e8ad1d8c166dc84fe55
pass in quick on openvpn inet6 proto {tcp udp} from {(openvpn:network)} to {any} port {444} keep state label "USER_RULE: Allow SNPP" # adccbb50a4719e8ad1d8c166dc84fe55
pass in quick on openvpn inet proto tcp from {(openvpn:network)} to {any} port {5222} keep state set prio (5, 5) label "USER_RULE: Allow XMPP" # 6df2e40a3ef9ac8fbd6d57645280816f
pass in quick on openvpn inet6 proto tcp from {(openvpn:network)} to {any} port {5222} keep state set prio (5, 5) label "USER_RULE: Allow XMPP" # 6df2e40a3ef9ac8fbd6d57645280816f

@fichtner
Copy link
Member

Does this help?

# opnsense-patch 71043f1

@agh1701
Copy link

agh1701 commented Aug 13, 2018

I too am receiving these errors hundreds in fact during boot and operation. I have 3 client VPN's in a gateway group. I also have a server VPN road warrior style. I get the following errors all the time. and this goes back to approx. 18.1.9ish.

There were errors(s) loading the rules: no IP address found for ovpnc2:network
There were errors(s) loading the rules: no IP address found for ovpnc3:network
There were errors(s) loading the rules: no IP address found for ovpnc4:network

eventually the interfaces will appear to be down. If I try to restart them I get an error indicating they are already running. Due to PID file corruption I believe, I must manually kill the processes and then start them.

I have tried the patch above and it dose not help. Is a restart required for it to work?
notifications
vpn stats

@fichtner
Copy link
Member

I need the rules grep to confirm....

@agh1701
Copy link

agh1701 commented Aug 13, 2018

root@router:~ # grep ovpnc2:network /tmp/rules.debug
nat on ovpnc2 inet proto {tcp udp} from (ovpnc2:network) to $nasip port {6881:6882} -> ovpnc2 port 1024:65535 # Forward Bittorrent
pass out log route-to ( ovpnc2 10.24.0.82 ) from {ovpnc2} to {!(ovpnc2:network)} keep state allow-opts label "let out anything from firewall host itself"
root@router:~ # grep ovpnc3:network /tmp/rules.debug
nat on ovpnc3 inet proto {tcp udp} from (ovpnc3:network) to $nasip port {6881:6882} -> ovpnc3 port 1024:65535 # Forward Bittorrent2
pass out log route-to ( ovpnc3 10.33.0.82 ) from {ovpnc3} to {!(ovpnc3:network)} keep state allow-opts label "let out anything from firewall host itself"
root@router:~ # grep ovpnc4:network /tmp/rules.debug
nat on ovpnc4 inet proto {tcp udp} from (ovpnc4:network) to $nasip port {6881:6882} -> ovpnc4 port 1024:65535 # Forward Bittorrent3
pass out log route-to ( ovpnc4 10.22.0.82 ) from {ovpnc4} to {!(ovpnc4:network)} keep state allow-opts label "let out anything from firewall host itself"

@fichtner
Copy link
Member

are you sure the errors still appear after firewall reboot?

@agh1701
Copy link

agh1701 commented Aug 13, 2018 via email

@agh1701
Copy link

agh1701 commented Aug 13, 2018

Just did a reboot. Patch does not help.

@fichtner
Copy link
Member

I can see the patch doing the right thing in your rules so you'll have to help me out with clearing the errors and letting me know how many errors a typical boot without and with the patch has...

@agh1701
Copy link

agh1701 commented Aug 15, 2018

I will get the screen shots.

@agh1701
Copy link

agh1701 commented Aug 16, 2018

here is a boot with patch not the errors that can not be seen are the same as have bee reported. the 1st 2 are new. I have never seen them before.
17e
Her is a boot with patch after removing my port forward rules. 24 hours later it doubled to 8
e4
Here is a boot without patch and without my port forward rules.
e2
I did have the feeling that with the patch there were more errors.

@agh1701
Copy link

agh1701 commented Aug 16, 2018

at some point maybe after 2-5 days and many errors the vpn's do go down like #2610 . I must do:
"ps auxw | grep openvpn" kill all vpn processes because they are running and then start the vpns from the webui,

@fichtner
Copy link
Member

@agh1701 ok, so the patch does work for the ":network" errors, and ":0" doesn't work yet. but you pasted them before. they were always there... look at your old screenshot.

I'll see if I can pin down the ":0" usage and fix that as well then. :)

Thanks so far!

fichtner added a commit that referenced this issue Aug 16, 2018
This simplifies the logic for the interface redirect target
as we use :0 and push it directly to the kernel to resolve.

From pf.conf:

   Tables can also be used for the redirect address of nat and
   rdr rules and in the routing options of filter rules, but only
   for round-robin pools.
@DanMc85
Copy link

DanMc85 commented Aug 16, 2018

I am having the same/similar issue as well... documented starting at this thread post:
https://forum.opnsense.org/index.php?topic=9133.msg42810#msg42810

@agh1701
Copy link

agh1701 commented Aug 17, 2018

is there a patch for testing or are we waiting for the next release?

@fichtner
Copy link
Member

It’s better to wait now. Quite a few spots have been changed to squash these errors.

Although I do have to ask: does this have an operational effect or is it just a cosmetic annoyance that boot takes longer and showed these errors but system works fine?

@sjorge
Copy link
Contributor Author

sjorge commented Aug 17, 2018

In my case I have to reload pf to get NAT to work after a boot.

@agh1701
Copy link

agh1701 commented Aug 17, 2018

Operational. At some point 2 or 3 days in operation some pid file corruption occurs. the VPN's appear down(red). because the VPN's are on a group gateway the gateway goes down. in actuality they are running, I must use "ps auxw | grep openvpn" and kill them by hand. after that I am able to start them from the webui. For some reason it affects my openvpn server as it shows as down but I am able to VPN into the router.

@agh1701
Copy link

agh1701 commented Aug 17, 2018

also during boot dnscrypt_proxy2 will not start. I must SSH in and start by hand.

@fichtner
Copy link
Member

fichtner commented Aug 17, 2018 via email

@agh1701
Copy link

agh1701 commented Aug 17, 2018

Yes, I agree feels like a timing issue. I have been questioning whether the "BEFORE:" directive is being observed in the scripts in /usr/local/etc/rc.d.

@agh1701
Copy link

agh1701 commented Aug 17, 2018

I have another question. what is causing the PID file corruption. after booting and making sure everything is started. 2-3 days later the services show down. I restart them and 2-3 days latter... with out rebooting.

@fichtner
Copy link
Member

Very little is used from the stock rc.d system ever since the m0n0wall fork. That being said most of the ordering is working, in this scope basically:

  1. Create OpenVPN devices
  2. Network setup
  3. Reload firewall rules
  4. Bring up OpenVPN service and connectivity
  5. Reload firewall rules
  6. Redo network connectivity if necessary (and reload firewall rules)

Now the underlying issue is that gateway(s/ groups) are relevant for step 3 and 5 and 6, so you have OpenVPN devices not configured but asked to be part of rules that are supposed to do network-related translations that have not yet been started in case of step 3. There may be more side-effects from slow links (PPPoE can be particularly slow in this regard to fail to provide basic connectivity in step 2 and thus to be redone in step 6 so you end up with multiple rules reloads that are not fully functional and will throw these errors.

There's no priority / connectivity hierarchy in gateway groups which can make this a daunting task to get just right to "program" which link is required to be able to upgrade a gateway or for it to be used at all, let alone set up and enabled in the rules dynamically.

But step by step, we have time :)

@sjorge
Copy link
Contributor Author

sjorge commented Aug 18, 2018

I've reverted back to 18.1 snapshot for now, I eliminated all the openvpn:network from my rules, but I cannot get them out of the NAT rules (They seem to be automatically added in all cases?).

This results in the not having a working gateway after boot and needing a reload of pf from the serial console before hosts get IP and traffic will flow.

My avoiding using the openvpn network I did get the error count down to 2 (NAT) coming from a few more.

fichtner added a commit that referenced this issue Oct 15, 2018
This simplifies the logic for the interface redirect target
as we use :0 and push it directly to the kernel to resolve.

From pf.conf:

   Tables can also be used for the redirect address of nat and
   rdr rules and in the routing options of filter rules, but only
   for round-robin pools.

(cherry picked from commit 3ddea14)
(cherry picked from commit 5810cc7)
fichtner added a commit that referenced this issue Oct 15, 2018
@fichtner
Copy link
Member

All the patches I wrote are in 18.7.7 now. Does the issue still persist?

@agh1701
Copy link

agh1701 commented Nov 12, 2018

I cleared the errors and they have not appeared in just under a day. Also since 18.7.6 my VPN's have not gone down.

@agh1701
Copy link

agh1701 commented Nov 12, 2018

Thanks fichtner!

@fichtner
Copy link
Member

@agh1701 nice, thanks for the feedback! Closing then... be sure to ping if issues seem to be back for some reason. :)

@sjorge
Copy link
Contributor Author

sjorge commented Nov 12, 2018

@fichtner they are not all gone for me, those generatred by firewall rules are... but those cause by forwarding/NAT are not.

NAT Rules

The two disabled one are what is still triggering this for me.

@fichtner
Copy link
Member

fichtner commented Nov 12, 2018 via email

@DanMc85
Copy link

DanMc85 commented Nov 27, 2018

With the latest 18.7.7-18.7.8...

After a while of running... in this case 4 hours of uptime. I still see two issues: "There were error(s) loading the rules: no IP address found for ovpnc2" which I need to acknowledge. Additionally and in my opinion, more importantly: OPNSense is losing the status of the OpenVPN clients and servers. They show the red down icons and lose IP addresses, but they are all really still up and active. I believe OPNSense is losing track of the status of the OpenVPN connections. I believe this issue was fixed once before in a recent past release but has since resurfaced.

OpenVPN errors:
Unable to contact daemon
Service not running?

However, logs show: "openvpn[46774]: Cannot open TUN/TAP dev /dev/tun2: Device busy (errno=16)"...
because it is still running and working.

@fichtner
Copy link
Member

I don't think the status has ever been fixed. It's a race somewhere in the setup / teardown code in conjunction with a bumpy line. For the other thing we need:

# grep ovpnc2 /tmp/rules

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cleanup Low impact changes
Development

No branches or pull requests

4 participants