interfaces: dhclient-script regression in 19.1 #3197
Comments
My OPNsense is running on a apu1d4 from PC Engines for some year without any issue. |
|
could be driver (realtek), other connectivity issues. or when ips is enabled related to https://github.com/opnsense/core/issues/3175 |
|
IPS is not enabled, ipv6 is also disabled. The newwanip script is called every 20 minutes. after every run my connections are being droped. |
|
newwanip is likely triggered due to another event, maybe dmesg has more info (or there's data before |
|
The data before your mentioned line is the end of my mentioned line 😄 how can i output dmesg? I have ssh access. |
|
just |
|
I see a lot more of: you need the whole output? |
|
cause and effect... so, yes, the logging provided so far doesn't tell a lot about your issue. |
|
WAN connection is gone too, I cannot find anything in dmesg output. |
|
Do you know where ovpns2 is listening on? WAN perhaps? |
|
Yes WAN |
|
Do you get a new IPv4 on WAN every time this happens? |
|
No, the IP is always the same, although I dont have a fixed ip. |
|
Please check /var/db/wan_cacheip and /var/db/wan_ip -- they shouldn't match, correct? |
|
But they match O_o |
|
It's not the driver issue, i have the same on different hardware. And i have static IP on WAN, but issued by DHCP. |
|
Let's try to find it... this one looks like a good candidate... c83bb8d: |
|
Same issue here, other hardware (Intel NIC's). My ISP has had a DHCP renew time of 30 seconds and after upgrade to 19.1.1 I get every 30 seconds a drop of my OpenVPN interface. In the meantime my ISP has changed the DHCP renew time to 12 hours => no issues so far. Additionally the NTPD service get's also and exit signal during the rc.newwanip script:
I have figured out how to easy manually reproduce the issue (VPN interface down/up + the NTPD exit signal in my case): In the Web Interface go to "Interfaces > Overview" and just klick "Reload" on the WAN interface |
|
Applied the patch c83bb8d and klicked "Reload" still causes my ovpns1 link DOWN/UP: and NTPD exit signal:
|
|
I had the same (or at least a similar) issue (IP renewal starting on WAN every 30 minutes), then applied c83bb8d about 75 minutes ago, and rebooted. Since then, no more IP renewal. |
|
Patched again + reboot... still VPN link down/up during IP renewal. |
|
After patching the tv stream didn't drop for over 1 hour. Tomorrow I can test if the vpn connection is stable again. Thank you very much for your quick interaction. |
|
I can't see any newwanip logs over the night. So it ran stable I think. I know now why the connection dropped everytime... I had "Firewall: Settings: Advanced: Dynamic state reset" active (Reset all states when a dynamic IP address changes.This option flushes the entire state table on IPv4 address changes in dynamic setups to e.g. allow VoIP servers to re-register.) But thank you for the patch! |
|
What happens to the patch if I install some updates through the web gui? Does it persist? |
|
Got it, the "Reload" button forces the rc.newwanip script but the patch doesn't call it from dhclient if it's not a real IP change, right? So the patch works also for me! There was a DHCP triggered renewal try on my WAN without real IP change a few minutes ago (log entry in the dhcp log). But nothing to see about the rc.newwanip in general system log and also no exit signal in NTPD log. So now my ISP can go back to stupid 30 second IP renewal time ;-) Thanks a lot!!! Same question as tbandixen: Will this patch persist or will it find the way into updates? |
|
The patch will not persist into 19.1.2 and needs to be reapplied worst case. opnsense-patch was designed as a tool to test changes between versions and to troubleshoot. As the change for has been proposed in issue #2542 for a long time for a good reason we now need to find the problem contained within the patch itself as a way to go forward with a better path than just to go back to what we had before. |
|
I have the same problem with 19.1.1 on two APU2C4 systems. Another system running 19.1.1 on APU1D4 doesn't show this problem. Is there anything I can help to diagnoses the issue? |
The FreeBSD one does, but we have different code that is "safe" for multiple DHCP connections. (cherry picked from commit b20f71b)
|
I have the same issue and started looking around in the log files. I believe the answer lies in System: Gateways: Log File and possibly with dpinger. I see this in the log file repeating every 15 minutes:
I noticed that by going to Firewall: Settings: Advanced and checking the "Disable State Killing on Gateway Failure" option I was able to stop the routine dropping of connections. I then noticed the following:
Could this be the culprit? |
|
what you see is dpinger reporting the very fact that there's a problem |
Are there any configuration difference between your systems on the APU2C4 and APU1D4 ? |
Could this be the solution? Is someone willing to provide a patch that I can test? |
|
No, we need to explain what happens first. Something changed, but the changes in the dhclient-script are too superficial to cause this so I'm afraid of later breakage if we don't pin this down. |
|
I totally understand that. How can I provide further help? |
|
Might be an idea to add some extra debug output to the dhclient-script and get @tbandixen to do some tests and give feedback. Unfortunately I'll not be able to add the extra bits myself until sometime next week! |
|
@tbandixen - Little test for you. Copy the attached script to /usr/local/opnsense/scripts/interfaces . Backup the existing script first. See if it makes any difference at all. Pointers are needed to try and identify the issues so all this script does is filter out the RENEW response and bypass any actions. |
|
Ok, I ignore the renew now (as provided in the script) lets see if the vpn is stable now. will report asap |
|
Well, as expected, the connection remains stable. I will include the ignored lines step by step to identify the causing line. |
|
How can I manually trigger this script so I dont have to wait 20minutes every time? |
|
It's more a question of if the 'RENEW' Bypass helps, what has changed elsewhere that's causing the issue. Has something changed in dhclient itself perhaps? @fichtner can answer that one. |
|
I think just bypassing the RENEW isnt that good, its there for a reason. But I dont know if behind the scenes something changed (BSD things maybe, I dont know the *nix subsystem at all). I will try to find the line that causes the drop and digg a bit further. |
|
Actually bypassing RENEW is good, it means nothing has changed so do nothing. |
|
The key is that the var 'changes' is changed to 'yes'. What I would do next is add a logger entry to each and every if... fi statement block to identify the culprit. Something like this:- |
|
I think we should check if |
|
I think setting |
|
My VPN connection remains stable if I add |
|
@tbandixen thanks for the hint, the check changed as it tried to align with FreeBSD. It seems the issue is present there, but it is not critical because it doesn't have a use case maybe shrugs That should be it then.... |
|
historic reference: pfsense/pfsense@d0d7f09ab3853b |
|
Its all stable now 😃 |
|
Thank you very much! |
Suggested by: @tbandixen (cherry picked from commit 90c0c39)
|
Ok, we'll wrap this up for 19.1.2. Thank you for the help. ❤️ |
Just want to be sure the dev are aware of "our" issue.
After 10-20 minutes of uptime all incoming connections are beeing dropped!
So, OpenVPN tunnels are dropped too. It was fine on 18.7.10.
In the General-system log I can see this every 20 minutes:
after every execution of rc.newwanip (even manually) all connection are dropped.
I watch TV over the internet and every 20 minutes the stream hangs, so I have to rewind to build up the stream again.
Here is the forum post:
https://forum.opnsense.org/index.php?topic=11456.0
Is there any solution? What could it be?
According to github, the last changes to rc.newwanip where 5 month ago (https://github.com/opnsense/core/commits/master/src/etc/rc.newwanip).
I think it has something to do with the switch to HardenedBSD, but I am absolutly not a unix guy...
The text was updated successfully, but these errors were encountered: