unbound: outgoing network interfaces not honored #5329
Comments
|
I'm experiencing the same issue on 21.7.4. What OPNsense version did you use when it worked? I tried rolling back to 21.7.2, but the issue persists. Were the following steps correct or am I missing something? opnsense-update -kr 21.7.2
opnsense-revert -r 21.7.2 base
opnsense-revert -r 21.7.2 opnsense |
|
This has worked for years in every version of opnsense up to and including 21.7.3. I had adguard exclusively using unbound on port 5353, and unbound doing DoT over my WG tunnels. I had also used the same config before starting to use adguard on top of it. Sorry, I don't know how to revert everything. Right now I'm still running adguard but it is using TLS directly rather than talking to unbound. dnsleaktest always worked with this config, showing only the DoT servers being used. |
So this would indicate that my revert didn't work. I've been trying to get this to work for the better part of last week without success. I might try to clean install an earlier version. Maybe even 21.1. |
|
I guess there's a revert problem for you, since before 21.7.4 my logs showed all queries to port 853 (DoT) where going through WG |
|
I clean installed and updated to the following version and everything works as expected. Might the updated versions of Is there a way to selectively upgrade to 21.7 to isolate the package upgrade that's causing this bug? I'll clone the live system as is and will gladly assist in debugging this. Is |
|
The situation in 21.7.5 is the same as 21.7.4, I can't specify WG outgoing interfaces for unbound, I have to specify WAN or it doesn't work. |
|
This MIGHT be similar to https://forum.opnsense.org/index.php?topic=25327.0 (German) Problem is you cant revert as the package was removed/replaced. Just as guess as this is the only DNS relevant change in 21.7.4. |
|
I don't see Unbound acting up for using dnspython v2 in alias resolving with regard to WireGuard as an outgoing interface. It's more likely that 6d57215 does something, but there are no relevant indicators here in this report like pf rules comparison or naming the steps for configuring wireguard for it and if any custom rules are in the way and whether WireGuard interface is assigned or not. |
I don't think this is related. @fichtner : no rules comparison or otherwise complex setup is needed to reproduce the issue. Here are my steps off the top of my head:
With this simple setup Unbound uses the default route and not the selected interface |
Sure but not doing it will not point us into the right direction. |
Are the steps I provided enough? I'll try to update my setup to 21.7.3 this weekend. Can I |
|
it should unpatch, yes, otherwise use to move the core package to the working release. |
|
I was running on 21.7.4 and tried to revert, but things were still broken, so I clean installed 21.1 and upgraded to 21.1.9, which is what I'm currently running. My thinking was to upgrade to 21.7.3, the latest version where Unbound works properly. Then apply the |
|
the good state of 21.1.9 works too I suppose. We need to compare the /tmp/rules.debug output on a good and the bad configuration. |
|
This worked for me on 21.7.3, but broke with the .4 update (and .5)
Thanks to both of you for your help!
Chris
…----------------------------------------
Nov 12, 2021 7:29:27 AM Michael Schnerring ***@***.***>:
I was running on 21.7.4 and tried to revert, but things were still broken, so I clean installed 21.1 and upgraded to 21.1.9, which is what I'm currently running.
My thinking was to upgrade to 21.1.3, the latest version where Unbound works properly. Then apply the *6d57215* patch to verify it breaks things. Or isn't that helpful to you?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub[#5329 (comment)], or unsubscribe[https://github.com/notifications/unsubscribe-auth/ARWE2I7M4GSRQN6HKGOOIULULUJBZANCNFSM5HE4QO2A].
Triage notifications on the go with GitHub Mobile for iOS[https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675] or Android[https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub]. [data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEgAAABICAYAAABV7bNHAAAABHNCSVQICAgIfAhkiAAAACtJREFUeJztwQENAAAAwqD3T20ON6AAAAAAAAAAAAAAAAAAAAAAAAAAAD4MUUgAARy2AfAAAAAASUVORK5CYII=###24x24:true###][Tracking image][https://github.com/notifications/beacon/ARWE2I2RIJ2RKAMAHBL67XTULUJBZA5CNFSM5HE4QO2KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOHGSRAJY.gif]
|
|
I clean installed 21.1.1 and performed the following steps:
With this config, Unbound uses the VPN tunnel. Running to this: And indeed, Unbound then uses WAN instead of WAN_VPN. Running I'm gonna do an upgrade from 21.1.9 to the latest version and try to revert the patch to see if that works, too. |
|
Reverting the patch works on the latest version:
Do future releases automatically re-apply reverted patches, or is that something I need to keep in mind? |
|
Great troubleshooting, well done! |
and what if you assign this gw as upstream gateway in |
I do that on the WG local peer. I check Disable Routes and specify the gateway IP there. I can try, but the docs say to leave the interface unconfigured. Or is that something that changed? |
|
I just looked at the code..it looks for gateways in interfaces properties
now imho.
Can you try to set upstream gateway in wan_vpn properties?(not only in wg
local peer).
On my test vm it starts to generate pf rules after that
…On Saturday, November 13, 2021, Michael Schnerring ***@***.***> wrote:
and what if you assign this gw as upstream gateway in WAN_VPN properties?
I do that on the WG local peer. I check *Disable Routes* and specify the
gateway IP there. I can try, but the docs say to leave the interface
unconfigured. Or is that something that changed?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#5329 (comment)>, or
unsubscribe
.
|
|
Yes, you're right. Statically configuring IPv4 on the interface fixes the issue. I don't really understand what's happening on a lower level when assigning the
Ever since I've "learned" doing it this way, I assumed that WG populates the IPv4 config of the interface with the values entered in the local peer, namely the (local) tunnel address and gateway IP. So is this a bug? Or is it really required to enter duplicate information on the local peer AND on the interface? A while back I read the following comment from @fichtner . I don't fully understand, but does it have something to do with that?
|
|
Interesting. If you leave the upstream gateway as "Auto-detect", is it still OK? I presume so since at least in my setup the WG gateway is the only one listed in the dropdown. I agree it seems duplicative to have to include the gateway info in the WG local config as well (and also the IPs, although presumably that must be in the WG config for WG to work). Particularly as the WG config only allows for one gateway, so both IPv4 and IPv6 can't be specified (but they both work regardless). I will update the how-to once the proper configuration has been clarified. I wonder if this also means that I should be configuring the IPs on the interface for my WG road warrior setup as well? |
|
(I haven't read all the comments, but thought to shed some light on gateways) If we're looking at #5230, the old behaviour was more or less a bug, some time ago we added In which case one can assign a gateway for an interface which is configured by the service itself (it generates rules like Auto detect gateways are only used if there's a file provided with an address in it (which dhcp clients do) or "Dynamic gateway policy" is set, which more or less explained in https://docs.opnsense.org/manual/gateways.html#missing-dynamic-gateway |
|
I'm not sure that will work for WG as when "Disable Routes" is selected in the WG local config (which is needed to allow selective routing of hosts through the tunnel), a gateway IP is required to be specified. |
|
so if I understood @AdSchellevis correctly, the changes are intentional (and I also think the previous behavior was less logical). And if I see correctly, the only thing that has changed is that now an allowing rule with the |
|
Much of this goes over my head, and as an end user of an appliance, I only want to concern myself with implementation details as much as I have to. I'll sure learn things along the way, but I didn't expect having to spend so much time to get to the bottom of this. Does this come down to WG still being experimental and its integration requiring ironing out which isn't really possible before being implemented on a kernel level? So if one of the solutions @kulikov-a mentions is the right / "logical" one, we should at least update the WG docs. |
|
I agree with @schnerring and I have no useful input as to what the right way is, but instructions in the documentation and perhaps removal of "outgoing network interfaces" in the unbound config would be good, since it seems to add confusion. I'm happy with whatever the "best" way is, and thank everyone involved. |
Ah - ntopng!
Well, Mullvad gives a /32 and a /128 for the tunnel, and that is what I configure in WG. In theory I could specify a larger netmask, I suppose (and in fact I do for IPv6 since otherwise I can't configure a gateway, since Far Gateways aren't a thing for IPv6). But I am not sure it would make any difference? |
|
With static config: Without static config: Btw, I'm using |
|
@schnerring could be the case, it doesn't really make sense that setting the same address again magically changes the routing. If that's the case, it should be easy to validate by manually setting only an address on |
|
So things work when I just add the floating rule that's also added by statically configuring the WG interface. I'd expect the user-defined floating rule to effectively override the auto-generated rule from the static WG interface configuration. But this only partially happens. Requests from Unbound still trigger the What's odd is that Unbound requests display
I remember having read somewhere that this is just a display error, but I don't remember where. edit: "yoy" is the custom floating rule that replicates the auto-generated rule from static IP configuration |
If that's the case we can safely ignore my previous comment, wireguard-kmod isn't doing anything weird then.
If both match I would expect the same indeed.
Doesn't ring an immediate bell on my end, if responses do go out using the correct interface, it might be possible that traffic is matched twice in some way, but that's guessing from my end. |
|
I'm going to factory reset and re-create a minimal setup just to rule out that this is caused by any other configuration on the system. 😫 It feels like that this is an issue between WireGuard and Unbound. |
|
@schnerring not sure if I understood your previous comment correctly, but reading it again I think you mean if you add the same rule as the floating rule (without an interface) it works, but not if you stick it to an interface, which would indicate that we are constructing the wrong rule here. Going back in the comments, the generated rules don't stick to an interface (so are floating without interfaces choosen). Which would also explain the log on wan (it is leaving there, but with the rule in place you physically reallocate it to another interface). I assume Unbound puts the packet on the line using a source address, which would logically go out on wan (destination routing), if unbound would force the packet on the interface, the route-to on the interface would make more sense. So long story short, @kulikov-a's rule (#5329 (comment)) should work, removing the validation (b567ddb) in this case shouldn't make a difference as the packet never enters |
|
@schnerring @AdSchellevis sorry, if i missed something. unbound binds to ip (not to interface). so imho the question is whether is wg interface acualy have address at the unbound start (and what address unbound binds to if its not. quick test shows that it binds to |
I'm sorry about that, you're right. I was sticking the rule to the
1, 2, and 3 are the configurations I tested. 3 is also what @Greelan implemented in his PR (https://github.com/opnsense/docs/pull/365/files) and what I configured in my original comment (#5329 (comment)) when trying to reconstruct the rule.
So yes, @kulikov-a meant 4. Such a small detail 🔬 ... I think this is resolved then? |
|
@schnerring only issue that could remain if I understand @kulikov-a correctly is that there might be race condition in unbound startup (if the address is not yet there), but if @Greelan adds [4] in his how-to, at least the interface and firewall sections are functional. I would close the ticket with @Greelan's documentation PR, sounds like we managed to find a solution indeed. |
|
In regards to the race condition, I quoted @fichtner in an earlier comment (#5329 (comment)), but didn't understand at the time. It's basically what @kulikov-a said. And now I think I understand why the help text of the Outgoing Interfaces setting says this:
It seems to me that statically configuring the WG interface seems to be the better option? Unless the race condition issue is something that needs to be addressed (I couldn't find an open issue). |
|
I don't think you can't fix the race by setting a static address on the interface as it's simply not there, but you should be able to add a loopback on use the same trickery to send it out on The quickest "fix" would be to assign the same address using an alias to |
If I am following correctly, I should simply change the floating rule in the PR so that no interface is selected? I am still quite curious why the static IP and gateway assignment on the interface is not the preferred solution, if it creates the same routing and appears simpler from a user perspective? BTW, @schnerring, your earlier post shows a /128 netmask with the IPv6 address. How did you create a gateway in that case? OPNsense wouldn't let me when I tried because the gateway IP necessarily wasn't in the same subnet as the interface address. Hence I use a /127. |
I didn't configure a GW for IPv6 because I don't use IPv6 (yet). However, for Mullvad interfaces I went ahead and already added the IPv6 addresses. |
It sounds like it, yes.
In my humble opinion the risk of doing so is that you're going to reconfigure an already configured (or missing) device with a configuration that might be different than the one chosen by the process responsible for it. Not all interface types will like this and in some cases it may lead to an unconfigured interface after the fact. An option might be to show the gateway on interfaces when there's no explicit type chosen so you are allowed to force one for devices configured elsewhere, but I'm not sure that's worth the trouble. |
Weirdly, I now tried the floating rule with no interface selected (and with no static configuration on the interface itself), and now my gateways tell me they are offline with 100% packet loss... EDIT: Scratch that, they are back. Seems something else was the issue.
But I'd consider that just a user configuration error, and not a design issue. But just my view.
I like the sound of that. And removing the need in the WG config to specify a gateway IP if Disable Routes is enabled. It seems some more general thinking on how interfaces and gateways are handled with WG in OPNsense might be needed? |
hm. like: UPD. sorry. misunderstood. the comment is irrelevant |
|
What @Greelan means is using the following rule (which is what @kulikov-a suggested):
The don't select any part is where we got it wrong before. We selected |
|
Correct, @schnerring, which generates: (The rule descriptions are mine.) My bad for not reading @kulikov-a's original screenshot correctly. |
My understanding is that the floating rule basically duplicates what the Gateway IP setting should do, right? It feels like this was way too hard to get right. I mean from an end-user perspective it would be nice if the Gateway IP setting on the WG local peer was a dropdown menu where one could select a gateway which generates the above rule. We'd also need an additional dropdown to select an IPv6 GW. |
I'd be fine with that too. I agree wholeheartedly that this fiddling under the hood shouldn't be necessary, and some simpler solution to achieve the desired outcome and behaviour is needed. Hopefully something that the plugin maintainer can look at. |
|
I have no idea how this should be done right. The gateway field was introduced in times where there was no init script of wireguard itself and at times where also the dynamic gateway checkbox wasn't there in interface configuration. With all this I'm not really sure if it's even required to still use the gateway field in wireguard. Currently it adds more complexity and might multiply failure reports, but removing it or changing the default will break most setups when used with Mullvad or Azire or PIA or similar. |
@schnerring right, if it knows the gateway, by default it will generate an outbound rule.
@schnerring I see what you mean, but unfortunately we have to know with certainty what the intend was. It can be fixed from the plugins perspective (adding a gateway file https://docs.opnsense.org/manual/gateways.html#missing-dynamic-gateway) when the interface is available. We could opt from detaching the gateway[v4/v6] setting in the interface so one could explicitly set an upstream for dynamic gateways too, but I'm not sure if this is worth the effort to be honest.
@mimugmail I assume you mean this part https://github.com/opnsense/plugins/blob/ff0e665a868a7ff7ec3ef5a516339b037b38b727/net/wireguard/src/opnsense/service/templates/OPNsense/Wireguard/wireguard-server.conf#L24 , in which case it should be possible to also create/remove the required gateway file after which I suppose the gateway will be automatically added as well. (same process as described above) |
|
So I did some more testing and ran into another issue. I also use WireGuard to connect from from the outside to my homelab. I use Mullvad port forwarding to connect through a WireGuard tunnel to OPNsense with WireGuard (so a tunnel inside a tunnel - I hope that's clear). I need to define two rules to make this work.
Using static IPv4 configuration on the WG interface works. But as soon as I remove the static IPv4 configuration from the interface it stops working. Also, the "tunnel inside a tunnel" configuration only seems to work with |
…rules as discussed in #5329 (comment) (cherry picked from commit b567ddb)
Wouldn't that break some configurations? I have two VPNs setup (site to site, and another for WAN purposes) and I think that would cause issues. edit: nevermind |
Not sure why it would. Your two VPNs would be using separate wgX devices and therefore a rule for one should not affect the other? |
|
This issue has been automatically timed-out (after 180 days of inactivity). For more information about the policies for this repository, If someone wants to step up and work on this issue, |
Important notices
Before you add a new report, we ask you kindly to acknowledge the following:
Describe the bug
I have a single WAN and two WG interfaces, and have configured unbound DoT to use the two WG interfaces as the "outgoing network interfaces" on the settings page. This has worked in all versions before 21.7.4, but in the current version WAN must be included in the outgoing interfaces or unbound stops working.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Up until 21.7.4 the DoT DNS lookups went over WG, now they don't do anything unless WAN is selected
Describe alternatives you considered
A clear and concise description of any alternative solutions or workaround you considered.
Screenshots
If applicable, add screenshots to help explain your problem.
Relevant log files
If applicable, information from log files supporting your claim.
Additional context
Add any other context about the problem here.
Environment
Quotom box
Software version used and hardware type if relevant, e.g.:
OPNsense 21.7.4-amd64
FreeBSD 12.1-RELEASE-p20-HBSD
OpenSSL 1.1.1l 24 Aug 2021
The text was updated successfully, but these errors were encountered: