High memory usage, when reading large file over NFS

[rtgnx]

What is the issue?

When reading large file over NFS tailscaled memory usage grows until host runs out of memory.
This is the case for both copying large files back and forth and playback over nfs.

EDIT:
This is much more generic issue, any large inbound or outbound traffic fills up the host memory.

Showing nodes accounting for 1274.84MB, 97.91% of 1302.10MB total
Dropped 64 nodes (cum <= 6.51MB)
Showing top 10 nodes out of 20
      flat  flat%   sum%        cum   cum%
 1248.37MB 95.87% 95.87%  1248.37MB 95.87%  github.com/tailscale/wireguard-go/device.(*Device).PopulatePools.func3
   18.02MB  1.38% 97.26%    18.02MB  1.38%  github.com/tailscale/wireguard-go/device.(*Device).PopulatePools.func2
    7.45MB  0.57% 97.83%     7.45MB  0.57%  tailscale.com/net/tstun.wrap
       1MB 0.077% 97.91%    11.59MB  0.89%  github.com/tailscale/wireguard-go/device.(*Device).NewPeer
         0     0% 97.91%  1248.37MB 95.87%  github.com/tailscale/wireguard-go/device.(*Device).GetMessageBuffer (inline)
         0     0% 97.91%    18.02MB  1.38%  github.com/tailscale/wireguard-go/device.(*Device).GetOutboundElementsContainer (inline)
         0     0% 97.91%    11.59MB  0.89%  github.com/tailscale/wireguard-go/device.(*Device).IpcSetOperation
         0     0% 97.91%  1147.78MB 88.15%  github.com/tailscale/wireguard-go/device.(*Device).NewOutboundElement
         0     0% 97.91%  1164.20MB 89.41%  github.com/tailscale/wireguard-go/device.(*Device).RoutineReadFromTUN
         0     0% 97.91%   101.59MB  7.80%  github.com/tailscale/wireguard-go/device.(*Device).RoutineReceiveIncoming

Steps to reproduce

Host: Voidlinux
Tailscale: 1.60.1 go version: go1.22.0

1. Mount share using tailent dns (lan, direct connection)
2. Copy large file in my case it was about 25G.

Are there any recent changes that introduced the issue?

No response

OS

Linux

OS version

Void Linux rolling release

Tailscale version

1.60.1

Other software

Standard 1G networking switch.
VLC for playback
macOS builtin nfs client

Bug report

BUG-3ce6694459da94c48ba57a4586b7d07948700ecbcbbedd6696bf310427941400-20240309152512Z-ed86f8854193f4de

[gitlankford]

this is a wireguard-go issue.
WireGuard/wireguard-go#69