Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net: Golang udp listener send reply with unexpected source ip #36421

Open
pmghong opened this issue Jan 7, 2020 · 3 comments
Open

net: Golang udp listener send reply with unexpected source ip #36421

pmghong opened this issue Jan 7, 2020 · 3 comments
Milestone

Comments

@pmghong
Copy link

@pmghong pmghong commented Jan 7, 2020

What version of Go are you using (go version)?

$ go version
go version go1.12.1 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/data/go_work"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build134300587=/tmp/go-build -gno-record-gcc-switches"

What did you do?

At the very first time, we found that when I dig the backend dns server, which is written in go, we got an unexpect source alert, it seems that the reply came from another interface from the machine.
image

So I want to stimulate this issue with a tiny program like:

package main
import (
    "fmt"
    "net"
)
func main() {
    listener, err := net.ListenUDP("udp", &net.UDPAddr{IP: net.ParseIP("0.0.0.0"), Port: 9999})
    if err != nil {
        fmt.Println(err)
        return
    }
    fmt.Printf("Local: <%s> \n", listener.LocalAddr().String())
    data := make([]byte, 1024)
    for {
        n, remoteAddr, err := listener.ReadFromUDP(data)
        if err != nil {
            fmt.Printf("error during read: %s", err)
        }
        fmt.Printf("<%s> %s\n", remoteAddr, data[:n])
        _, err = listener.WriteToUDP([]byte("world"), remoteAddr)
        if err != nil {
            fmt.Printf(err.Error())
        }
    }
}

UDP server listen on (:9999 or 0.0.0.0:9999)
image

What did you expect to see?

The machine we are gonna access is bound with multiple interface, i.e. it has eth0/eth1/tun0/tun1 etc.
so when the client get access to this service from eth0, it is expect that the reply must be sent from interface eth0.
image

What did you see instead?

image

Udp server local ip (multi)
x.x.52.207
x.x.56.205
x.x.78.207

client (remote IP) : x.x.218.174

run tcpdump from udp server
image

as you can see, the server always reply from the default interface, the steps are as belows:

  1. client(x.x.218.174) -> udp server(x.x.52.207) got reply from udp server(x.x.52.207)
  2. client(x.x.218.174) -> udp server(x.x.56.205) got reply from udp server(x.x.52.207)

TCP connections all work as expect except UDP connections.

However, when we test this scenario with Nginx, it works:

image

image

image

As you can see it reply the client with the incoming interface instead of the default one.

Here is the simplified config:

# nginx listen on 0.0.0.0:9999
stream {

    upstream dns {
       server x.x.41.44:9999;
    }

    server {
        listen 9999 udp;
... ...
    }
}
@pmghong pmghong changed the title Golang udp listener from multiple netlink machine get reply from the wrong interface Golang udp listener send reply with unexpected source ip Jan 7, 2020
@toothrot toothrot changed the title Golang udp listener send reply with unexpected source ip net: Golang udp listener send reply with unexpected source ip Jan 7, 2020
@toothrot
Copy link
Contributor

@toothrot toothrot commented Jan 7, 2020

From the discussions I've read in similar issues (#23367 (comment), #17930), there hasn't been a consensus on a better UDP API. I'm no networking expert, but I believe nginx is copying over the address from the inbound UDP to the outbound UDP message. I believe it's doable in Go, but there's not an obvious way to me in the standard library right now.

I'm not sure how WriteToUDP would know which interface you read from unless you were explicitly telling it, given the way the listener works.

/cc @mikioh

@toothrot toothrot added this to the Backlog milestone Jan 7, 2020
@networkimprov
Copy link

@networkimprov networkimprov commented Jan 7, 2020

@toothrot FYI mikioh has not been active on the issue tracker in many months.

@qingyunha
Copy link
Contributor

@qingyunha qingyunha commented Jan 8, 2020

You may try x/net.

l := ipv4.NewPacketConn(listener)
l.SetControlMessage(ipv4.FlagDst, true)

n, cs, remoteAddr, err := l.ReadFrom(data)
cs.Src = cs.Dst
n, err = l.WriteTo([]byte("world"), cs, remoteAddr)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.