Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net: UDPConn.SetReadBuffer unexpected behavior on BSD architectures #13193

Closed
sparrc opened this issue Nov 9, 2015 · 4 comments

Comments

Projects
None yet
3 participants
@sparrc
Copy link

commented Nov 9, 2015

When using UDPConn.SetReadBuffer, I expect to be able to set the buffer up to the kernel limit kern.ipc.maxsockbuf, but I am getting failures on Darwin and FreeBSD when trying to do that.

From what I can tell, this should be supported, assuming the code is doing a setsockopt on the SO_RCVBUF constant, from the bsd setsockopt man page:

SO_SNDBUF and SO_RCVBUF are options to adjust the normal buffer sizes
     allocated for output and input buffers, respectively.  The buffer size
     may be increased for high-volume connections, or may be decreased to
     limit the possible backlog of incoming data.  The system places an abso-
     lute maximum on these values, which is accessible through the sysctl(3)
     MIB variable ``kern.ipc.maxsockbuf''.

Versions

$ go version
go version go1.5.1 darwin/amd64
$ uname -srm
Darwin 15.0.0 x86_64

and FreeBSD:

$ uname -srm
FreeBSD 10.1-RELEASE amd64

What did you do?

This script fails on any BSD/Darwin architecture, where ReadBuffer is set equal to kern.ipc.maxsockbuf, error output I get is set udp [::]:55647: setsockopt: no buffer space available

// setreadbuffer.go
package main

import (
    "fmt"
    "net"
)

const (
    // Darwin Limit
    // sysctl kern.ipc.maxsockbuf -> 8388608
    ReadBuffer = 8388608 // - (911 * 1024) // (this works if you subtract 911kb)

    // FreeBSD Limit
    // sysctl kern.ipc.maxsockbuf -> 2097152
    // ReadBuffer = 2097152

    // Linux Limit
    // sysctl net.core.rmem_max -> 212992
    // ReadBuffer = 212992 // (this works on Linux)
)

func main() {
    addr, _ := net.ResolveUDPAddr("udp", ":80891")

    conn, err := net.ListenUDP("udp", addr)
    if err != nil {
        fmt.Printf("Failed to set up UDP listener at address %s: %s", ":8089", err)
        panic(err)
    }
    defer conn.Close()

    err = conn.SetReadBuffer(ReadBuffer) // THIS FAILS
    if err != nil {
        fmt.Printf("Failed to set UDP read buffer to %d: %s",
            ReadBuffer, err)
    }
}

What did you expect to see?

I would expect conn.SetReadBuffer(ReadBuffer) to succeed when ReadBuffer is less than or equal to kern.ipc.maxsockbuf

What did you see instead?

On BSD/Darwin, conn.SetReadBuffer(ReadBuffer) fails when ReadBuffer is equal to kern.ipc.maxsockbuf. From my limited testing it doesn't start passing until you subtract ~12% from the max.

@ianlancetaylor ianlancetaylor changed the title UDPConn.SetReadBuffer unexpected behavior on BSD architectures net: UDPConn.SetReadBuffer unexpected behavior on BSD architectures Nov 9, 2015

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Nov 9, 2015

Can you test what happens with the equivalent C code?

The Go code really is just calling setsockopt(SO_RCVBUF, bytes); I don't know why Go would behave differently than C here.

@ianlancetaylor ianlancetaylor added this to the Unplanned milestone Nov 9, 2015

@sparrc

This comment has been minimized.

Copy link
Author

commented Nov 9, 2015

I can test but I'm not very good with C, could you provide any guidance on how I might call the C function directly?

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Nov 9, 2015

Try this. READBUFFER is the size of the read buffer being requested. By the way, in your Go example, you should check the error return of ResolveUDPAddr: ":80891" is not a valid UDP address.

#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>

#define READBUFFER 8388608

static void die(const char *s) {
  fprintf(stderr, "%s: %s\n", s, strerror(errno));
  exit(EXIT_FAILURE);
}

int main() {
  int s;
  struct sockaddr_in sin;
  int size;

  s = socket(AF_INET, SOCK_DGRAM, 0);
  if (s < 0) {
    die("socket");
  }
  memset(&sin, 0, sizeof sin);
  sin.sin_port = htons(8891);
  if (bind(s, (struct sockaddr *)(&sin), sizeof sin) < 0) {
    die("bind");
  }
  size = READBUFFER;
  if (setsockopt(s, SOL_SOCKET, SO_RCVBUF, &size, sizeof size) < 0) {
    die("setsockopt");
  }
  return 0;
}
@sparrc

This comment has been minimized.

Copy link
Author

commented Nov 9, 2015

Okay, @fenderle helped me figure this out, it looks like BSD makes some sort of adjustment to the actual limit here:

https://github.com/freebsd/freebsd/blob/master/sys/kern/uipc_sockbuf.c#L385-L399

which gets triggered from here:

https://github.com/freebsd/freebsd/blob/master/sys/kern/uipc_sockbuf.c#L420

so the actual buffer limit is not 8388608, it's actually

sb_max_adj = (u_quad_t)sb_max * MCLBYTES / (MSIZE + MCLBYTES);
8388608 * (1 << 11) / (256 + (1 << 11))  =  8388608 * (2048) / (2304)  = 7456540

(where 8388608 == kern.ipc.maxsockbuf)

So I think we can close this issue....I don't think it's within Go's domain to adjust for this, BSD is weird.

@golang golang locked and limited conversation to collaborators Nov 10, 2016

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.