Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

syscall: Setrlimit behavior changed on Go 1.12 on macos #30401

Open
karalabe opened this issue Feb 26, 2019 · 10 comments

Comments

Projects
None yet
8 participants
@karalabe
Copy link
Contributor

commented Feb 26, 2019

We've noticed a while back that when setting the file descriptor allowance on macos, Go restricts it to a lower value than requested:

package main

import (
	"fmt"
	"syscall"
)

func main() {
	// Get the current file descriptor limit
	var limit syscall.Rlimit
	if err := syscall.Getrlimit(syscall.RLIMIT_NOFILE, &limit); err != nil {
		panic(err)
	}
	// Try to update the limit to the max allowance
	limit.Cur = limit.Max
	if err := syscall.Setrlimit(syscall.RLIMIT_NOFILE, &limit); err != nil {
		panic(err)
	}
	// Try to get the limit again and see where it's at
	if err := syscall.Getrlimit(syscall.RLIMIT_NOFILE, &limit); err != nil {
		panic(err)
	}
	fmt.Println(limit.Cur)
}

The above code on Linux works as expected, setting the limit to the max returned by the syscall package.

On macos with Go 1.11.5 (and prior) it silently set it to the minimum of limit.Max and 10240. We had to patch our code to explicitly check the actual value set instead of relying on no-error-returned since we falsely alloted more fds to databases than available.

On macos with Go 1.12, our code again started to blow up, now returning invalid argument errors instead of the silent capping previously.

I'd probably say both behaviors are bad, but the new one is a lot worse than the previous. Now it only tells us it's invalid, but unless you know the magic number defined in the your own kernel version, there's no way for user code to figure out how many fds are actually available.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Feb 26, 2019

@randall77

This comment has been minimized.

Copy link
Contributor

commented Feb 26, 2019

Yes, this is unfortunate. We switched to using the implementation of getrlimit in libSystem.dylib instead of raw system calls, and that's the behavior we get. In particular, the new implementation lies to us about the maximum value available.
I had to modify the TestRlimit with code like this:

	if runtime.GOOS == "darwin" && set.Cur > 10240 {
		// The max file limit is 10240, even though
		// the max returned by Getrlimit is 1<<63-1.
		// This is OPEN_MAX in sys/syslimits.h.
		set.Cur = 10240
	}

I suppose we could make our implementation of Getrlimit incorporate this fix internally. But my view of the syscall package is that you get what the OS provides, warts and all. And one day Apple may fix this issue (or change OPEN_MAX) and then any patch we have is obsolete.

You can always binary search for the largest allowable value. Yuck.

@randall77

This comment has been minimized.

Copy link
Contributor

commented Feb 26, 2019

Note that this is not just a Go problem - C people run into similar issues:

https://postfix-devel.postfix.narkive.com/kVL334x7/mac-os-x-and-setrlimit-2

@karalabe

This comment has been minimized.

Copy link
Contributor Author

commented Feb 27, 2019

Ah yes, I wasn't particularly complaining that Go is doing something wrong (given the restrictions), just trying to figure out what the best solution is. I guess my main complaint wasn't that Go works one way or the other, rather that it changed it behavior, breaking existing code.

We can fix it to handle both cases (i.e. hard code that magic number), just would like to know which solution will Go stay with in the end so we don't have to do a 3rd fixup round for Go 1.12.1 :)

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Feb 27, 2019

Going forward on Darwin Go is always going to call the C function.

@jeffallen

This comment has been minimized.

Copy link
Contributor

commented Mar 7, 2019

I found that on Go 1.11.5, this code:

func init() {
	raiseFdLimit = func() {
		var rLimit syscall.Rlimit
		err := syscall.Getrlimit(syscall.RLIMIT_NOFILE, &rLimit)
		if err != nil {
			log.Fatal("Error Getting Rlimit ", err)
		}

		if rLimit.Cur < rLimit.Max {
			rLimit.Cur = rLimit.Max
			err = syscall.Setrlimit(syscall.RLIMIT_NOFILE, &rLimit)
			if err != nil {
				log.Warn("Error Setting Rlimit:", err)
			}
		}

		err = syscall.Getrlimit(syscall.RLIMIT_NOFILE, &rLimit)
		log.Info("File descriptor limit is:", rLimit.Cur)
	}
}

reports File descriptor limit is: 24576. On Go 1.12 it reports invalid argument. So I've added the workaround from above (#30401 (comment)) but instead of 10240, I've used 24576 and that returns the behavior of my server compiled with Go 1.12 to the same as it was on Go 1.11.5. (@randall77: MacOS version 10.14.3 (18D109))

The manpage says: setrlimit() now returns with errno set to EINVAL in places that historically succeeded. It no longer accepts "rlim_cur = RLIM_INFINITY" for RLIM_NOFILE. Use "rlim_cur = min(OPEN_MAX, rlim_max)".

I searched for OPEN_MAX in /Application/Xcode.app and found that it is 10240, even though the observed behavior is 24576 works. 🤷‍♂️

@randall77

This comment has been minimized.

Copy link
Contributor

commented Mar 7, 2019

@jeffallen What Mac OS are you running?
I can't get ask for more than 10240 on Sierra (10.12.6) without getting an invalid argument error.

@jeffallen

This comment has been minimized.

Copy link
Contributor

commented Mar 7, 2019

Here is an excellent explanation: https://unix.stackexchange.com/questions/350776/setting-os-x-macos-bsd-maxfile

It says this could be a problem on FreeBSD and OpenBSD as well.

Another useful reference, showing that reading the setrlimit manpage and paying any attention whatsoever to OPEN_MAX would be not a good idea: https://julio.meroh.net/2019/01/open-files-limit-macos-and-the-jvm.html

@bep

This comment has been minimized.

Copy link
Contributor

commented Apr 6, 2019

For what it's worth;

On my Mojave 10.14.4 on Go 1.12.2, my tests tell me that I can set values in Setrlimit <= the value of limit as defined in /Library/LaunchDaemons/limit.maxfiles.plist (I assume that 10240 is the default?).

This can be set for the current session with something ala:

sudo launchctl limit maxfiles 1234 200000

In the above example, I get invalid argument for any argument to Setrlimit> 1234.

If I run Setrlimit as sudo, I can use any value.

@andybons andybons modified the milestones: Go1.12.3, Go1.12.4, Go1.12.5 Apr 8, 2019

@andybons andybons modified the milestones: Go1.12.5, Go1.12.6 May 6, 2019

grailbot pushed a commit to grailbio/reflow that referenced this issue May 13, 2019

reflow: Make raising of FD limit work on OS X/Go 1.12
Summary:
This commit implements a workaround to an issue [[ golang/go#30401 |  introduced in Go 1.12 ]].

The behavior of `Setrlimit` changed to no longer succeed when the requested limit exceeded the maximum.  (It used to succeed, clamping the value to the allowed maximum).

Test Plan: Manual testing

Reviewers: smahadevan, pgopal, marius

Reviewed By: marius

Maniphest Tasks: T17776

Differential Revision: https://phabricator.grailbio.com/D27211

fbshipit-source-id: 8be13f0
@dmitshur

This comment has been minimized.

Copy link
Member

commented Jun 6, 2019

This issue was originally filed under the Go1.12.1 milestone with NeedsInvestigation and release-blocker labels. Many point releases have been made since then, and this issue was continuously moved to the next point milestone.

Since there's no fix yet and it's not blocking the 1.12 point releases, I'll move this to Go 1.13 milestone so we can look at it again for 1.13, and determine if it should block that release or not.

@dmitshur dmitshur modified the milestones: Go1.12.6, Go1.13 Jun 6, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.