unix: cap RLIMIT_NOFILE soft limit in TestRlimit on darwin

On some machines, kern.maxfilesperproc is 4096. If Rlimit.Cur is larger
than that, Setrlimit will get an errEINVAL.

Same as CL 246658 did in package syscall.

Updates golang/go#40564

Change-Id: I8c45a23352fa2039772e04205680640e8a0e1840
Reviewed-on: https://go-review.googlesource.com/c/sys/+/250000
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Matt Layher <mdlayher@gmail.com>

unix/syscall_unix_test.go

@@ -377,11 +377,11 @@ func TestRlimit(t *testing.T) {
 	}
 	set := rlimit
 	set.Cur = set.Max - 1
-	if runtime.GOOS == "darwin" && set.Cur > 10240 {
-		// The max file limit is 10240, even though
-		// the max returned by Getrlimit is 1<<63-1.
-		// This is OPEN_MAX in sys/syslimits.h.
-		set.Cur = 10240
+	if runtime.GOOS == "darwin" && set.Cur > 4096 {
+		// rlim_min for RLIMIT_NOFILE should be equal to
+		// or lower than kern.maxfilesperproc, which on
+		// some machines are 4096. See #40564.
+		set.Cur = 4096
 	}
 	err = unix.Setrlimit(unix.RLIMIT_NOFILE, &set)
 	if err != nil {
@@ -394,6 +394,9 @@ func TestRlimit(t *testing.T) {
 	}
 	set = rlimit
 	set.Cur = set.Max - 1
+	if runtime.GOOS == "darwin" && set.Cur > 4096 {
+		set.Cur = 4096
+	}
 	if set != get {
 		// Seems like Darwin requires some privilege to
 		// increase the soft limit of rlimit sandbox, though