x/sys/unix: IoctlGetInt broken on 64-bit big-endian platforms

[greatroar]

What version of Go are you using (go version)?

$ go version
go version go1.20.3 linux/amd64
$ go list -m golang.org/x/sys
golang.org/x/sys v0.8.1-0.20230523194307-b5c7a0975ddc

What operating system and processor architecture are you using (go env)?

GOARCH=s390x
GOOS=linux

(using QEMU on an amd64 host)

What did you do?

package main

import (
	"fmt"
	"os"

	"golang.org/x/sys/unix"
)

func main() {
	tty, err := os.OpenFile("/dev/tty", os.O_RDWR, 0)
	if err != nil {
		panic(err)
	}
	pgid, err := unix.IoctlGetInt(int(tty.Fd()), unix.TIOCGPGRP)
	fmt.Println(pgid, unix.Getpgrp())
}

What did you expect to see?

Same number twice, as on amd64:

$ go run pgrp.go
101981 101981

What did you see instead?

$ GOARCH=s390x go run pgrp.go
438060894388224 101994

The first number is 101994<<32. I believe this happens because IoctlGetInt passes a pointer to a Go int to the system call:

func IoctlGetInt(fd int, req uint) (int, error) {
        var value int
        err := ioctlPtr(fd, req, unsafe.Pointer(&value))
        return value, err
}

The ioctl system call expects a pointer to a C int. Passing a Go int means it stores the pgid in the upper 32 bits.

Note that IoctlSetPointerInt passes a pointer to an int32:

func IoctlSetPointerInt(fd int, req uint, value int) error {
        v := int32(value)
        return ioctlPtr(fd, req, unsafe.Pointer(&v))
}

[bcmills]

(CC @tklauser @golang/s390x @golang/ppc64)

[greatroar]

I guess the underlying issue was already discovered, #45585.

[greatroar]

I also wasn't aware of IoctlGetUint32. That solves my original problem (emulating tcgetpgrp), so the only remaining issue is documentation: IoctlGetInt is inconsistent with IoctlSetPointerInt, which is needed to emulate tcsetpgrp.

[mknyszek]

Just quickly skimming this, I think that indeed it would be sufficient to just document this. Though, we should also just fix GetInt on the platforms where it's broken. (Document the "int" discrepancy and then do a conversion inside GetInt on the affected platforms.)

Would you be willing to send a CL for this? :) (https://go.dev/doc/contribute)