Proposal: x/sys/unix: autogenerate ioctl API surface for linux

[danderson]

This proposal is coming out of the discussion on https://go-review.googlesource.com/c/sys/+/150321 .

Problem statement

I'm implementing a PPP client in Go, which involves talking to Linux's PPP and PPPoE kernel drivers. Both are driven through ioctls on fds. This has led to me having to make a couple of changes to x/sys/unix, to add more ioctl request codes as well as new Ioctl* getters and setters.

The current API for ioctls in x/sys/unix is:

• An unexported "raw" ioctl function that takes a uintptr value: func ioctl(fd int, req uint, value uintptr) error
• Exported type-safe implementations that require specific argument types, and cast to uintptr under the hood, e.g. func IoctlSetInt(fd int, req uint, value int) error or fund IoctlSetTermios(fd int, req uint, value *Termios).

The overall aim of this API is to make ioctl invocation type safe, or at least as type safe as possible. The current incarnation only partially succeeds: the ioctl request code is still exposed to the user, meaning they are free to invoke an ioctl using the wrong "type helper", e.g. passing an int into an ioctl that expects a *Termios. It's still a benefit over the fully raw ioctl call, in that you have to explicitly type the wrong parameter name into your editor (and hopefully realize the discrepancy at that point), but it's not ideal.

Proposal

I propose defining an ioctl API surface where one function maps to a single ioctl request code, and thus the functions can be completely type-safe. With the public API, it would become impossible to invoke an ioctl with the wrong parameters, or invoke a write ioctl using a read idiom.

The obvious problem with this approach is that it's a lot of busywork to define all those boilerplate functions. The obvious answer is to autogenerate them.

At least on Linux, this is somewhat straightforward: the kernel headers uses a set of macros to facilitate the definition of ioctl request codes. These macros carry the information needed to generate a strongly typed wrapper for every ioctl in the kernel API surface.

As a randomly selected example, include/uapi/linux/sed-opal.h has the following definition:

#define IOC_OPAL_SET_PW             _IOW('p', 224, struct opal_new_pw)

When evaluated through the C preprocessor, the macro evaluates to the ioctl request number. mkerrors.sh already demonstrates how to pull those definitions out into Go definitions.

When not evaluated, this macro invocation tells us a few more useful things: _IOW means the ioctl is a write ioctl (userspace-to-kernel), and that its parameter is a struct opal_new_pw. With that additional information, we can autogenerate:

func IoctlOpalSetPw(fd int, value *OpalNewPw) error {
  return ioctl(fd, 0x12345678, uintptr(unsafe.Pointer(value)))
}

Pros/Cons

The pro for me is that I can do a little bit more work once, and not have to send another CL to x/sys/unix each time I discover a new strange ioctl.

It would make ioctl invocation completely type-safe for users of x/sys/unix on linux.

In opposition, I found #14873 , in which r objects to making ioctl nice. The existing API is already going against his wishes, so I'm assuming there was another unrecorded discussion which shifted things.

This will also bloat the x/sys/unix implementation quite a bit. If we autogenerate all read and write ioctls, that adds >500 functions. We could follow current standard practice for x/sys, and only add new ioctl definitions when people want to use a particular subsystem, but that's still a large chunk of the x/sys/unix API surface. Another option might be to define a subpackage for ioctl, which is entirely autogenerated. That way the main x/sys API surface isn't polluted by ioctl, and invocations look like ioctl.OpalSetPw(...), which is still nice. That may get a bit tricky because this hypothetical new package would have to define all the structs and whatnots that the ioctl calls want, or reference their definitions in the main unix package.

It makes the linux implementation of x/sys/unix diverge from that of other unix derivatives. Currently the ioctl API is relatively uniform across unix variants, but this change would pull linux away. Maybe this is fine since x/sys/unix is completely OS dependent anyway. Maybe there's similar tricks we can play with the BSDs and OpenSolaris family tree, but I don't know their API conventions well enough to know.

[danderson]

cc @ianlancetaylor , since this braindump comes from his explanation of the current ioctl API surface in a code review.

[ianlancetaylor]

I like the idea of an x/sys/ioctl package (or perhaps x/sys/unix/ioctl if people prefer). That package can import x/sys/unix. If the required types are not defined in x/sys/unix, don't auto-generate the ioctl. That makes an easy two step process where required. And eventually we'll handle all the ioctls (right? There is a finite number?).

I would definitely like to know whether this is possible to do on other systems. I would hope that it is.

CC @robpike (because of comments on #14873)

[danderson]

There is a finite number of ioctls in the linux API surface, although perhaps not a static number. In theory, the ioctl surface should no longer be growing in linux, as netlink is mandated for new "I just need to do one custom little thing to my kernel object" use cases.

[robpike]

I believe that the time for me to argue that the 35-year-old ioctl API is ugly has passed, and with so many more Go programmers in the wold now, attempting to hold back the tsunami of awfulness of modern Unix by blocking it with a finger in the ioctl dam is not an effective strategy.

Since this is not part of x/sys/unix itself, but a separate package, the horror can be contained. In fact, one concession to my sense of decorum would be not to make it x/sys/unix/ioctl but x/sys/unix/linux, x/sys/unix/freebsd, etc., and locate the truly system-dependent failures of design in the corresponding subdirectories. This not only avoids the +build dance, but also means code that uses it announces its nonportability.

[tklauser]

I would definitely like to know whether this is possible to do on other systems. I would hope that it is.

The BSDs (FreeBSD, NetBSD, OpenBSD, Dragonfly) and also Darwin all define their ioctls in a similar way as Linux does, i.e. (again a completely arbitrary example)

#define TIOCGETA        _IOR('t', 19, struct termios) /* get termios struct */
#define TIOCSETA        _IOW('t', 20, struct termios) /* set termios struct */

So for these systems is seems to be possible as well to auto-generate.

Solaris unfortunately seems to define its ioctls in another way without type information:

#define TCGETA  (_TIOC|1)
#define TCSETA  (_TIOC|2)

I'm not sure about AIX.

[krytarowski]

NetBSD generates the tables of ioctl operations independently for ktrace(1) and sanitizers.

https://github.com/llvm-mirror/compiler-rt/blob/master/utils/generate_netbsd_ioctls.awk

The problem is that these operations are sometimes with duplicated numbers for different devices. We are using a hack to ignore the one probably less important and assume the other one.

Another issue is that people must regenerate the table frequently, as ioctl operations quickly come and depart. There are around 1200 of ioctl operations generated for sanitizers from NetBSD.

Another issue is that some ioctls are for internal use, and thus there is no access in userland headers to exact types.

[rsc]

500+ new functions seems like a bit much. The syscall (or x/sys/unix) package is unsafe in many ways. It's not like this would eliminate all of them.

[rsc]

No one has replied to my previous comment (500+ new functions seems like a bit much). That still seems true to me. The original reporter no longer has this need either. Declining.

[danderson]

I no longer have an immediate need, but I do many strange things to kernels from Go, so I'm sure it'll come up again. I didn't reply just because life got in the way - unfortunately this isn't my full time job :).

500 functions is much, which is why I suggested isolating to its own package. People who need to use ioctl will still have a better experience if the experience is "wade through a large function index" rather than "hope I don't screw up the unsafe casts and GC pinning to make this call, and wait on a PR round-trip to x/sys/unix to add my weird thing."

That said, if it comes to that... I can just go and implement this myself outside of x/sys/unix. It's a pity because there's a bunch of autogen machinery already in x/sys that I'll have to copy and diverge, but it's workable.

I guess this is my closing plea for "please reconsider", or failing that it's a marker for the next person to go see if I implemented this as a standalone package outside x/sys/unix :)