x/sys/unix: add System V shared memory functions

[virtuald]

Update, June 2 2021: The current proposed API is in #46084 (comment). - rsc

---

What did you expect to see?

It would be mildly useful to have shmget/shmat/shmdt/shmctl available in pure go. System V shared memory isn't very common in modern applications, but there are still uses for it that can't be accomplished otherwise. Usages I'm aware of:

• X11 MIT-SHM extension uses shm to exchange image data
• The American Fuzzy Lop fuzzer uses shm as a mechanism to communicate with the program being fuzzed.

I think the primary question I'd like to have answered is whether inclusion is appropriate for x/sys/unix and how likely it would be for it to be accepted, given that its usage is fairly esoteric.

Prior art

There are several cgo implementations I had found previously, but my google-fu is failing me at the moment. I had proposed an initial API at golang/sys#108 , but hadn't noticed the other pure go prior art in my initial search. There are several mostly pure go implementations:

• https://github.com/gen2brain/shm provides a consistent higher level API for all of its supported OS, only requires cgo on solaris (see syscall: Syscall6 not implemented on Solaris #24357)
• https://github.com/nvi-inc/svipc is linux only and provides raw syscalls only

cc @gen2brain

[virtuald]

It appears the solaris cgo implementation is required because an asmsysvicallN implementation doesn't exist yet. https://go-review.googlesource.com/c/go/+/101135/ tried to add Syscall6 but abandoned for that reason.

... I don't actually care about solaris and would be happy if this were able to go forward without it. :-D

[rsc]

/cc @jayconrod in case fuzzing has run into this as well

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[jayconrod]

I think this would be good to have. I looked for this before deciding how to build our fuzzing shared memory implementation and was sad not to find it. Ultimately, I went with a memory mapped file to minimize platform differences, but it would be nice not to have to write anything to disk.

[virtuald]

Keeping in mind Ian's feedback on my initial PR:

Most additions to this repo are fairly mechanical expressions of underlying Unix concepts, and we just accept them. This is more complex, and I think it ought to go through the proposal process.

If this were to be accepted + implemented, there are probably several ways this could be implemented. I vote to follow gen2brain/shm's approach instead of my original PR, as it's more of a mechanical expression of the underlying API. For reference, here's the original C functions copied from linux glibc:

/* Shared memory control operation.  */
extern int shmctl (int __shmid, int __cmd, struct shmid_ds *__buf) __THROW;

/* Get shared memory segment.  */
extern int shmget (key_t __key, size_t __size, int __shmflg) __THROW;

/* Attach shared memory segment.  */
extern void *shmat (int __shmid, const void *__shmaddr, int __shmflg)
     __THROW;

/* Detach shared memory segment.  */
extern int shmdt (const void *__shmaddr) __THROW;

shmget/shmdt

shmget and shmdt can be straightforward autogenerated wrappers around the syscall in the normal fashion.

shmat

shmat syscall does not provide the size of the returned memory region. gen2brain/shm's implementation of shmat calls the syscall to get a pointer to the memory region, and then calls another syscall to query the size of the region. It then uses the size to construct the byte slice header. Should an x/sys/unix implementation:

1. Use this approach (arguably safer, but at the expense of another syscall)
2. Require the user to pass in the size of the slice to be created and hope they get it right (this is low-level functionality after all?)
3. Provide a means for the user to do both?
4. Return a uintptr and let the caller decide what they want

I prefer 1 or 3. Converting to a byte slice is likely the common case, and follows how mmap works.

shmctl

shmctl can use the normal autogenerated syscall wrappers, but has slightly different control structures for each OS. Should an x/sys/unix implementation:

1. Just autogenerate the appropriate struct for each OS and let users deal with the differences using build tags (this is how gen2brain deals with it)
2. Try to unify the structs with a common interface for features present on all OS, and then have a Sys() function that returns the underlying OS-specific struct (as os.ProcessState does)

I lean towards 1 as I don't think the struct argument is used all that often.

helper functions in gen2brain/shm

There are two really useful helper functions that gen2brain added in addition to the standard sysv API

• Rm: removes the shared memory segment (wrapper for shmctl(IPC_RMID))
• Size: returns the size of the shared memory segment

Given Ian's initial comment, I imagine these two functions would not be adapted to x/sys/unix as they're straightforward to implement if you know how the shm API works.

[rsc]

It would be nice to have a more standard API that hides some of the per-OS variation, like syscall.Mmap and syscall.Munmap.
The gen2brain/shm API looks reasonable, although it would be good if we didn't need fields like PadCgo and GlibcReserved4.

The gen2brain/shm API is:

func At(shmId int, shmAddr uintptr, shmFlg int) (data []byte, err error)
func Ctl(shmId int, cmd int, buf *IdDs) (int, error)
func Dt(data []byte) error
func Get(key int, size int, shmFlg int) (shmId int, err error)
func Rm(shmId int) error
func Size(shmId int) (int64, error)
type IdDs
type Perm

For x/sys we would need more prefixes, like:

func SysvShmAttach(id int, addr uintptr, flag int) (data []byte, err error)
func SysvShmCtl(id, cmd int, desc *SysvShm) (int, error)
func SysvShmDetach(data []byte) error
func SysvShmGet(key, size, flag int) (id int, err error)
type SysvShm
type SysvIpcPerm

The _ds in shmid_ds apparently stands for data structure, so shortening shmid_ds to Shm seems OK.
The Rm and Size helpers seem to be wrappers around shmctl.
The Sysv prefixes are to avoid confusion with POSIX shm_open and shm_unlink if those are ever added
(Those would be PosixShmOpen and PosixShmUnlink.)

Thoughts?

[virtuald]

The proposed API seems mostly fine to me, with two caveats.

The shmid_ds seems to me to be a shorthand for 'descriptor', not 'data structure' -- though I can't find a citation for it either way. In either case, SysvShm seems like a bad name because to me it implies that it is the shared memory itself. It just describes it. How about SysvShmDesc instead?

It seems like most other API in sys follow the unix names pretty closely, so using the Sysv prefix might make it slightly less discoverable. However, it might be Good Enough to just make sure that the unix names show up in the documentation. I don't have a strong objection to it.

Answering my final question then:

Should an x/sys/unix implementation:

1. Just autogenerate the appropriate struct for each OS and let users deal with the differences using build tags (this is how gen2brain deals with it)
2. Try to unify the structs with a common interface for features present on all OS, and then have a Sys() function that returns the underlying OS-specific struct (as os.ProcessState does)

Your API is option number 1. I think it will be the easiest to implement, so I agree.

[rsc]

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

[rsc]

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group