HOWTO Add a system call

HOWTO Add a system call

This article walks through adding a new system call to NyxOS end to end: reserving the number, implementing the kernel side, exposing a userspace wrapper, and verifying it. The worked example adds SYS_GETUID (number 61), which returns the calling process's user id.

See also: Syscalls, Userspace, Security, Kernel-Data-Structures, Building

Before you start

Requirement	Detail
A working build	make -C kernel succeeds — see Building
Free syscall number	The next unused number; 0–60 are taken as of v6.4.363
SYS_TABLE_SIZE	256 (kernel/core/kernel.h) — the number must be below this

Important

Syscall numbers are defined in two files that must stay in sync: kernel/core/kernel.h (used by the kernel) and user/syscall.h (used by ring-3 programs). Adding the number to only one of them produces a call that silently falls through to the dispatcher's default case and returns an error.

Step 1 — Reserve the number

Add the definition to both headers, at the end of the existing run.

FILE — kernel/core/kernel.h

#define SYS_GETKEYEVENT 56
#define SYS_GETUID      61   /* new */

FILE — user/syscall.h

#define SYS_GETKEYEVENT 56
#define SYS_GETUID      61   /* new */

Warning

Never renumber an existing syscall. Every .elf in the initramfs is compiled against the old numbers, so renumbering silently redirects calls in binaries you did not rebuild.

Step 2 — Implement the kernel side

The dispatcher is one switch in syscall_handler():

FILE — kernel/core/syscall.c

uint64_t syscall_handler(uint64_t no, uint64_t a1, uint64_t a2, uint64_t a3,
                         uint64_t a4, uint64_t a5, uint64_t a6)
{
    switch (no) {
        /* … */
        case SYS_GETUID: {
            process_t* cur = get_cur_proc();
            return cur ? cur->uid : 0;
        }
    }
}

Arguments arrive as a1…a6 and map to the registers documented in Syscalls. The return value is whatever the case returns, in RAX.

Validating user pointers

Any argument that is a pointer must be validated before use. Never dereference a user pointer directly.

Helper	Use
user_ptr_ok(ptr, len)	The buffer [ptr, ptr+len) lies inside the user half
user_str_ok(ptr)	The pointer can hold a NUL-terminated string
copy_from_user(dst, usrc, len)	Copy in, walking the user page tables
copy_to_user(udst, src, len)	Copy out
copy_str_from_user(dst, usrc, maxlen)	Copy a string in
copy_path_from_user(out, outsz, uptr)	Copy a path and resolve it against the caller's cwd

SYS_STAT is the canonical example of all three concerns — validate, copy in, copy out:

FILE — kernel/core/syscall.c (existing code)

case SYS_STAT: {
    /* stat(path, struct stat*) — statbuf = {u32 st_size, st_mode, st_ino}. */
    if (!user_str_ok(a1) || !user_ptr_ok(a2, 12)) return -1;
    char path[MAX_PATH];
    if (copy_path_from_user(path, sizeof(path), a1) != 0) return -1;
    uint32_t size = 0; int isdir = 0;
    if (vfs_stat(path, &size, &isdir) != 0) return -1;
    uint32_t sb[3] = { size, isdir ? (0x4000u | 0755u) : (0x8000u | 0644u), 0 };
    return copy_to_user(a2, sb, sizeof(sb)) == 0 ? 0 : -1;
}

Caution

Skipping validation is a ring-3 → ring-0 arbitrary read or write. Several releases in the v5.9.0-rc* audit series exist because exactly this was missed. See Security.

Constraints inside the handler

Constraint	Consequence
Interrupts are masked (IF=0)	current_idx is stable, so get_cur_proc() is safe
You are on the process's own kernel stack	The call may block; see below
Shared state needs a lock	On SMP, preempt_disable() is not enough — use a spinlock. See SMP
Do not free the current process	It is running on the stack you would free; mark it a zombie instead

Blocking

To block, park the task and yield using the check-then-sleep pattern:

cli();
if (!condition_ready()) {
    cur->state = PROC_BLOCKED;
    sti();
    __asm__ volatile("hlt");
}
sti();

The cli before the test is what makes it atomic against the wakeup — otherwise a wakeup arriving between the test and the hlt is lost forever.

Step 3 — Add the userspace wrapper

Wrappers are static inline in user/syscall.h, so they cost nothing and need no libc rebuild for the symbol.

FILE — user/syscall.h

/* getuid(): the calling process's user id. Always succeeds. */
static inline int getuid(void) {
    return (int)syscall1(SYS_GETUID, 0);
}

Pick the arity helper that matches: syscall1 … syscall6. Document the call in a comment above it — every existing wrapper does, and that comment is the de-facto man page.

Tip

If the call fills a struct shared with the kernel, define the struct in user/syscall.h with a comment stating that its field order must match the kernel's fill order. nyx_dirent_t, nyx_procinfo_t and struct stat all do this.

Step 4 — Build

CODE — Rebuild kernel and all userspace programs

host $ make -C kernel

Important

Because you edited a header, the -MMD dependency files force a rebuild of everything that includes it. If you suspect otherwise, do a clean build — the Makefile has shipped stale kernels before. See Building.

host $ make -C kernel clean && make -C kernel

Step 5 — Write a test program

FILE — user/uidtest.c

#include "libc.h"

int main(void) {
    printf("uid = %d\n", getuid());
    return 0;
}

Add it to the build, following the pattern every other coreutil uses:

FILE — kernel/Makefile

USER_ELFS = … $(USER_DIR)/uidtest.elf

$(USER_DIR)/uidtest.elf: $(USER_DIR)/crt0.o $(USER_DIR)/libc.so $(USER_DIR)/uidtest.o
	$(LD) -nostdlib -m elf_x86_64 -e _start -Ttext 0x10000 -o $@ \
	    $(USER_DIR)/crt0.o --just-symbols=$(USER_DIR)/libc.so $(USER_DIR)/uidtest.o

$(USER_DIR)/uidtest.o: $(USER_DIR)/uidtest.c $(USER_DIR)/libc.h $(USER_DIR)/syscall.h
	$(CC) -std=gnu99 -Os -ffreestanding -nostdlib -m64 -mno-red-zone \
	    -I$(USER_DIR) -c $< -o $@

Regenerate the initramfs so the program is actually present at boot:

CODE — Regenerate the initramfs

host $ python3 tools/mkinitramfs.py kernel/fs/initramfs_data.h c
host $ make -C kernel

Step 6 — Verify

CODE — Run and test

host $ ./run.ps1 -Mode serial

Then in the kernel shell:

nyx> uidtest
uid = 0
[exec] PID 7 exited (code 0)

A bare name that is not a builtin auto-execs /uidtest.elf, so no exec prefix is needed.

Checklist

• Number added to both kernel/core/kernel.h and user/syscall.h
• Number is below SYS_TABLE_SIZE and not reused
• case added to syscall_handler()
• Every pointer argument validated with user_ptr_ok / user_str_ok
• Data moved with copy_from_user / copy_to_user, never by direct dereference
• Shared kernel state protected by a spinlock, not just preempt_disable()
• Wrapper added to user/syscall.h with a documenting comment
• Test program written and added to USER_ELFS
• Initramfs regenerated
• Clean build passes with zero warnings
• Syscalls table updated in this wiki

Troubleshooting

Symptom	Cause	Fix
Call returns −1 immediately, no kernel output	Number missing from kernel/core/kernel.h, so the case never matches	Add it to both headers
Ring-3 #UD on syscall	EFER.SCE not set	This is set by setup_syscall_msrs() at boot; if you moved boot code, restore the order
Kernel #PF inside your handler	You dereferenced a user pointer directly	Use copy_from_user
Data copied out is garbage	Struct layout differs between kernel and user/syscall.h	Make the field orders identical; check for implicit padding
Works on one CPU, corrupts on -smp 4	Unlocked shared state	Take a spinlock; see SMP
Nothing changed after rebuild	Stale build	make -C kernel clean
getdents/getprocs-style call returns partial data	The user buffer had unfaulted lazy-heap pages	memset the buffer in userspace before the call, or use a .bss array

See also

• Syscalls — the full table and the calling convention
• Kernel-Data-Structures — process_t and the per-process fd table
• Security — what the syscall boundary defends against
• HOWTO-Write-a-userspace-program — the other side of the interface

External resources

• System V AMD64 ABI — the calling convention NyxOS follows
• Intel SDM Vol. 2, SYSCALL/SYSRET — Intel