Security

Security

NyxOS is a hobby operating system and offers no security guarantees. It has never been audited by anyone outside the project, and it should not be trusted with anything that matters. What follows is an honest description of the boundaries that do exist, and of the ones that do not.

See also: Syscalls, Memory-Management, Process-Management, Cryptography-and-TLS, Architecture

Privilege boundaries

Ring	Runs	Can reach
0	Scheduler, drivers, VFS, network stack, compositor	All physical memory; user memory only through copy_from_user/copy_to_user
3	ELF programs	Its own mapped pages, and the kernel only through syscall

Memory protection

Mechanism	Effect
Page-table isolation	The kernel occupies the top PML4 entry; user page tables carry no identity mapping of physical memory, so a process cannot name kernel memory even by guessing
NX	User stack, heap and data pages are mapped non-executable
SMEP	Ring 0 cannot execute a ring-3 page — enabled at boot by enable_smep_smap()
SMAP	Ring 0 cannot access a ring-3 page outside an explicit window — also enabled at boot
CR0.WP	Supervisor writes to read-only pages fault; required for copy-on-write correctness
Guard page	Below the user stack's growth ceiling, so an overflow faults instead of running into other mappings
Page 0 unmapped	USER_SPACE_MIN is 0x1000, so a NULL dereference always traps

Syscall boundary

This is the only path from ring 3 into the kernel and is where most hardening lives.

• Pointer validation. user_ptr_ok() rejects anything outside [0x1000, 0x800000000000).
• Opaque file descriptors. Userspace gets small integers indexed into a per-process table; kernel VFS handles are never exposed. Before this, ring 3 received raw kernel pointers — a combined information leak and arbitrary-kernel-read/write reachable from any program.
• copy_from_user / copy_to_user. These walk the user page tables to a physical address and bounce through the higher-half alias, rather than dereferencing a user pointer directly. Since v6.4.14 they are page-granular — they translate once per page instead of once per byte, so a large copy is both faster and validated a page at a time.
• Interrupts masked. Syscalls run with IF=0.
• Per-CPU entry state. Saved ring-3 registers live in a per-CPU block, so two cores entering the kernel simultaneously cannot clobber each other.
• Flags sanitised. Every interrupt and syscall entry path issues cld. Ring 3 could otherwise leave RFLAGS.DF set and reverse the direction of every string operation the kernel performs.
• ELF validation. Headers and program headers are checked before anything is mapped; libseg_load (the shared-library path) once validated nothing at all.

Fault isolation

A ring-3 fault does not panic the kernel. Page faults, divide errors and illegal instructions are converted into SIGSEGV, SIGFPE and SIGILL and delivered to the faulting process. With a handler installed the program can recover — with sigsetjmp/siglongjmp it can survive repeated faults; without one it dies with status 128 + signo and the system carries on.

Authentication

Accounts live in /etc/passwd on the EXT2 disk, one line per user:

user:salt_hex:iterations:hex_hash:avatar

Property	Value
KDF	PBKDF2-HMAC-SHA256
Iterations	10 000, stored per entry so the work factor can be raised without breaking existing accounts
Salt	Random 8 bytes per user, generated from RDTSC jitter, the tick counter and the RTC
Minimum password length	4 (AUTH_MIN_PASS)
Failed-login lockout	3 attempts (LOGIN_MAX_ATTEMPTS), then a cooldown

Accounts persist across reboots and can be created from the login screen. With no EXT2 disk attached, an in-memory fallback account is used so the machine still boots.

This is not strong authentication. The salt generator is not a CSPRNG, 10 000 PBKDF2 iterations is low by modern standards, and the disk is unencrypted — anyone with the image has the hashes.

TLS

The TLS 1.2 stack verifies certificate chains against pinned trust anchors, checks hostnames against subjectAltName, checks validity dates, and verifies the ServerKeyExchange signature. In strict mode (tlsstrict on) a handshake that fails any of those checks is refused rather than warned about.

It is off by default, it has five bundled trust anchors rather than a real root store, and none of the cryptographic implementations have been reviewed by anyone else. Constant-time behaviour is not claimed. See Cryptography-and-TLS.

What is not defended

Being explicit about this is more useful than a list of features:

• No IOMMU. A malicious PCI device has full DMA access.
• No Spectre, Meltdown or MDS mitigations. No retpolines, no KPTI, no speculation barriers.
• No ASLR. Programs load at fixed addresses; shared libraries are at fixed addresses by design, which is why the dynamic loader performs no relocation.
• No stack canaries. -fno-stack-protector is a build flag.
• Partial filesystem permissions. Mode bits are stored, and read-only is now enforced on writes (v6.4.352), but there is still no per-user ownership model, so this is a safety rail, not access control.
• No process privilege model. There is no root/non-root distinction after login, no capabilities, no setuid.
• Minimal network hardening. The TCP/IP stack is not defended against malformed or adversarial packets.
• No audit or side-channel review of the cryptographic code.
• No secure boot, no disk encryption.

Ongoing work

The v5.9.0-rc* release series was a systematic security and correctness audit; each release in it closes a specific confirmed finding, including a ring-3 → ring-0 arbitrary write, missing ELF validation, a double free in dup(), and ring-3 flags surviving into the kernel. v5.9.0-LTS is the snapshot at the end of that pass. See Version-History.

The v5.9.100+ general-maintenance pass

After the feature work stopped at v5.9.99, a standing maintenance sweep began — one file or subsystem per release, each with its self-tests re-run. It has closed several defects that matter for robustness against untrusted input:

Release	Finding
v5.9.104	Remote DoS. An unchecked kmalloc in the TCP receive path let a peer NULL-dereference and crash the kernel under memory pressure. 81 kmalloc sites were audited; two were real
v5.9.105	get_phys_addr leaked the NX bit into the returned physical address — a booby-trap for the next caller (see Memory-Management)
v5.9.106	The X.509 date parser accepted 'Z' in any position, passing a malformed certificate Time (see Cryptography-and-TLS)
v5.9.108	A snprintf-return over-read in the HTTP request builder could stream kernel stack to a server, reachable via a malicious redirect Location (see Networking-Stack)
v5.9.109	A failed cross-directory rename left the source half-renamed (see Filesystem)
v5.9.110	Heap fragmentation could fail a large allocation with ample free memory
v5.9.111–112	TCP receive-path correctness — data+FIN handling and an in-order sequence gate
v5.9.200	Remotely-triggerable heap buffer overflow in the HTTP chunked-transfer decoder (net/http.c), reachable by any server Selene fetches from
v5.9.201	Kernel stack buffer overflow in the DNS query-name encoder (net/dns.c), now bounded

The pass also made the build 100 % warning-clean (C and NASM), added -Wshadow permanently, and ran -Wsign-conversion file-by-file over the untrusted parsers (DER, the image decoders, HTTP). The image-decode surface a web page can reach — inflate/PNG/BMP/GIF/JPEG — was audited and found bounds-sound.

Note

None of these turn NyxOS into a hardened system. They remove specific, identified footguns; the what-is-not-defended list above still stands in full.

The v6.4 hardening-and-SMP pass

Alongside the daily-driver userland work, a sustained robustness pass ran from v6.4.44 through v6.4.181. Its throughline: everything that parses untrusted input or is reachable from another core got tightened, and each fix was locked with a self-test or KAT so it cannot silently regress.

Release	Finding
v6.4.44	xbm path-traversal + cc-arg-injection. The package manager now validates package/recipe names before compiling them (see [[Toolchain#packages--xbm
v6.4.54	DNS response spoofing. The resolver now uses a random CSPRNG transaction ID per query and rejects responses whose ID does not match (see Networking-Stack)
v6.4.62	Image-decoder reject self-test — PNG/BMP/GIF/JPEG proven to refuse empty, garbage and over-sized input
v6.4.69	X.509 basicConstraints bypass. A non-CA certificate is now rejected as a chain issuer, so a valid end-entity leaf cannot sign for another host (see Cryptography-and-TLS)
v6.4.76	Audited the HTTP response parser now that xbm feeds it untrusted server data
v6.4.83	Physical page allocator rejects non-aligned frees, double-frees and incref of a free frame (see Memory-Management)
v6.4.84	Kernel heap: krealloc keeps the block on OOM; kfree rejects unknown-magic headers
v6.4.85	VFS path resolution fails on a missing intermediate directory instead of acting on a parent
v6.4.86	TCP checksum validated on receive — a segment with a bad checksum is dropped, not acted on
v6.4.87	/proc made SMP-safe (a real spinlock, not preempt_disable, which only stops the local core)
v6.4.88–90	TCP close handshake completed (FIN_WAIT/LAST_ACK reach CLOSED), the connection table serialised under a spinlock, and a retransmit queue added
v6.4.91	VFS fd handles are pool indices, not truncated node pointers (see Filesystem)
v6.4.97	PNG decoder validates every chunk's CRC-32
v6.4.106	The password KDF (PBKDF2-HMAC-SHA256) locked with a KAT
v6.4.109	DNS response parser hardened and locked with an adversarial KAT
v6.4.123	Strict ipv4_parse (+ KAT) — no more lax address acceptance
v6.4.126	Overflow-checked integer parsers (numparse) underneath the rest
v6.4.148	File-name validation on touch/mkdir — strict UTF-8, no control characters
v6.4.151	Terminal CSI parser bounded against numeric-parameter overflow
v6.4.156	Constant-time ct_memcmp on the crypto/TLS verify paths
v6.4.161	One hardened URL parser — bounds the port, rejects an over-long host
v6.4.162	Hardened glob matcher — no catastrophic backtracking (+ KAT); the shell uses it (v6.4.172)
v6.4.165	Strict RFC 4291 IPv6 address parser (+ KAT)
v6.4.181	DHCP option parsing behind one bounds-checked walker (+ KAT)
v6.4.234	Oversized ls/auth buffers moved off the 4 KB kernel stack; GIF's 16 KB LZW dictionary too (v6.4.233) — a stack-overflow class (#68)
v6.4.237	xbm package SHA-256 integrity manifests + verify (tamper detection)
v6.4.240	Login brute-force lockout — rate-limit failed logins (+ KAT)
v6.4.243	Kernel task-stack overflow canary + stackcheck (+ KAT)
v6.4.285	kfree poisons a freed block's header so a double-free is dropped, not re-coalesced
v6.4.298	KAT the syscall pointer/length validators (user_ptr_ok/user_str_ok)
v6.4.312	secure_zero (explicit_bzero) — the login password is wiped from memory after use
v6.4.341	/dev/urandom served from the CSPRNG, not a tick-seeded xorshift (#79)

Security reports go to the address in SECURITY.md at the repository root.

See also

• Syscalls - the hardened boundary
• Memory-Management - page-level protections
• Cryptography-and-TLS - the TLS trust model and its limits
• Contributing - how to report a vulnerability

External resources

• NyxOS security policy - repository
• Intel SDM Vol. 3A, Ch. 4.6 - Access Rights (SMEP/SMAP) - Intel