Skip to content

Format Reference

Jump to bottom
kazah-png edited this page Jul 27, 2026 · 1 revision

Format reference

Byte-level layout of every wire protocol, on-disk structure and file format NyxOS parses or produces. Offsets are decimal byte offsets from the start of the structure unless stated otherwise.

See also: Networking-Stack, Filesystem, Cryptography-and-TLS, Image-Decoders, Userspace

Byte order

Caution

Every field that crosses the network is stored in network byte order (big-endian), funnelled through htons / htonl / ntohs / ntohl. This is the single largest source of bugs in this subsystem's history: an IP header checksum stored little-endian made the host drop every frame, which in turn masked the same mistake in the TCP header, DNS question counts and ARP addresses — none of which could be observed until the checksum was fixed. If you touch a header field, check the endianness.

On-disk EXT2 structures and every image format below are little-endian, except JPEG, which is big-endian.

Network protocols

Ethernet II frame

Offset Size Field
0 6 Destination MAC
6 6 Source MAC
12 2 EtherType
14 n Payload
EtherType Protocol
0x0800 IPv4
0x0806 ARP

Minimum frame size is 60 bytes excluding FCS, so short frames are padded.

Warning

That padding is counted by the NIC, not by IP. Deriving payload_len from the frame length rather than the IP header's total_length adds phantom bytes — which produced a wrong TCP ACK and an immediate RST.

ARP packet

Offset Size Field NyxOS value
0 2 Hardware type 0x0001 Ethernet
2 2 Protocol type 0x0800 IPv4
4 1 Hardware address length 6
5 1 Protocol address length 4
6 2 Operation 1 request, 2 reply
8 6 Sender MAC
14 4 Sender IP
18 6 Target MAC
24 4 Target IP

Cache: 16 entries with TTL (kernel/net/arp.c).

IPv4 header

Offset Size Field Notes
0 4 bits Version 4
0 4 bits IHL Header length in 32-bit words; 5 = 20 bytes
1 1 DSCP / ECN
2 2 Total length Header plus payload — the authoritative length
4 2 Identification
6 3 bits Flags
6 13 bits Fragment offset NyxOS neither fragments nor reassembles
8 1 TTL
9 1 Protocol
10 2 Header checksum
12 4 Source address
16 4 Destination address
Protocol Value
ICMP 1
TCP 6
UDP 17

Checksum. One's-complement sum of the header as 16-bit words with the checksum field zeroed, then complemented. A correct header verifies to 0xFFFF.

Routing. Off-subnet destinations go via the gateway rather than being ARP'd directly. Loopback (127.0.0.0/8) is delivered internally and skipped when auto-selecting a source address.

ICMP header

Offset Size Field
0 1 Type
1 1 Code
2 2 Checksum
4 2 Identifier
6 2 Sequence number
8 n Payload
Type Meaning
0 Echo reply
8 Echo request

NyxOS answers type 8 with type 0. ping sends 4 requests and reports RTT and loss.

UDP header

Offset Size Field
0 2 Source port
2 2 Destination port
4 2 Length — header plus data
6 2 Checksum

TCP header

Offset Size Field
0 2 Source port
2 2 Destination port
4 4 Sequence number
8 4 Acknowledgement number
12 4 bits Data offset, in 32-bit words
12 12 bits Reserved and flags
14 2 Window
16 2 Checksum
18 2 Urgent pointer
20 n Options, then payload

Flags

kernel/net/tcp.h

Constant Value
TCP_FLAG_FIN 0x01
TCP_FLAG_SYN 0x02
TCP_FLAG_RST 0x04
TCP_FLAG_PSH 0x08
TCP_FLAG_ACK 0x10

The data offset and flags share one 16-bit field, stored as offset_flags in network order.

Checksum pseudo-header

The TCP checksum covers a 12-byte pseudo-header before the real one:

Offset Size Field
0 4 Source address
4 4 Destination address
8 1 Zero
9 1 Protocol (6)
10 2 TCP length

Connection states

Constant Value
TCP_STATE_CLOSED 0
TCP_STATE_SYN_SENT 1
TCP_STATE_SYN_RCVD 2
TCP_STATE_ESTABLISHED 3
TCP_STATE_FIN_WAIT1 4
TCP_STATE_FIN_WAIT2 5
TCP_STATE_CLOSE_WAIT 6
TCP_STATE_LAST_ACK 7
TCP_STATE_TIME_WAIT 8
TCP_STATE_LISTEN 9

Retransmission

Constant Value Meaning
TCP_RTO_INITIAL 300 ms Before the first retransmit
TCP_RTO_MAX 2400 ms Cap after exponential backoff
TCP_MAX_RETRIES 5 Then reset the connection

One outstanding segment is buffered verbatim (rt_seg) and cleared by the cumulative ACK that passes rt_ack_seq. HTTP is request/response with a single segment in flight, so one slot suffices.

Connection table

TCP_MAX_CONNS is 32. A bidirectional loopback session costs two slots — the client's tcp_connect side and the server's tcp_accept side — plus one for the listener, so N concurrent clients to one service need 1 + 2N slots.

Note

At the previous value of 8, four concurrent socket processes overflowed the table and connect() failed. That presented as a "concurrent multi-process networking" Heisenbug for several releases, because a stale build hid the fix. 32 supports roughly 15 concurrent sessions, and conns[] holds only pointers to per-connection buffers, so a larger table costs almost no static memory.

DHCP

kernel/net/dhcp.c. UDP, client port 68, server port 67. BOOTP layout:

Offset Size Field
0 1 Op — 1 request, 2 reply
1 1 Hardware type — 1 Ethernet
2 1 Hardware address length — 6
3 1 Hops
4 4 Transaction id (xid)
8 2 Seconds
10 2 Flags
12 4 Client IP
16 4 Your IP — the assigned address
20 4 Server IP
24 4 Gateway IP
28 16 Client hardware address
44 64 Server name
108 128 Boot file name
236 4 Magic cookie — 0x63825363
240 n Options, terminated by 0xFF

Important

The magic cookie sits at offset 236, not 240, and is stored big-endian. Getting either wrong makes every server ignore the packet. Both mistakes were made here.

Option Code
Subnet mask 1
Router 3
DNS server 6
Requested IP 50
Lease time 51
Message type 53
Server identifier 54
End 255
Message type Value
DISCOVER 1
OFFER 2
REQUEST 3
ACK 5

Exchange: DISCOVER → OFFER → REQUEST → ACK. Auto-DHCP runs at boot when a NIC is present.

DNS

kernel/net/dns.c. UDP port 53.

Header

Offset Size Field
0 2 Transaction ID
2 2 Flags
4 2 Question count
6 2 Answer count
8 2 Authority count
10 2 Additional count

Question

Name in label form — a length byte followed by that many characters, repeated, terminated by a zero byte. example.com encodes as 07 'example' 03 'com' 00. Then:

Size Field
2 Type — 1 for A
2 Class — 1 for IN

Warning

A question count stored little-endian reads as 256 questions, and the server drops the query. This happened.

NyxOS sends a single A-record query and parses the first matching answer.

TLS 1.2

kernel/crypto/tls/tls.c. Full protocol description in Cryptography-and-TLS.

Record layer

Offset Size Field
0 1 Content type
1 2 Version
3 2 Length
5 n Fragment
Content type Value
ChangeCipherSpec 20
Alert 21
Handshake 22
Application data 23
Version Value
TLS 1.0 0x0301 — used as the record version in the ClientHello for compatibility
TLS 1.2 0x0303 — the client_version inside the ClientHello

Encrypted fragment

Once the cipher is active, a record's fragment is:

explicit_nonce(8) || GCM_ciphertext || GCM_tag(16)

Handshake message

Offset Size Field
0 1 Handshake type
1 3 Length, 24-bit big-endian
4 n Body
Type Value
ClientHello 1
ServerHello 2
Certificate 11
ServerKeyExchange 12
ServerHelloDone 14
ClientKeyExchange 16
Finished 20

ClientHello body

Size Field NyxOS value
2 client_version 0x0303
32 Random From the CSPRNG
1 Session ID length 0
2 Cipher-suite list length 16 (8 suites)
2 × 8 Cipher suites See below
1 Compression method count 1
1 Compression method 0, null
2 Extensions length
n Extensions

Cipher suites offered, in order

Value Suite
0xC02F ECDHE_RSA_WITH_AES_128_GCM_SHA256
0xC02B ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
0xC030 ECDHE_RSA_WITH_AES_256_GCM_SHA384
0xC02C ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
0x009C RSA_WITH_AES_128_GCM_SHA256
0x009D RSA_WITH_AES_256_GCM_SHA384
0x002F RSA_WITH_AES_128_CBC_SHA
0x0035 RSA_WITH_AES_256_CBC_SHA

Extensions

Extension Type Contents
server_name (SNI) 0x0000 Host name, list length len+3, name type 0
supported_groups 0x000A x25519, secp256r1, secp384r1

Signature schemes advertised and verified

As of v5.9.92 NyxOS advertises — and can verify a ServerKeyExchange signature under — every common scheme, so strict mode enforces key-exchange authenticity whichever one a server picks. The ClientHello's signature_algorithms extension sends these five, in this order:

Value Scheme Live-verified?
0x0401 RSA PKCS#1 v1.5 with SHA-256 Yes
0x0403 ECDSA-P256 with SHA-256 Yes
0x0503 ECDSA-P384 with SHA-384 Yes
0x0804 RSA-PSS with SHA-256, MGF1-SHA256, 32-byte salt Yes
0x0601 RSA PKCS#1 v1.5 with SHA-512 KAT-only (no reachable server picks it)

X.509 and DER

kernel/crypto/der.c, kernel/crypto/x509.c.

DER TLV

Every element is Tag-Length-Value. A length byte below 0x80 is the length itself; 0x80 | n means the next n bytes are the length, big-endian.

Tag Type
0x02 INTEGER
0x03 BIT STRING
0x04 OCTET STRING
0x05 NULL
0x06 OBJECT IDENTIFIER
0x30 SEQUENCE (constructed)
0x31 SET (constructed)
0xA0 Context-specific [0] (constructed)

The reader is a cursor over a byte range:

typedef struct { const uint8_t* p; const uint8_t* end; } der_t;
int der_read(der_t* c, uint8_t* tag, const uint8_t** val, uint32_t* vlen);
int der_enter(der_t* c, uint8_t expect, der_t* inner);

Chain verification results

Constant Value Meaning
X509_OK 0 Every link verified and the top is a pinned trusted root
X509_INCOMPLETE 1 Links checked but not anchored — unknown root or unsupported algorithm. Not a detected forgery
X509_FORGED −1 A supported-algorithm link failed cryptographic verification

x509_check_host returns 0 on a SAN dNSName match, 1 if none matched, −1 if there is no usable subjectAltName. x509_check_validity returns 0 if in date, 1 if not yet valid, 2 if expired, −1 if the dates cannot be parsed.

ELF64

kernel/proc/elf.h. Programs link at 0x10000.

File header, 64 bytes

Offset Size Field Required value
0 16 e_ident 7F 45 4C 46 = \x7fELF, then class 2 (64-bit), data 1 (LSB)
16 2 e_type 2 = ET_EXEC
18 2 e_machine 62 = EM_X86_64
20 4 e_version 1
24 8 e_entry Entry point
32 8 e_phoff Program-header table offset
40 8 e_shoff Section-header table offset
48 4 e_flags
52 2 e_ehsize 64
54 2 e_phentsize 56
56 2 e_phnum
58 2 e_shentsize 64
60 2 e_shnum
62 2 e_shstrndx

EI_NIDENT is 16.

Program header, 56 bytes

Offset Size Field
0 4 p_type
4 4 p_flags
8 8 p_offset
16 8 p_vaddr
24 8 p_paddr
32 8 p_filesz
40 8 p_memsz
48 8 p_align
p_type Value NyxOS handling
PT_NULL 0 Skipped
PT_LOAD 1 Mapped at p_vaddr
PT_DYNAMIC 2 Skipped
PT_INTERP 3 Skipped
PT_NOTE 4 Skipped
p_flags Value Page mapping
PF_X 1 Executable — NX cleared
PF_W 2 Writable
PF_R 4 Readable

W^X is applied per segment from these flags. p_filesz bytes are copied and the remainder up to p_memsz is zeroed — that difference is .bss.

Caution

The loader validates the header and every program header before mapping anything. The shared-library path (libseg_load) once validated nothing at all, which was the last confirmed CRITICAL finding of the v5.9.0 audit.

Section header, 64 bytes

Offset Size Field
0 4 sh_name
4 4 sh_type
8 8 sh_flags
16 8 sh_addr
24 8 sh_offset
32 8 sh_size
40 4 sh_link
44 4 sh_info
48 8 sh_addralign
56 8 sh_entsize

Symbol, 24 bytes

Offset Size Field
0 4 st_name — index into the linked string table
4 1 st_info
5 1 st_other
6 2 st_shndx
8 8 st_value
16 8 st_size

Used by dlsym to resolve a name in a loaded shared object.

SysV entry stack

Built by execve, read by crt0.asm:

[rsp]        argc
[rsp+8]      argv[0]
[rsp+16]     argv[1]
   …
             argv[argc-1]
             NULL
             envp[0]
   …
             NULL
Limit Value
Maximum arguments 8
Maximum length per argument 63 characters

CPIO newc — the initramfs

kernel/fs/initramfs.c, generated by tools/mkinitramfs.py into initramfs_data.h.

Every field is 8 ASCII hexadecimal digits, not binary.

Offset Size Field
0 6 Magic — 070701
6 8 inode
14 8 mode
22 8 uid
30 8 gid
38 8 nlink
46 8 mtime
54 8 filesize
62 8 devmajor
70 8 devminor
78 8 rdevmajor
86 8 rdevminor
94 8 namesize
102 8 check
110 namesize File name, NUL-terminated

The name is padded to a 4-byte boundary, then the file data follows, also padded to 4 bytes. The archive ends with an entry named TRAILER!!!.

Observed at boot: [INITRAMFS] Loaded 64 files.

EXT2

kernel/fs/ext2.h. All fields little-endian.

Constant Value
EXT2_SUPER_MAGIC 0xEF53
EXT2_ROOT_INO 2

Superblock

At byte offset 1024 from the start of the volume.

Offset Size Field
0 4 total_inodes
4 4 total_blocks
8 4 blocks_su — reserved for superuser
12 4 free_blocks
16 4 free_inodes
20 4 first_data_block
24 4 log_block_size — block size is 1024 << this
28 4 log_frag_size
32 4 blocks_per_group
36 4 frags_per_group
40 4 inodes_per_group
44 4 mtime
48 4 wtime
52 2 mnt_count
54 2 max_mnt_count
56 2 magic — must be 0xEF53
58 2 state
60 2 errors
62 2 minor_rev
64 4 last_check
68 4 check_interval
72 4 creator_os

Block group descriptor, 32 bytes

Offset Size Field
0 4 block_bitmap
4 4 inode_bitmap
8 4 inode_table
12 2 free_blocks_count
14 2 free_inodes_count
16 2 used_dirs_count
18 2 pad
20 12 reserved

Inode, 128 bytes

Offset Size Field
0 2 mode
2 2 uid
4 4 size
8 4 atime
12 4 ctime
16 4 mtime
20 4 dtime
24 2 gid
26 2 links_count
28 4 blocks_512 — in 512-byte units
32 4 flags
36 4 osd1
40 60 block[15] — 12 direct, 1 indirect, 1 double, 1 triple
100 4 generation
104 4 file_acl
108 4 dir_acl
112 4 faddr
116 12 osd2[3]
Mode constant Value
EXT2_S_IFREG 0x8000
EXT2_S_IFDIR 0x4000

Directory entry

Offset Size Field
0 4 inode — 0 means a free slot
4 2 rec_len — bytes to the next entry
6 1 name_len
7 1 file_type
8 name_len Name, not NUL-terminated
File type Value
EXT2_FT_UNKNOWN 0
EXT2_FT_REG_FILE 1
EXT2_FT_DIR 2

Entries are padded so rec_len is a multiple of 4; the last entry in a block has a rec_len reaching the end of the block.

Observed at boot: [EXT2] Found: 16384 blocks, 4096 inodes, block size 1024.

Important

The write path is verified against e2fsck, not against itself. A driver that reads back what it wrote can be self-consistently wrong. The sector cache is write-through, so it cannot lose data on a crash.

Image formats

Full behaviour in Image-Decoders. Every decoder produces the same output:

typedef struct {
    uint32_t width, height;
    uint8_t* pixels;        /* RGBA, 4 bytes per pixel, width*height*4, kmalloc'd */
} image_t;

DEFLATE and zlib

kernel/image/inflate.c.

zlib wrapper — RFC 1950

Offset Size Field
0 1 CMF — compression method and window size; method 8 = DEFLATE
1 1 FLG — check bits, preset dictionary flag, compression level
2 n DEFLATE stream
end 4 Adler-32 of the uncompressed data, big-endian — verified

DEFLATE blocks — RFC 1951

Each block starts with a 3-bit header: 1 bit BFINAL, 2 bits BTYPE.

BTYPE Value Meaning
00 0 Stored — byte-aligned LEN, ~LEN, then raw bytes
01 1 Fixed Huffman codes
10 2 Dynamic Huffman codes
11 3 Reserved, error

All three are implemented.

PNG

kernel/image/png.c. Signature: 89 50 4E 47 0D 0A 1A 0A.

Chunk

Offset Size Field
0 4 Length of the data field, big-endian
4 4 Type, four ASCII characters
8 length Data
— 4 CRC-32 over type and data
Chunk Purpose
IHDR Image header
PLTE Palette
IDAT Compressed image data — a zlib stream, possibly split across chunks
IEND End of file

IHDR, 13 bytes

Offset Size Field
0 4 Width
4 4 Height
8 1 Bit depth — 8 only
9 1 Colour type
10 1 Compression method — 0
11 1 Filter method — 0
12 1 Interlace method — 0 only
Colour type Value Channels Supported
Greyscale 0 1 Yes
Truecolour 2 3 Yes
Palette 3 1 Yes
Greyscale + alpha 4 2 Yes
Truecolour + alpha 6 4 Yes

Scanline filters

Each decompressed scanline is prefixed by a filter-type byte. All five are implemented.

Type Name Reconstruction
0 None x
1 Sub x + a
2 Up x + b
3 Average x + (a + b) / 2
4 Paeth x + Paeth(a, b, c)

Where a is the byte to the left, b the byte above, c the byte above-left.

Not supported: 16-bit depth, Adam7 interlacing.

BMP

kernel/image/bmp.c.

File header, 14 bytes

Offset Size Field
0 2 Signature — BM (0x42 0x4D)
2 4 File size
6 4 Reserved
10 4 Offset to pixel data

BITMAPINFOHEADER, 40 bytes

Offset Size Field
14 4 Header size — 40
18 4 Width
22 4 Height — negative means top-down
26 2 Planes — 1
28 2 Bits per pixel
30 4 Compression — BI_RGB (0) only
34 4 Image size
38 4 Horizontal resolution
42 4 Vertical resolution
46 4 Palette colours used
50 4 Important colours
Depth Layout Supported
8 Palette index Yes
24 BGR Yes
32 BGRX Yes

Rows are padded to a 4-byte boundary. Bottom-up (positive height) and top-down (negative height) are both handled.

GIF

kernel/image/gif.c.

Offset Size Field
0 6 Signature — GIF87a or GIF89a
6 2 Logical screen width
8 2 Logical screen height
10 1 Packed — global colour table flag, colour resolution, sort flag, table size
11 1 Background colour index
12 1 Pixel aspect ratio
13 n Global colour table, 3 × 2^(size+1) bytes
Block introducer Value
Extension 0x21
Image descriptor 0x2C
Trailer 0x3B

Image descriptor, 10 bytes

Offset Size Field
0 1 0x2C
1 2 Left position
3 2 Top position
5 2 Width
7 2 Height
9 1 Packed — local colour table flag, interlace flag, sort, table size

Graphic Control Extension

Field Meaning
Disposal method 0 none, 1 leave, 2 restore background, 3 restore previous
Transparent colour flag Whether the index below is transparent
Delay time Centiseconds
Transparent colour index

Animation

typedef struct { uint8_t* pixels; uint16_t delay_cs; } gif_frame_t;
typedef struct { uint32_t width, height; int nframes; gif_frame_t* frames; int loop_count; } gif_anim_t;

int  gif_decode_anim(const uint8_t* src, uint32_t srclen, gif_anim_t* out);
void gif_anim_free(gif_anim_t* a);

gif_decode_anim walks every image descriptor and composites each onto a logical-screen canvas, honouring the per-frame sub-rectangle, transparent index and disposal method, so every frame is a ready-to-blit full-size image. A static GIF yields nframes == 1. Delays are in centiseconds.

gif_decode decodes only the first frame.

LZW

Image data is LZW-compressed with a variable code size starting at the minimum code size plus one, with explicit clear and end-of-information codes and a dictionary that grows to 12 bits. Interlaced images are stored in four passes.

JPEG

kernel/image/jpeg.c. Big-endian. Baseline only: sequential DCT, Huffman-coded, 8-bit.

Markers

Every marker is 0xFF followed by a code byte.

Marker Code Meaning
SOI FFD8 Start of image — also the magic bytes Selene dispatches on
APPn FFE0–FFEF Application segments, skipped
DQT FFDB Define quantisation table
SOF0 FFC0 Start of frame, baseline
SOF2 FFC2 Progressive — rejected
DHT FFC4 Define Huffman table
DRI FFDD Define restart interval
SOS FFDA Start of scan
RSTn FFD0–FFD7 Restart markers
EOI FFD9 End of image

Except for SOI, EOI and RSTn, each marker is followed by a 2-byte segment length that includes itself.

SOF0 body

Size Field
1 Sample precision — 8
2 Height
2 Width
1 Component count
3 × n Per component: id, sampling factors (H:V nibbles), quantisation table id
Layout Components Supported
Greyscale 1 Yes
YCbCr 4:4:4 3, all 1×1 Yes
YCbCr 4:2:2 (h2v1) 3 Yes
YCbCr 4:2:0 (h2v2) 3 Yes
CMYK / YCCK 4 Rejected

Rejected cleanly rather than mis-decoded: progressive, arithmetic coding, 12-bit precision, 4-component.

Chroma upsampling

Subsampled chroma uses libjpeg's triangle-filter ("fancy") upsampling for the two common photo layouts, 4:2:2 (h2v1) and 4:2:0 (h2v2). The original decoder used nearest-neighbour, replicating each chroma sample across its block, which is visibly blocky at colour edges.

Note

jpegtest compares against a reference decode within a tolerance, not for exact equality. Exact agreement across IDCT implementations is impossible — libjpeg's integer IDCT differs from any other by a rounding step.

Byte-order summary

Format Endianness
All network protocols Big-endian (network order)
ELF64 Little-endian (e_ident[5] = 1)
CPIO newc ASCII hexadecimal
EXT2 Little-endian
PNG Big-endian
GIF Little-endian
BMP Little-endian
JPEG Big-endian
TLS Big-endian

See also

External resources

NyxOS v6.4.363 · GPL v2 · GitHub · uselessalter on Discord · nyxos@inbox.lv

NyxOS Wiki

Getting started

Kernel

Storage & network

Graphics & apps

Userspace

HOWTO

Reference

Project

Clone this wiki locally