# MDFS – How to Replicate / Port (Reference Guide) This page is a practical guide for implementing an **MDFS (ModularFS)** reader/writer in another environment: - your own custom OS - a Linux userspace tool (FUSE-style) - a Windows driver (or parsing library) - a BSD driver The goal is to document **what is on disk** and the minimum logic needed to interoperate. > MDFS is inspired by **EXT2** (inodes + bitmaps + inode table layout) and **exFAT** (directory entry sets + checksums). --- ## 0) Quick facts (v2) From `include/moduos/fs/MDFS/mdfs.h`: - Magic: `MDFS_MAGIC = 0x5346444D` (ASCII "MDFS" little-endian) - Version: `MDFS_VERSION = 2` - Block size: `MDFS_BLOCK_SIZE = 4096` - Inode size: `MDFS_INODE_SIZE = 256` - Max direct blocks: `MDFS_MAX_DIRECT = 12` - Directory record size: `MDFS_DIR_REC_SIZE = 32` - Max name bytes: `MDFS_MAX_NAME = 255` (UTF-8 bytes) --- ## 1) On-disk layout ## 1.1) Canonical on-disk structs (C reference) These structs are the *authoritative* reference for MDFS v2. They are copied from `include/moduos/fs/MDFS/mdfs.h`. ```c #define MDFS_MAGIC 0x5346444Du /* 'MDFS' little-endian */ #define MDFS_VERSION 2 #define MDFS_BLOCK_SIZE 4096u #define MDFS_INODE_SIZE 256u #define MDFS_MAX_DIRECT 12u #define MDFS_MAX_NAME 255u #define MDFS_DIR_REC_SIZE 32u #define MDFS_DIRREC_PRIMARY 1u #define MDFS_DIRREC_NAME 2u #define MDFS_DIRFLAG_VALID 0x01u #define MDFS_DIRFLAG_DELETED 0x02u typedef struct __attribute__((packed)) { uint16_t mode; // 0x4000 dir, 0x8000 file uint16_t _pad0; uint32_t uid; uint32_t gid; uint64_t size_bytes; uint32_t link_count; uint32_t flags; uint64_t direct[MDFS_MAX_DIRECT]; uint64_t indirect1; uint8_t _pad[MDFS_INODE_SIZE - 2 - 2 - 4 - 4 - 8 - 4 - 4 - (8*MDFS_MAX_DIRECT) - 8]; } mdfs_inode_t; typedef struct __attribute__((packed)) { uint8_t rec_type; // MDFS_DIRREC_PRIMARY uint8_t flags; // MDFS_DIRFLAG_* uint8_t entry_type; // 1=file,2=dir uint8_t record_count; // total records in entry set (including this primary) uint32_t inode; uint16_t name_len; // UTF-8 bytes uint16_t _rsv0; uint32_t checksum; // CRC32 over entry set with this field zero uint8_t _pad[32 - 1 - 1 - 1 - 1 - 4 - 2 - 2 - 4]; } mdfs_dir_primary_t; typedef struct __attribute__((packed)) { uint8_t rec_type; // MDFS_DIRREC_NAME uint8_t name_bytes[31]; } mdfs_dir_name_t; typedef struct __attribute__((packed)) { uint32_t magic; uint32_t version; uint32_t block_size; uint32_t _reserved0; uint64_t total_blocks; uint64_t free_blocks; uint64_t total_inodes; uint64_t free_inodes; uint64_t block_bitmap_start; uint64_t block_bitmap_blocks; uint64_t inode_bitmap_start; uint64_t inode_bitmap_blocks; uint64_t inode_table_start; uint64_t inode_table_blocks; uint64_t root_inode; uint8_t uuid[16]; uint32_t features; uint32_t checksum; /* CRC32 over superblock with this field zero */ uint8_t pad[MDFS_BLOCK_SIZE - (4*4) - (6*8) - (1*8) - 16 - 4 - 4]; } mdfs_superblock_t; ``` MDFS is block-based. “Block N” means block index in units of `4096` bytes. The superblock (block 1) tells you where metadata lives: - `block_bitmap_start` / `block_bitmap_blocks` - `inode_bitmap_start` / `inode_bitmap_blocks` - `inode_table_start` / `inode_table_blocks` - `root_inode` ### Required reads to mount 1. Read block 1 into `mdfs_superblock_t`. 2. Validate: - `magic == MDFS_MAGIC` - `version == 2` - `block_size == 4096` 3. Optionally validate `superblock.checksum` (CRC32) if non-zero. --- ## 2) CRC32 algorithm Reference implementation (copied from `src/fs/MDFS/mdfs_disk.c`): ```c #include #include uint32_t mdfs_crc32_buf(const void *data, size_t len) { const uint8_t *p = (const uint8_t*)data; uint32_t crc = 0xFFFFFFFFu; for (size_t i = 0; i < len; i++) { uint32_t x = (crc ^ p[i]) & 0xFFu; for (int b = 0; b < 8; b++) { x = (x >> 1) ^ (0xEDB88320u & (-(int)(x & 1u))); } crc = (crc >> 8) ^ x; } return ~crc; } ``` MDFS uses CRC32 with the common reversed polynomial `0xEDB88320`. Reference implementation exists in ModuOS: - `src/fs/MDFS/mdfs_disk.c: mdfs_crc32_buf()` Usage: - Superblock checksum: CRC32 over the superblock with `checksum` field set to 0. - Directory entry set checksum: CRC32 over the full entry set with the primary’s `checksum` field set to 0. --- ## 3) Inodes ### Inode location C-like pseudocode: ```c // byte offset of inode inside inode table region uint64_t byte_off = (uint64_t)ino * (uint64_t)MDFS_INODE_SIZE; uint64_t block = sb->inode_table_start + (byte_off / MDFS_BLOCK_SIZE); uint64_t off = byte_off % MDFS_BLOCK_SIZE; read_block(block, tmp4096); memcpy(out_inode, tmp4096 + off, sizeof(mdfs_inode_t)); ``` Inode `ino` is stored in the inode table region: - byte offset = `ino * MDFS_INODE_SIZE` - block = `inode_table_start + (byte_off / 4096)` - offset inside block = `byte_off % 4096` See ModuOS reference: - `src/fs/MDFS/mdfs_disk.c: mdfs_disk_read_inode()` ### Inode format (`mdfs_inode_t`) Key fields: - `mode`: `0x4000` directory, `0x8000` regular file - `size_bytes` - `direct[12]`: block pointers - `indirect1`: reserved for future expansion Current implementation supports direct blocks plus `indirect1/2/3` (single/double/triple-indirect). --- ## 4) Allocation bitmaps MDFS uses EXT2-like bitmaps. ### Inode bitmap - bitmap is at block `inode_bitmap_start` - inode `i` is free if bit `i` is 0 ### Block bitmap - bitmap is at block `block_bitmap_start` - block `b` is free if bit `b` is 0 > Note: ModuOS uses multi-block bitmap scanning via `block_bitmap_blocks` / `inode_bitmap_blocks` (see `mdfs_api.c`), allowing large volumes. --- ## 5) Directories (entry sets, exFAT-like) Directories are files whose data blocks contain an array of 32-byte records. ### Record types - Primary record: `rec_type = MDFS_DIRREC_PRIMARY (1)` - Name record: `rec_type = MDFS_DIRREC_NAME (2)` ### Primary record (`mdfs_dir_primary_t`) Fields of interest: - `flags`: includes `VALID` and `DELETED` - `entry_type`: `1=file`, `2=dir` - `record_count`: total record count (primary + name records) - `inode`: inode number - `name_len`: UTF-8 byte length - `checksum`: CRC32 over the entire entry set ### Name records (`mdfs_dir_name_t`) - each name record stores 31 bytes of UTF-8 payload - long names are concatenated across multiple name records ### Parsing a directory Algorithm (compatible with ModuOS `mdfs_dir.c`): 1. Read the directory inode. 2. For each direct block pointer in `direct[]`: - read block - walk records at offsets 0..4096 in steps of 32 - if `rec_type == 0`, stop (end-of-directory marker) - if record is not a valid primary, skip using `record_count` when possible - when you find a valid primary: - verify `checksum` (optional but recommended) - read `record_count-1` name records and reconstruct UTF-8 name ### Lookup Compare the reconstructed name with the query name (case-sensitive in current ModuOS logic). ### Deletion ModuOS marks entries as deleted by setting the `DELETED` flag and rewriting the checksum. --- ## 6) Path resolution MDFS path resolution is hierarchical: - start at `root_inode` - split `/a/b/c` into components - for each component: - look up entry in current directory - ensure it’s a directory if there are remaining components See ModuOS reference: - `src/fs/MDFS/mdfs_api.c: mdfs_lookup_path()` --- ## 7) Reading files 1. Resolve path to inode. 2. Read inode to get `size_bytes` and direct blocks. 3. Read blocks in order until you have `size_bytes`. Current ModuOS implementation reads using `direct[]` plus `indirect1/2/3` via the `mdfs_get_block_ptr()` helper (single/double/triple-indirect, EXT2-style). --- ## 8) Writing files (ModuOS behavior) ModuOS supports both whole-file overwrite and offset writes: - `mdfs_write_file_by_path()` (offset 0) - `mdfs_write_file_at_by_path()` (offset-aware, used by the FD layer) ModuOS `mdfs_write_file_at_by_path()` does: - resolves parent directory + basename - if file exists: - overwrites file contents in-place (allocating blocks as needed) - if file does not exist: - allocates inode - adds directory entry set - allocates blocks and writes data Important notes: - `unlink`/`rmdir` do free previously allocated blocks (direct + indirect) in the current implementation. - Truncation/block-reclamation on overwrite is **partially implemented**: - `mdfs_create_file_trunc(..., truncate=1, ...)` has a TODO to free blocks (so true truncation is not complete yet). - Normal writes support `direct[]` + `indirect1/2/3` and include a fast sequential preallocation path (batched bitmap alloc + `mdfs_set_block_ptr_range()`). If you’re implementing a driver elsewhere, decide whether you want to match this behavior or implement full block reclamation. --- ## 9) Minimal “portable” implementation plan ### A) Read-only library (fastest) - Implement superblock read/verify - Implement inode read - Implement directory parse (entry sets) - Implement path lookup - Implement file read This is enough for: - Linux userspace tool - FUSE read-only filesystem - Windows/BSD read-only driver ### B) Read-write (more work) Add: - bitmap allocation/free - inode write - directory entry set add/remove - data block write And decide on crash-consistency strategy (MDFS is not journaled). --- ## 10) Tips for specific platforms ### Linux (FUSE) - Create a FUSE filesystem that maps MDFS inodes to FUSE inodes. - Cache directory listings and inode reads for performance. - Implement `readdir`, `lookup`, `read` first. ### Windows - Consider starting with a userspace parser (WinFsp) before writing a kernel driver. - If writing a kernel driver, be careful with paging/IRQL rules. ### BSD - Implement as a VFS module; start with lookup/read-only. --- ## 11) Reference code in ModuOS - Structures: `include/moduos/fs/MDFS/mdfs.h` - Disk IO + CRC: `src/fs/MDFS/mdfs_disk.c` - Directory logic: `src/fs/MDFS/mdfs_dir.c` - Path + VFS-facing API: `src/fs/MDFS/mdfs_api.c` --- ## 12) Compatibility and versioning - This doc describes **MDFS v2**. - If you change on-disk layout/record formats, bump `version` and update this doc. A small test image generator is included so external implementations have a known-good corpus. ### Known-good corpus generator Script: - `tools/mdfs_make_test_image.py` It creates an MBR-partitioned disk image with a single MDFS v2 partition and a few deterministic sample files. Run: ```sh python tools/mdfs_make_test_image.py --size-mib 64 --out dist/mdfs_test.img ``` Sample contents: - `/test.txt` => `MDFS OK`\n - `/hello/world.txt` => `Hello from ModularFS (MDFS)!`\n - `/big.bin` => deterministic pattern (used for verifying reads) This image is intended for driver/FUSE/parser bring-up.