DiskMesh

A simplified, production-shaped distributed file system written from scratch in Java, inspired by HDFS, the Google File System (GFS) and Ceph. It splits files into checksummed chunks, replicates them across storage nodes with a chained write pipeline, detects node failures via heartbeats, and automatically restores the replication factor when nodes die.

No frameworks, no external storage libraries — only the Java Standard Library.

              upload / download / delete / ls / nodes / status / stats
   ┌────────┐        (metadata plane)          ┌──────────────────┐
   │ Client │ ───────────────────────────────► │   Master Node    │
   └────────┘                                   │  namespace       │
        │                                       │  chunk table     │
        │ (data plane: chunks)                  │  node table      │
        │                                       │  placement       │
        ▼                                       │  failure detector│
   ┌──────────┐  pipeline  ┌──────────┐  ───────┤  re-replication  │
   │ Storage  │ ─────────► │ Storage  │ heartbeat└──────────────────┘
   │ Node A   │            │ Node B   │ ─────────► (block reports)
   └──────────┘            └──────────┘

---

Requirements

• A Java 22+ runtime (java). DiskMesh runs directly from source using the JEP 458 multi-file source-code launcher — no javac, no build step, no dependencies.
• Verified on OpenJDK 25.

Why no build tool? The brief asks for standard-library-only code. The source launcher lets the whole project run as .java files, which keeps the repository free of a compiler toolchain or dependency manager while still being a normal, idiomatic Java codebase you can point javac at if you prefer.

---

Quickstart

Run the automated test suite (spins up a full in-process cluster)

bash scripts/test.sh

Run the end-to-end multi-process demo (master + 4 nodes + CLI)

bash scripts/demo.sh

Run a cluster by hand

# terminal 1 — master
bash scripts/master.sh --port 9090 --rf 3 --chunk-size 8M

# terminals 2..5 — storage nodes (data dir must differ per node)
bash scripts/node.sh --master-port 9090 --bind-host 127.0.0.1 --data /tmp/dfs-n1
bash scripts/node.sh --master-port 9090 --bind-host 127.0.0.1 --data /tmp/dfs-n2
bash scripts/node.sh --master-port 9090 --bind-host 127.0.0.1 --data /tmp/dfs-n3

# terminal 6 — client
alias dfs='bash scripts/dfs.sh --master-port 9090'
dfs upload ./movie.mp4 /movie.mp4
dfs ls
dfs nodes
dfs status
dfs download /movie.mp4 ./movie.copy.mp4
dfs stats
dfs delete /movie.mp4

---

CLI

Command	Description
dfs upload <local> [dfs-path]	Split, checksum, replicate and register a file
dfs download <dfs-path> [local]	Fetch chunks, reassemble, verify SHA-256
dfs delete <dfs-path>	Remove a file; chunks are garbage-collected
dfs ls [prefix]	List files
dfs nodes	List storage nodes and their utilisation
dfs status	Cluster status summary
dfs stats	Cluster + throughput metrics

Global options: --master-host, --master-port, --chunk-size (accepts K/M/G suffixes).

---

Configuration flags

Flag	Applies to	Default	Meaning
--port	master, node	9090 / 0	Listen port (0 = ephemeral for nodes)
--master-host/-port	node, client	127.0.0.1	Where the master lives
--bind-host	node, master	0.0.0.0	Bind / advertise address
--data	master, node	dfs-data	Data directory
--rf	master	3	Replication factor
--chunk-size	master, client	4M	Chunk size (1M, 8M, 64M, ...)
--heartbeat-interval	node	2000 ms	Heartbeat cadence
--heartbeat-timeout	master	6000 ms	Silence before a node is declared dead
--capacity	node	disk free	Advertised capacity (for load balancing)
--rpc-retries	all	4	Max RPC retries (exponential backoff)

---

Project layout

src/dfs/
  common/       Config, Log, Ids, Checksums, DfsException, NodeRef
  protocol/     MessageType, Message, Codec, Frame, ControlMessages, DataMessages
  network/      TcpServer (virtual threads), RpcClient (retries + backoff)
  metadata/     FileMetadata, ChunkMetadata, NodeInfo, MetadataStore, snapshot store
  storage/      ChunkStore (on-disk chunks + checksum sidecars)
  replication/  PlacementPolicy (load-aware replica placement)
  scheduler/    Scheduler (periodic failure detection / GC / snapshot)
  monitoring/   Metrics (counters + throughput)
  master/       MasterNode (namespace, placement, failure detection, admin API)
  storagenode/  StorageNode (store/serve/replicate/verify chunks)
  client/       DfsClient (library) + Cli (command-line front end)
  tests/        Assert, TestCluster, IntegrationTests
docs/           ARCHITECTURE.md (diagrams, protocol spec, design decisions)
scripts/        master.sh, node.sh, dfs.sh, test.sh, demo.sh

See docs/ARCHITECTURE.md for architecture, sequence and class diagrams, the wire protocol specification, the failure-recovery model, the thread model, and design trade-offs.

---

What is implemented

• Chunking with configurable size; supports files larger than RAM (streamed chunk-by-chunk).
• SHA-256 integrity verified on upload, at rest (sidecar), during replication, and on download.
• Chained write pipeline (client → A → B → C); success is acknowledged only after full replication.
• Configurable replication factor with load-aware placement (free space + utilisation).
• Heartbeat protocol with block reports; failure detection and automatic re-replication.
• Garbage collection of orphaned chunks; stale-upload expiry.
• Metadata snapshots persisted to disk and reloaded on restart.
• Admin API: cluster status, active/dead nodes, storage usage, under-replicated / missing chunks.
• Metrics: read/write bytes and throughput, replication counts, node availability, chunk counts.
• Concurrency built on virtual threads, ReadWriteLock, ConcurrentHashMap and atomics.
• Fault tolerance: RPC retries with exponential backoff + jitter, read-path replica failover.

Not implemented (documented as future work)

Raft leader election, compression, encryption/TLS, versioning, deduplication, quotas, a web dashboard and a REST API. The architecture doc explains where each would slot in.