-
Notifications
You must be signed in to change notification settings - Fork 0
05_fault_tolerance___replication_monitoring.md
Welcome back to the world of cdfs-ton! In our last chapter, Network Communication Protocol, we learned the "language" that all parts of cdfs-ton use to talk to each other. Now that everyone can communicate, let's tackle a very important question: What happens if something breaks?
Imagine you're storing your precious family photos in cdfs-ton. You've uploaded them, and the Metadata Server knows where all the chunks are, and the Storage Nodes are diligently holding onto them. But what if one of those Storage Nodes (one of the "storage lockers") suddenly crashes, or its hard drive fails? Would your photos be lost forever?
Absolutely not! This is where Fault Tolerance & Replication Monitoring comes to the rescue. This critical system is like an ever-vigilant guardian that ensures your data remains safe and available, even if a Storage Node disappears. It continuously watches over the "health" of all Storage Nodes and quickly acts if a problem arises.
In any large system with many computers (like cdfs-ton!), it's not a matter of if a computer will fail, but when. Hard drives can break, network connections can drop, power can go out. If your data is stored on just one machine, and that machine fails, your data is gone. This is a single point of failure.
cdfs-ton solves this problem with two main ideas:
-
Replication (Redundancy): Instead of storing just one copy of each data chunk,
cdfs-tonmakes multiple copies (replicas) of every chunk and stores them on different Storage Nodes. If one node fails, there are still other copies available. - Monitoring (Vigilance): The Metadata Server constantly watches all Storage Nodes. If a node goes silent, the Metadata Server quickly figures out what data was on it and orchestrates a plan to replace any lost copies.
Let's break down how this guardian system works!
When you upload a file, cdfs-ton splits it into chunks. For each chunk, it doesn't just store one copy; it stores several. The number of copies is determined by the Replication Factor.
In cdfs-ton, the REPLICATION_FACTOR is set to 3. This means every chunk of your file will have three identical copies stored on three different Storage Nodes.
// File: common/dfs.h
#define REPLICATION_FACTOR 3Why 3? If you only had two copies and one node failed, you'd be left with just one copy. If that node then failed before you could make a new copy, your data would be lost. With three copies, if one node fails, you still have two safe copies, giving the system time to create a new third copy.
Every few seconds, each Storage Node sends a small "heartbeat" message to the Metadata Server. This is like saying, "Hey, I'm here, I'm working, and everything's fine!"
The Metadata Server keeps a record of when it last "heard" from each Storage Node.
If the Metadata Server doesn't receive a heartbeat from a Storage Node for a certain amount of time (e.g., 15 seconds), it marks that node as "dead." It assumes the node has failed and can no longer be trusted to hold data.
Once a node is declared dead, the Metadata Server springs into action:
- It checks its records to see which chunks were stored on the dead node.
- For each of those chunks, it looks to see if enough healthy copies still exist on other Storage Nodes.
- If a chunk is now "under-replicated" (meaning it has fewer than
REPLICATION_FACTORcopies), the Metadata Server finds a healthy Storage Node that still has a copy of that chunk (the "source" node). - It then finds another healthy, available Storage Node (the "destination" node).
- Finally, it tells the "source" node to send a copy of the chunk to the "destination" node. This restores the required number of copies.
Sometimes, a Storage Node might end up with chunks that no longer belong to any active file in the cdfs-ton system. This can happen if a file was deleted, but a particular chunk wasn't cleaned up correctly, or if a replication process failed halfway. The monitoring system also periodically checks for these "orphaned" chunks and instructs the Storage Nodes to delete them, keeping the system tidy.
Let's see how cdfs-ton handles a Storage Node failure in practice.
Imagine you have three Storage Nodes: Node A, Node B, and Node C. A chunk of your photo, let's call it Chunk_123, has copies on all three.
- Node B Crashes: Suddenly, Node B becomes unresponsive.
- Heartbeats Stop: Node B stops sending heartbeats to the Metadata Server.
- Dead Node Detected: After 15 seconds of silence, the Metadata Server declares Node B "dead."
-
Under-Replication Identified: The Metadata Server checks its records. It knows
Chunk_123was on Node B. Now,Chunk_123only exists on Node A and Node C (2 copies). This is less than theREPLICATION_FACTORof 3. -
Re-replication Triggered: The Metadata Server decides to restore the third copy:
- It picks Node A as the "source" (it has a healthy copy of
Chunk_123). - It picks Node C (or any other available healthy node) as the "destination." Let's assume Node C is already busy, so it finds a new healthy Node D.
- The Metadata Server sends a command to Node A: "Hey Node A, please send a copy of
Chunk_123to Node D."
- It picks Node A as the "source" (it has a healthy copy of
-
Chunk Transfer: Node A loads
Chunk_123from its local disk and sends it over the network to Node D. -
New Copy Stored: Node D receives
Chunk_123and saves it to its disk. -
Metadata Updated: Node A confirms the transfer to the Metadata Server. The Metadata Server updates its records to show that
Chunk_123now has copies on Node A, Node C, and Node D.
Your data is safe again, and you didn't have to do anything!
The "guardian" of cdfs-ton lives primarily within the Metadata Server, specifically in a special background process called the replication_monitor thread. This thread constantly runs, performing all the health checks and re-replication tasks.
The Metadata Server needs to know about all the Storage Nodes in the system. It stores this information in an array of storage_node_t structures:
// File: metadata_server/metadata_server.c
typedef struct {
uint8_t ip[16]; // IP address of the Storage Node
int32_t port; // Port the Storage Node is listening on
time_t last_seen; // When was the last heartbeat received?
int32_t dead; // 1 if declared dead, 0 otherwise
} storage_node_t;
static storage_node_t active_nodes[MAX_NODES]; // Array of all known nodes
static int32_t active_node_count = 0; // How many nodes are there?Each storage_node_t entry tells the Metadata Server the node's network address, when it was last heard from, and if it's currently considered "dead."
This is the main function that handles fault tolerance. It runs in an endless loop, checking node health and ensuring replication.
// File: metadata_server/metadata_server.c (simplified)
void* replication_monitor(void *arg) {
(void)arg; // Argument not used
while (1) {
sleep(30); // Wait for 30 seconds before checking again
time_t now = time(NULL);
pthread_mutex_lock(&nodes_mutex); // Lock to safely access node list
// --- 1. Detect Dead Nodes and Trigger Re-replication ---
for (int32_t i = 0; i < active_node_count; i++) {
if (!active_nodes[i].dead && (now - active_nodes[i].last_seen) > 15) {
LOG_WARN("MD", "Node %s:%d appears dead!\n", active_nodes[i].ip, active_nodes[i].port);
active_nodes[i].dead = 1; // Mark as dead
// Logic to find source and destination nodes for re-replication
int32_t src_idx = -1; // Find a live node to copy FROM
int32_t dst_idx = -1; // Find another live node to copy TO
// ... (simplified: find src_idx and dst_idx) ...
if (src_idx < 0 || dst_idx < 0) {
LOG_ERR("MD", "Not enough living nodes to re-replicate.\n");
continue;
}
// Iterate through all chunks *this dead node* held
for (int32_t c = 0; c < node_chunk_counts[i]; c++) {
int32_t cid = node_chunks[i][c]; // Get chunk ID
size_t chunk_sz = get_chunk_size_from_metadata(cid); // Get its size
if (chunk_sz == 0) continue; // Skip unknown chunks
// Connect to the source node and tell it to replicate
int32_t nsock = connect_to_node(active_nodes[src_idx].ip, active_nodes[src_idx].port);
if (nsock < 0) { LOG_ERR("MD", "Cannot connect to src node.\n"); continue; }
req_replicate_chunk_t rreq = { cid, chunk_sz, /* src_ip, src_port, */ dst_ip, dst_port };
net_header_t rhdr = { OP_REPLICATE_CHUNK, sizeof(rreq) };
send_exact(nsock, &rhdr, sizeof(rhdr)); // Send header
send_exact(nsock, &rreq, sizeof(rreq)); // Send replication request
// Wait for confirmation from the source node
// ... (receive ack and check status) ...
close(nsock);
// Update the Metadata Server's records with the new copy location
update_chunk_replica(cid, active_nodes[dst_idx].ip, active_nodes[dst_idx].port);
}
save_fsimage(); // Save changes after re-replication
}
}
// --- 2. Garbage Collection: Delete Orphaned Chunks ---
for (int32_t i = 0; i < active_node_count; i++) {
if (active_nodes[i].dead) continue; // Skip dead nodes
for (int32_t c = 0; c < node_chunk_counts[i]; c++) {
int32_t cid = node_chunks[i][c];
if (is_chunk_orphaned(cid)) {
LOG_INFO("MD", "GC: Deleting orphaned chunk %d from node %s:%d\n", cid, active_nodes[i].ip, active_nodes[i].port);
// Connect to the Storage Node and tell it to delete the chunk
// ... (send OP_DELETE_CHUNK request) ...
}
}
}
pthread_mutex_unlock(&nodes_mutex);
// Periodically compact the edit log (explained in Chapter 7)
// ... (compact_edit_log();) ...
}
return NULL;
}-
sleep(30): This makes the thread pause for 30 seconds before going through the loop again, so it's not constantly hogging the CPU. -
now - active_nodes[i].last_seen > 15: This is the core logic for detecting a dead node. If the time difference is more than 15 seconds, the node is markeddead = 1. -
node_chunks[i]andnode_chunk_counts[i]: These arrays (managed by the Metadata Server) store the block reports received from each Storage Node, listing all the chunks they hold. This is how the Metadata Server knows which chunks were on the dead node. -
connect_to_node(...): The Metadata Server uses this to establish a network connection to a Storage Node to give it instructions. -
OP_REPLICATE_CHUNK: This is an operation code (from Network Communication Protocol) that the Metadata Server sends to a source Storage Node, telling it to send a chunk to a destination Storage Node. -
update_chunk_replica: Once a chunk has been successfully re-replicated, the Metadata Server updates its internal file_metadata_t records to include the new location.
Here's a simplified sequence of re-replication:
sequenceDiagram
participant MS as Metadata Server
participant SN_DEAD as Storage Node (Dead)
participant SN_SRC as Storage Node (Source)
participant SN_DST as Storage Node (Destination)
Note over MS: 1. Detects SN_DEAD stopped sending heartbeats
MS->>MS: 2. Marks SN_DEAD as dead
MS->>MS: 3. Identifies under-replicated chunks
MS->>MS: 4. Finds SN_SRC (has copy) and SN_DST (new home)
MS->>SN_SRC: 5. OP_REPLICATE_CHUNK (Chunk ID, SN_DST IP/Port)
SN_SRC->>SN_SRC: 6. Loads Chunk Data from local disk
SN_SRC->>SN_DST: 7. OP_STORE_CHUNK (Chunk ID, Data, Checksum)
SN_DST->>SN_DST: 8. Stores Chunk Data to local disk
SN_DST-->>SN_SRC: 9. Acknowledges storage (Success!)
SN_SRC-->>MS: 10. Acknowledges replication (Success!)
MS->>MS: 11. Updates metadata (Chunk ID now on SN_DST)
After a chunk is re-replicated to a new Storage Node, the Metadata Server must update its records. The update_chunk_replica function does this:
// File: metadata_server/metadata.c (simplified)
void update_chunk_replica(int32_t chunk_id, const uint8_t *new_ip, int32_t new_port) {
pthread_mutex_lock(&namespace_mutex); // Lock to safely update metadata
for (int32_t i = 0; i < file_count; i++) { // Loop through all files
for (int32_t j = 0; j < files[i].chunk_count; j++) { // Loop through chunks in each file
if (files[i].chunks[j].chunk_id == chunk_id) {
chunk_info_t *ci = &files[i].chunks[j];
// If there's space for another replica:
if (ci->node_count < REPLICATION_FACTOR) {
// Add the new Storage Node's IP and port to the chunk_info_t
strncpy((char *)ci->node_ips[ci->node_count], (const char *)new_ip, 15);
ci->node_ports[ci->node_count] = new_port;
ci->node_count++; // Increment the count of replicas
}
}
}
}
pthread_mutex_unlock(&namespace_mutex);
}This function finds the specific chunk_info_t (part of the file_metadata_t) for the re-replicated chunk and adds the new Storage Node's location to its list of replica holders.
The replication_monitor also runs a "garbage collection" routine to find chunks that exist on Storage Nodes but are no longer referenced by any file in the Metadata Server's catalog.
// File: metadata_server/metadata.c (simplified)
int32_t is_chunk_orphaned(int32_t chunk_id) {
pthread_mutex_lock(&namespace_mutex);
// Check if the chunk_id is linked to any file in the metadata
for (int32_t fi = 0; fi < file_count; fi++) {
for (int32_t ci = 0; ci < files[fi].chunk_count; ci++) {
if (files[fi].chunks[ci].chunk_id == chunk_id) {
pthread_mutex_unlock(&namespace_mutex);
return 0; // Found it, not an orphan
}
}
}
// Heuristic: if it's a very recently allocated chunk, it might still be uploading.
// Give it some grace period before declaring it an orphan.
if (chunk_id >= next_chunk_id - 500) {
pthread_mutex_unlock(&namespace_mutex);
return 0;
}
pthread_mutex_unlock(&namespace_mutex);
return 1; // Not found anywhere, it's an orphan!
}If is_chunk_orphaned returns 1, the Metadata Server will send an OP_DELETE_CHUNK request to the Storage Node to remove that chunk from its disk, keeping the system clean.
Fault Tolerance & Replication Monitoring is the unsung hero of cdfs-ton. It silently works in the background, making sure your data is always safe and available. By constantly monitoring Storage Nodes, detecting failures, and automatically re-replicating lost data chunks, cdfs-ton provides a robust and reliable storage solution. You can rest easy knowing that even if a part of the system breaks, your files are protected.
Next, we'll learn about System Configuration, understanding how to set up and customize cdfs-ton to fit your needs.