Skip to content

03_chunk_storage___management.md

Jump to bottom
Lavnish edited this page Mar 31, 2026 · 2 revisions

Chapter 3: Chunk Storage & Management

Welcome back to cdfs-ton! In Chapter 1: Client Application Interface, you learned how to use simple commands like put and get. In Chapter 2: Metadata Management, we explored the "control tower" (the Metadata Server) that keeps track of all your file information, like their names, how they're broken into chunks, and which storage locations hold those chunks.

But here's the big question: where are these "storage locations"? And how do they actually store and retrieve the physical pieces of your files?

This is where Chunk Storage & Management comes in! This chapter focuses on the "workhorses" of cdfs-ton – the Storage Nodes. Imagine them as individual, dedicated storage lockers or shelves in our giant library. They don't know the full story of your file (that's the Metadata Server's job), but they are experts at handling the individual chunks of data they are assigned.

The Storage Nodes: Your Data's Home

When you put a file into cdfs-ton, your client application (after getting instructions from the Metadata Server) sends the file's chunks directly to various Storage Nodes. Each Storage Node then saves its assigned chunks onto its own local disk.

Similarly, when you get a file, your client asks the Metadata Server where to find its chunks. Then, your client directly contacts the relevant Storage Nodes, which then retrieve and send those chunks back to your client.

In short, Storage Nodes are responsible for:

  1. Storing Chunks: Physically saving file chunks to their local disk.
  2. Retrieving Chunks: Finding and sending requested chunks back to clients or other nodes.
  3. Reporting: Regularly telling the Metadata Server which chunks they hold and their overall health.
  4. Replicating: Making copies of chunks and sending them to other Storage Nodes for safety.

How a Storage Node Works (The Big Picture)

Let's follow a chunk of your my_vacation.jpg file as it interacts with a Storage Node.

Storing a Chunk (Part of a cdfs_put request)

When your cdfs_client needs to store a chunk:

  1. Client Connects: Your cdfs_client connects directly to a chosen Storage Node.
  2. Send Request & Data: It sends a message (an OP_STORE_CHUNK request) to the Storage Node, saying, "Here's chunk ID 123, it's this big, and here's its data." It also sends a "checksum" (a unique digital fingerprint of the data) to make sure the chunk isn't corrupted during transfer.
  3. Save to Disk: The Storage Node receives the chunk data and saves it as a file on its local hard drive. It also saves the checksum separately.
  4. Confirm Storage: The Storage Node replies to the client, saying, "Got it! Chunk 123 is safely stored."

Retrieving a Chunk (Part of a cdfs_get request)

When your cdfs_client needs to retrieve a chunk:

  1. Client Connects: Your cdfs_client connects directly to a Storage Node known to hold chunk ID 123.
  2. Send Request: It sends a message (an OP_LOAD_CHUNK request) to the Storage Node, saying, "Please give me chunk ID 123."
  3. Load from Disk: The Storage Node finds chunk ID 123 on its local disk. It also reads the stored checksum.
  4. Send Data & Checksum: The Storage Node sends the chunk data and its checksum back to the client.
  5. Confirm Retrieval: The Storage Node replies, "Here's your chunk!"

Keeping the Metadata Server Informed

Storage Nodes don't just passively store data. They actively communicate with the Metadata Server:

  • Heartbeats: Every few seconds, each Storage Node sends a "heartbeat" message to the Metadata Server, saying, "I'm alive and well!" This tells the Metadata Server which nodes are active.
  • Block Reports: Periodically, Storage Nodes also send a "block report" to the Metadata Server. This is a list of all the chunk IDs they currently hold. This is how the Metadata Server can update its records and know exactly where every single chunk is stored.
sequenceDiagram
    participant C as CDFS Client
    participant MS as Metadata Server
    participant SN1 as Storage Node 1
    participant SN2 as Storage Node 2

    Note over C,SN2: Initial setup (from previous chapters)
    C->>MS: 1. Request file metadata for "my_vacation.jpg"
    MS-->>C: 2. Here are chunk IDs and their locations (e.g., Chunk 1 is on SN1, SN2)

    C->>SN1: 3. Request Chunk 1 data (OP_LOAD_CHUNK)
    SN1->>SN1: 4. Read Chunk 1 from local disk
    SN1-->>C: 5. Send Chunk 1 data & checksum

    C->>SN2: 6. Request Chunk 2 data (OP_LOAD_CHUNK)
    SN2->>SN2: 7. Read Chunk 2 from local disk
    SN2-->>C: 8. Send Chunk 2 data & checksum

    Note over C: Client reconstructs file from all chunks

    Note over SN1,SN2: Background operations
    loop every few seconds
        SN1->>MS: Heartbeat (I'm alive!)
        SN2->>MS: Heartbeat (I'm alive!)
    end
    loop periodically
        SN1->>MS: Block Report (List of chunks I hold)
        SN2->>MS: Block Report (List of chunks I hold)
    end
Loading

Inside a Storage Node: The storage_node/storage.c and storage_node/storage_node.c Files

The magic of Storage Nodes happens primarily in two files:

  • storage_node/storage.c: This file contains the core logic for how chunks are saved to and loaded from the local disk. It's like the detailed instructions for managing the contents of each locker.
  • storage_node/storage_node.c: This file contains the main program for the Storage Node, including how it listens for network requests (from clients or the Metadata Server) and dispatches them to the storage.c functions. It's like the main interface for the entire storage locker facility.

Let's dive into some key functions!

Setting Up the Storage Directory (storage_init)

Before a Storage Node can store anything, it needs a place on its disk. The storage_init function handles this.

// File: storage_node/storage.c
static uint8_t g_storage_dir[256] = "./storage_data";

void storage_init(const uint8_t *dir) {
    strncpy((char *)g_storage_dir, (const char *)dir, sizeof(g_storage_dir) - 1);
    g_storage_dir[sizeof(g_storage_dir) - 1] = '\0';
    mkdir((const char *)g_storage_dir, 0755);
}
  • g_storage_dir: This is a global variable that holds the path to the directory where all chunks will be stored (e.g., ./storage_data).
  • storage_init: This function takes a directory path (dir) and uses mkdir to create that directory if it doesn't already exist. This ensures the Storage Node has a dedicated place for its chunks.

Storing a Chunk Stream (store_chunk_stream)

This is the core function a Storage Node uses to receive chunk data over the network and save it to its local disk.

// File: storage_node/storage.c (simplified)
int32_t store_chunk_stream(int32_t chunk_id, int32_t sockfd, size_t size, uint32_t expected_checksum) {
    char path[512];
    // Create a unique file name for the chunk data
    snprintf(path, sizeof(path), "%s/chunk_%d.dat", (char *)g_storage_dir, chunk_id);

    FILE *fp = fopen(path, "wb"); // Open file for writing (binary mode)
    if (!fp) return -1; // Error opening file

    uint8_t buf[8192];
    size_t remaining = size;
    uint32_t calc_chk = CHKSUM_INIT; // Initialize checksum

    // Loop to receive data from the network and write to file
    while (remaining > 0) {
        size_t to_read = remaining < sizeof(buf) ? remaining : sizeof(buf);
        if (recv_exact(sockfd, buf, to_read) != 0) { // Receive bytes from socket
            fclose(fp); remove(path); return -1; // Error, clean up
        }
        fwrite(buf, 1, to_read, fp); // Write bytes to file
        calc_chk = update_checksum(calc_chk, buf, to_read); // Update checksum
        remaining -= to_read;
    }
    fclose(fp);

    if (calc_chk != expected_checksum) { // Check if data was corrupted
        LOG_WARN("SN", "Checksum mismatch for chunk %d!\n", chunk_id);
        remove(path); return -1; // Mismatch, delete corrupted file
    }

    // Save the checksum in a separate .crc file
    snprintf(path, sizeof(path), "%s/chunk_%d.crc", (char *)g_storage_dir, chunk_id);
    fp = fopen(path, "wb");
    if (fp) { fwrite(&expected_checksum, sizeof(uint32_t), 1, fp); fclose(fp); }
    return 0; // Success
}
  • chunk_%d.dat: Each chunk's data is saved in a file named chunk_ID.dat (e.g., chunk_123.dat).
  • fopen and fwrite: These standard C functions are used to open a file and write the received network data into it.
  • recv_exact: This helper function (from common/serialization.h) ensures that exactly to_read bytes are received from the network socket (sockfd), preventing incomplete data.
  • Checksum Verification: After all data is received, the Storage Node calculates its own checksum (calc_chk) and compares it to the expected_checksum sent by the client. If they don't match, the data is considered corrupted, and the chunk file is deleted. This is crucial for data integrity!
  • chunk_%d.crc: The correct checksum is saved in a separate file (e.g., chunk_123.crc) alongside the data file.

Loading a Chunk Stream (load_chunk_stream)

This function allows a Storage Node to read a chunk from its local disk and send it over the network.

// File: storage_node/storage.c (simplified)
int32_t load_chunk_stream(int32_t chunk_id, int32_t sockfd, size_t file_size, uint32_t *out_checksum) {
    char path[512];

    // First, read the checksum from the .crc file
    snprintf(path, sizeof(path), "%s/chunk_%d.crc", (char *)g_storage_dir, chunk_id);
    FILE *fp = fopen(path, "rb");
    if (fp) { fread(out_checksum, sizeof(uint32_t), 1, fp); fclose(fp); }
    else { *out_checksum = 0; } // If no checksum file, assume 0

    // Then, open and stream the chunk data from the .dat file
    snprintf(path, sizeof(path), "%s/chunk_%d.dat", (char *)g_storage_dir, chunk_id);
    fp = fopen(path, "rb");
    if (!fp) return -1; // Error: chunk not found

    uint8_t buf[8192];
    size_t remaining = file_size;
    // Loop to read data from file and send over network
    while (remaining > 0) {
        size_t to_read = remaining < sizeof(buf) ? remaining : sizeof(buf);
        size_t bytes = fread(buf, 1, to_read, fp); // Read bytes from file
        if (bytes == 0) break; // End of file

        if (send_exact(sockfd, buf, bytes) != 0) { // Send bytes over socket
            fclose(fp); return -1; // Error sending
        }
        remaining -= bytes;
    }
    fclose(fp);

    if (remaining > 0) return -1; // Incomplete file
    return 0; // Success
}
  • Reading Checksum: The function first opens the .crc file to retrieve the stored checksum.
  • fread and send_exact: It then opens the .dat file, reads chunks of data using fread, and sends them over the network socket (sockfd) using send_exact.

Deleting Chunk Files (delete_chunk_files)

When a file is removed from cdfs-ton, the Metadata Server tells the relevant Storage Nodes to delete its chunks.

// File: storage_node/storage.c (simplified)
int32_t delete_chunk_files(int32_t chunk_id) {
    char path[512];
    snprintf(path, sizeof(path), "%s/chunk_%d.dat", (char *)g_storage_dir, chunk_id);
    remove(path); // Delete the data file
    snprintf(path, sizeof(path), "%s/chunk_%d.crc", (char *)g_storage_dir, chunk_id);
    remove(path); // Delete the checksum file
    return 0;
}
  • remove: This standard C function is used to delete both the .dat and .crc files associated with a given chunk_id.

Scanning for Chunks (scan_chunks)

This function is used when a Storage Node needs to generate a "block report" to send to the Metadata Server. It scans its storage directory to find all the chunk files it currently holds.

// File: storage_node/storage.c (simplified)
int32_t scan_chunks(int32_t *out_ids, int32_t max_count) {
    DIR *dir = opendir((char *)g_storage_dir); // Open the storage directory
    if (!dir) return 0;

    int32_t count = 0;
    struct dirent *entry;
    // Loop through each item in the directory
    while ((entry = readdir(dir)) != NULL && count < max_count) {
        int32_t chunk_id;
        // Check if the filename matches "chunk_ID.dat" pattern
        if (sscanf(entry->d_name, "chunk_%d.dat", &chunk_id) == 1) {
            out_ids[count++] = chunk_id; // Add chunk ID to the list
        }
    }
    closedir(dir); // Close the directory
    return count; // Return total chunks found
}
  • opendir and readdir: These functions are used to list the contents of the g_storage_dir.
  • sscanf: This function tries to extract a number (the chunk_id) from filenames that match the chunk_ID.dat pattern.
  • out_ids: This array will be filled with the IDs of all chunks found on the node, which is then sent to the Metadata Server in a block report.

The Heartbeat and Block Report Thread (heartbeat_thread)

This is a separate, continuous process within each Storage Node that handles its regular communication with the Metadata Server.

// File: storage_node/storage_node.c (simplified)
void* heartbeat_thread(void* arg) {
    (void)arg; // Unused argument
    while (1) { // Loop forever
        // --- Send Heartbeat ---
        int32_t sock = connect_to(g_config.meta_ip, g_config.meta_port);
        if (sock >= 0) {
            req_heartbeat_t hb = { storage_port, 1024 /*dummy space*/ };
            net_header_t hdr   = { OP_HEARTBEAT, sizeof(hb) };
            send_exact(sock, &hdr, sizeof(hdr));
            send_exact(sock, &hb, sizeof(hb));
            close(sock);
        }

        // --- Send Block Report ---
        sock = connect_to(g_config.meta_ip, g_config.meta_port);
        if (sock >= 0) {
            req_block_report_t report;
            report.storage_port = storage_port;
            // Use scan_chunks to get list of held chunks
            report.chunk_count  = scan_chunks(report.chunk_ids, MAX_CHUNKS);

            net_header_t hdr = { OP_BLOCK_REPORT, sizeof(report) };
            send_exact(sock, &hdr, sizeof(hdr));
            send_exact(sock, &report, sizeof(report));
            close(sock);
        }
        sleep(5); // Wait 5 seconds before repeating
    }
    return NULL;
}
  • connect_to: A helper function to establish a network connection to the Metadata Server.
  • OP_HEARTBEAT: This operation code signals a heartbeat message. It tells the Metadata Server that this Storage Node is active.
  • OP_BLOCK_REPORT: This operation code signals a block report. The scan_chunks function is called to get a list of all chunk IDs on this node, which is then sent to the Metadata Server.
  • sleep(5): This causes the thread to pause for 5 seconds, ensuring heartbeats and block reports are sent regularly, but not constantly.

Handling Chunk Replication (OP_REPLICATE_CHUNK)

Sometimes, the Metadata Server might notice that a chunk doesn't have enough copies (replicas). It then instructs one Storage Node to copy that chunk to another. This is handled by the OP_REPLICATE_CHUNK operation.

// File: storage_node/storage_node.c (simplified handle_client function)
void handle_client(int32_t client_sock) {
    net_header_t header;
    recv_exact(client_sock, &header, sizeof(header)); // Read request header

    // ... (other op_code handlers like OP_STORE_CHUNK, OP_LOAD_CHUNK) ...

    if (header.op_code == OP_REPLICATE_CHUNK) {
        req_replicate_chunk_t req;
        recv_exact(client_sock, &req, sizeof(req)); // Read replication request

        // This node (the source) loads the chunk data
        // ... (find chunk_id, get its size and checksum from local disk) ...

        // Connect to the destination Storage Node
        int32_t dst_sock = connect_to(req.dst_ip, req.dst_port);
        if (dst_sock >= 0) {
            // Send an OP_STORE_CHUNK request to the destination node
            req_store_chunk_t sreq  = { req.chunk_id, st.st_size, checksum };
            net_header_t      shdr  = { OP_STORE_CHUNK, (uint32_t)(sizeof(sreq) + st.st_size) };
            send_exact(dst_sock, &shdr,  sizeof(shdr));
            send_exact(dst_sock, &sreq,  sizeof(sreq));

            // Stream the actual chunk data to the destination node
            int32_t push_status = load_chunk_stream(req.chunk_id, dst_sock, st.st_size, &checksum);

            // ... (receive confirmation from destination node) ...
            close(dst_sock);
        }
        // ... (send response back to Metadata Server) ...
    }
    close(client_sock);
}
  • When a Storage Node receives an OP_REPLICATE_CHUNK request from the Metadata Server, it acts as a "source" node.
  • It first reads the specified chunk from its own local disk using load_chunk_stream.
  • Then, it connects to the destination Storage Node (whose IP and port are provided in the req_replicate_chunk_t structure).
  • It sends an OP_STORE_CHUNK request to the destination node and streams the chunk data, effectively telling the destination node, "Hey, store this chunk!"
  • This shows how Storage Nodes can also communicate with each other, orchestrated by the Metadata Server, to maintain data replication.

Conclusion

The Storage Nodes are the physical backbone of cdfs-ton, the "storage lockers" that actually hold your file chunks. They efficiently store and retrieve data on their local disks, verify data integrity using checksums, and keep the Metadata Server updated on their status and holdings through heartbeats and block reports. Their ability to replicate chunks ensures your data remains safe and available, even if some nodes fail.

Now that we understand how clients interact with the system, how metadata is managed, and how chunks are stored, we need to understand the language they speak. In the next chapter, we'll explore the Network Communication Protocol that ties all these components together.

Clone this wiki locally