Skip to content

04_network_communication_protocol.md

Jump to bottom
Lavnish edited this page Mar 31, 2026 · 2 revisions

Chapter 4: Network Communication Protocol

Welcome back, cdfs-ton adventurer! In our previous chapters, we explored how you, as a user, interact with cdfs-ton through the Client Application Interface, how the Metadata Server acts as the system's brain, and how Storage Nodes diligently store your file chunks.

Now, imagine our client, metadata server, and storage nodes are all team members in a big project. They all have important jobs, but how do they talk to each other? What if the client tries to ask for a file, but the metadata server expects a request to store a chunk? It would be pure chaos! They'd be speaking different "languages."

This is where the Network Communication Protocol comes in. It's the standardized "language" and the set of rules that all parts of cdfs-ton (client, metadata server, storage nodes) use to understand each other when they communicate over the network. It's like everyone agrees to use the same dictionary and grammar rules.

Why Do We Need a Protocol?

Think about how you use a smartphone. When you tap an icon, your phone knows exactly what app to open. When you send a text, your phone knows how to package that message and send it to the right person. This happens because there's an agreed-upon way for apps, your phone's operating system, and the network to communicate.

In cdfs-ton, the Metadata Server and Storage Nodes are separate programs, often running on different computers. When your cdfs_client application wants to, say, put a file:

  1. It needs to tell the Metadata Server: "I want to register a new file."
  2. It needs to tell a Storage Node: "I want to store this chunk."
  3. Later, when retrieving, it needs to tell a Storage Node: "I want to load chunk ID 123."

Without a clear protocol, these components wouldn't understand each other. The protocol defines:

  • What kind of request is this? (e.g., "This is a request to store data!")
  • What information is included? (e.g., "Here's the chunk ID, its size, and the actual data.")
  • How should the information be structured? (e.g., "Always send the request type first, then the size of the following data.")
  • How do we know the message arrived correctly? (e.g., "Let's check the data's 'fingerprint'!")

The "Language" of CDFS-TON: Messages and Rules

The cdfs-ton protocol defines how messages are built and exchanged. Every message sent between components generally follows a two-part structure: a Header and a Payload.

1. The net_header_t: The Message Envelope

Every single message in cdfs-ton starts with a small, fixed-size network header. Think of this as the envelope of your message. It tells the recipient two crucial things right away:

  • op_code (Operation Code): What kind of action or request is inside this message? (e.g., "This is a 'register file' message," or "This is a 'store chunk' message.")
  • payload_size: How big is the actual data (the "letter" inside the envelope) that follows this header? This is important because message contents can vary in size.

Here's what this header looks like in common/protocol.h:

// File: common/protocol.h
typedef struct {
    uint32_t op_code;      // What type of operation is this?
    uint32_t payload_size; // How many bytes of data follow?
} net_header_t;
  • uint32_t means an "unsigned 32-bit integer," which is just a number that can store values up to about 4 billion. It's used here to represent different operation codes and sizes.

2. The op_code_t: The Specific Request

The op_code tells the receiving component what to do. It's like a list of predefined verbs. When a Metadata Server or Storage Node receives a message, it first reads the op_code from the net_header_t to know how to interpret the rest of the message.

Here are some examples of op_code_t from common/protocol.h:

// File: common/protocol.h
typedef enum {
    OP_REGISTER_FILE = 1,  // Client -> MD: I've uploaded a file, here's its metadata
    OP_GET_METADATA  = 2,  // Client -> MD: Give me info about this file
    OP_STORE_CHUNK   = 3,  // Client -> SN: Store this chunk for me
    OP_LOAD_CHUNK    = 4,  // Client -> SN: Give me this chunk
    OP_HEARTBEAT     = 5,  // SN -> MD: I'm alive!
    // ... many more operations ...
    OP_ERROR         = 99  // Something went wrong!
} op_code_t;
  • An enum (short for enumeration) is a way to assign names to integer values. So, OP_REGISTER_FILE is just a friendly name for the number 1, OP_GET_METADATA for 2, and so on. This makes code much easier to read and understand.

3. The Payload Structures: The Message Content

After the net_header_t, the actual message content, called the payload, is sent. The structure of this payload depends entirely on the op_code. For example:

  • If the op_code is OP_STORE_CHUNK, the payload will contain the chunk ID, its size, a checksum, and then the raw chunk data itself.
  • If the op_code is OP_REGISTER_FILE, the payload will contain the filename, the number of chunks, and then information about each chunk (like its ID and where it's stored).

Here's an example of a payload structure for storing a chunk, found in common/protocol.h:

// File: common/protocol.h
// OP_STORE_CHUNK (Client -> Storage Node)
typedef struct {
    int32_t chunk_id;   // Unique ID of the chunk
    size_t size;        // How big is this chunk's data?
    uint32_t checksum;  // A 'fingerprint' to check data integrity
    // Followed by exactly `size` bytes of raw chunk data
} req_store_chunk_t;

// Example response from Storage Node after storing (SN -> Client)
typedef struct {
    int32_t status; // 0 for success, -1 for error
} resp_store_chunk_t;
  • Notice the comment: "Followed by exactly size bytes of raw chunk data." This means the payload_size in the net_header_t would be sizeof(req_store_chunk_t) + size (where size is the chunk's data size). The receiver knows to expect the struct, then the additional raw data.

4. Reliable Sending and Receiving: send_exact and recv_exact

When sending data over a network, it's not always guaranteed that all bytes will be sent or received in one go. Sometimes, the network might send only part of a message, or receive only part of it. This is unreliable!

To solve this, cdfs-ton uses special helper functions: send_exact and recv_exact. These functions (from common/serialization.h and common/serialization.c) ensure that all the bytes you intend to send actually get sent, and all the bytes you expect to receive actually arrive.

Imagine sending a long letter by mail. send_exact is like a postal worker who makes sure every page of your letter is in the envelope before it leaves. recv_exact is like a diligent recipient who won't say "I got the letter!" until every single page has arrived.

Use Case: Client Stores a Chunk on a Storage Node

Let's trace how your cdfs_client uses this protocol to tell a Storage Node to store a chunk. This happens after the client has contacted the Metadata Server and found out which storage nodes are available (as we saw in Chapter 1: Client Application Interface).

  1. Client Prepares: The client has a chunk of data, its chunk_id, its size, and has calculated its checksum.
  2. Client Builds req_store_chunk_t: It fills in a req_store_chunk_t structure with this information.
  3. Client Builds net_header_t: It creates a net_header_t with OP_STORE_CHUNK as the op_code and calculates sizeof(req_store_chunk_t) + chunk_data_size for the payload_size.
  4. Client Sends:
    • It first uses send_exact to send the net_header_t.
    • Then, it uses send_exact to send the req_store_chunk_t.
    • Finally, it uses send_exact to send the raw chunk data.
  5. Storage Node Receives:
    • The Storage Node first uses recv_exact to read the incoming net_header_t.
    • It sees op_code = OP_STORE_CHUNK and knows to expect a req_store_chunk_t and then raw data.
    • It uses recv_exact to read the req_store_chunk_t.
    • It then uses recv_exact (for the payload_size minus sizeof(req_store_chunk_t)) to read the actual chunk data.
  6. Storage Node Processes: It stores the chunk data on its disk, verifies the checksum, and prepares a response.
  7. Storage Node Responds: It sends back a net_header_t with an op_code indicating success or failure, followed by a resp_store_chunk_t payload.

Here's a simplified sequence diagram showing this interaction:

sequenceDiagram
    participant C as CDFS Client
    participant SN as Storage Node

    C->>SN: 1. Send net_header_t (OP_STORE_CHUNK, total_payload_size)
    C->>SN: 2. Send req_store_chunk_t (chunk_id, size, checksum)
    C->>SN: 3. Send raw chunk data (size bytes)

    SN->>SN: 4. Storage Node receives and processes
    SN->>SN: 5. Saves chunk to disk, verifies checksum

    SN-->>C: 6. Send net_header_t (OP_STORE_CHUNK, resp_payload_size)
    SN-->>C: 7. Send resp_store_chunk_t (status: 0 for success)
Loading

Diving Deeper: Inside the Protocol Code

Let's look at the actual C code that makes this happen.

common/protocol.h: Defining the Language

This header file is like the dictionary for cdfs-ton. It defines all the operation codes (op_code_t) and the structures (net_header_t, req_store_chunk_t, etc.) that make up the messages.

// File: common/protocol.h (snippet)
// Operation codes
typedef enum {
    OP_REGISTER_FILE = 1,
    OP_GET_METADATA  = 2,
    OP_STORE_CHUNK   = 3,
    OP_LOAD_CHUNK    = 4,
    // ...
    OP_ERROR         = 99
} op_code_t;

// Common request/response header
typedef struct {
    uint32_t op_code;
    uint32_t payload_size;
} net_header_t;

// OP_STORE_CHUNK request
typedef struct {
    int32_t chunk_id;
    size_t size;
    uint32_t checksum;
    // raw chunk data follows this struct
} req_store_chunk_t;

// OP_STORE_CHUNK response
typedef struct {
    int32_t status; // 0 for success, -1 for error
} resp_store_chunk_t;

This file is included by all cdfs-ton components, ensuring they all "speak" the same dialect and understand the format of every message.

common/serialization.c: Ensuring Complete Messages

This file contains the crucial send_exact and recv_exact functions, which are used everywhere a message is sent or received.

// File: common/serialization.c (simplified send_exact)
int32_t send_exact(int32_t sockfd, const void *buf, size_t len) {
    size_t total_sent = 0;
    const uint8_t *p = (const uint8_t *)buf; // Pointer to current position
    while (total_sent < len) { // Keep sending until all bytes are sent
        ssize_t s = send(sockfd, p + total_sent, len - total_sent, 0);
        if (s <= 0) return -1; // Error or connection closed
        total_sent += s;       // Update count of bytes sent
    }
    return 0; // All bytes sent successfully
}
  • sockfd: This is the "socket file descriptor," an ID number for an open network connection (like a phone line).
  • buf: This is a pointer to the data you want to send.
  • len: This is the total number of bytes you want to send.
  • The while loop is the key here. It keeps calling send (which might only send some bytes) until total_sent equals len.

The recv_exact function works similarly but for receiving:

// File: common/serialization.c (simplified recv_exact)
int32_t recv_exact(int32_t sockfd, void *buf, size_t len) {
    size_t total_recv = 0;
    uint8_t *p = (uint8_t *)buf; // Pointer to current position
    while (total_recv < len) { // Keep receiving until all bytes are received
        ssize_t r = recv(sockfd, p + total_recv, len - total_recv, 0);
        if (r <= 0) return -1; // Error or connection closed
        total_recv += r;       // Update count of bytes received
    }
    return 0; // All bytes received successfully
}
  • This function ensures that the program waits and keeps trying to receive until the full len bytes have arrived, preventing incomplete messages from being processed.

Checksums: Data Integrity's Fingerprint

The protocol also includes checksum values (also defined and implemented in common/serialization.h and .c). A checksum is a small value calculated from a larger block of data. If even one byte in the data changes, the checksum will almost certainly change.

  • When the client sends a chunk to a Storage Node, it calculates the chunk's checksum and sends it along.
  • The Storage Node, upon receiving the chunk, calculates its own checksum from the received data.
  • If the two checksums match, the Storage Node knows the data arrived without corruption. If they don't, something went wrong during transmission, and the Storage Node will reject the chunk (as we saw in Chapter 3: Chunk Storage & Management).

This is a vital part of the protocol, ensuring that the precious data you store in cdfs-ton remains accurate and uncorrupted.

Conclusion

The Network Communication Protocol is the invisible but essential glue that holds cdfs-ton together. It provides the standardized "language" and rules for all components (clients, Metadata Server, and Storage Nodes) to talk to each other reliably. By using a clear message structure with headers, operation codes, precise payload definitions, and helper functions like send_exact and recv_exact, cdfs-ton ensures that every request is understood, every piece of data is completely transferred, and every byte arrives correctly and in order. Without this common language, our distributed file system wouldn't be able to function as a cohesive unit.

Next, we'll explore how cdfs-ton deals with problems and ensures your data is always available, even if parts of the system fail, by diving into Fault Tolerance & Replication Monitoring.

Clone this wiki locally