01_client_application_interface.md

Chapter 1: Client Application Interface

Welcome to the exciting world of cdfs-ton, your very own distributed file system! Think of cdfs-ton as a super smart storage system spread across many computers. But how do you, as a user, tell it what to do? That's where the Client Application Interface comes in.

Imagine you have a powerful robot that can store and retrieve your files from anywhere, reliably, and super fast. You wouldn't want to type complex commands to control each part of the robot, right? You'd want a simple remote control!

The Client Application Interface is exactly like that remote control for cdfs-ton. It's the primary tool you'll use to interact with your distributed file system. It hides all the complex technical details, allowing you to focus on managing your files with easy-to-understand commands.

Your Remote Control Buttons

The cdfs_client program provides you with a set of simple commands, just like buttons on a remote. Each button does a specific job:

put (Upload a file): This button lets you upload a file from your local computer into the cdfs-ton system.
get (Download a file): Need a file back from cdfs-ton? This button fetches it and saves it to your local computer.
ls (List files): Ever wonder what files you have stored? This button shows you a list of files in cdfs-ton.
rm (Delete a file): If you no longer need a file, this button helps you remove it from the system.
status (Check system health): Curious about how your cdfs-ton system is doing? This button provides a health report.

These commands are powered by functions defined in the common/dfs.h file, which is like the instruction manual for your remote control:

// File: common/dfs.h
// ... (other definitions) ...

// Client API functions
int32_t cdfs_put(const uint8_t *local_path, const uint8_t *cdfs_path);
int32_t cdfs_get(const uint8_t *cdfs_path, const uint8_t *local_path);
int32_t cdfs_ls(const uint8_t *cdfs_path);
int32_t cdfs_rm(const uint8_t *cdfs_path);
int32_t cdfs_status(void);

#endif

These functions are the core of the client interface. They are what the cdfs_client program calls when you type a command.

How to Use Your Remote Control

Let's imagine you want to upload your favorite photo, my_vacation.jpg, to cdfs-ton.

Uploading a file (`put`)

You would use the put command. The cdfs_client program handles your request by calling the cdfs_put function:

// File: main.c (simplified)
// ...
int main(int argc, char *argv[]) {
    // ... command parsing logic ...
    if (strcmp(argv[1], "put") == 0) {
        // Here, argv[2] is local_path, argv[3] is cdfs_path
        int32_t status = cdfs_put((const uint8_t *)argv[2], (const uint8_t *)argv[3]);
        if (status == 0) printf("Successfully put '%s' -> '%s'\n", argv[2], argv[3]);
        else             printf("Failed to put file.\n");
    }
    // ... other commands ...
    return 0;
}

Example Input:

cdfs_client put my_vacation.jpg /photos/summer/my_vacation.jpg

Example Output:

Successfully put 'my_vacation.jpg' -> '/photos/summer/my_vacation.jpg'

This command uploads your local my_vacation.jpg file and stores it in cdfs-ton under the path /photos/summer/my_vacation.jpg.

Downloading a file (`get`)

To get that photo back, you'd use get:

// File: main.c (simplified)
// ...
int main(int argc, char *argv[]) {
    // ... command parsing logic ...
    else if (strcmp(argv[1], "get") == 0) {
        // Here, argv[2] is cdfs_path, argv[3] is local_path
        int32_t status = cdfs_get((const uint8_t *)argv[2], (const uint8_t *)argv[3]);
        if (status == 0) printf("Successfully retrieved '%s' -> '%s'\n", argv[2], argv[3]);
        else             printf("Failed to get file.\n");
    }
    // ... other commands ...
    return 0;
}

Example Input:

cdfs_client get /photos/summer/my_vacation.jpg downloaded_photo.jpg

Example Output:

Successfully retrieved '/photos/summer/my_vacation.jpg' -> 'downloaded_photo.jpg'

The file /photos/summer/my_vacation.jpg from cdfs-ton is now saved as downloaded_photo.jpg on your local computer.

Listing files (`ls`)

To see what files are in /photos/summer/:

// File: main.c (simplified)
// ...
int main(int argc, char *argv[]) {
    // ... command parsing logic ...
    else if (strcmp(argv[1], "ls") == 0) {
        // Here, argv[2] is cdfs_dir_path
        cdfs_ls((const uint8_t *)argv[2]);
    }
    // ... other commands ...
    return 0;
}

Example Input:

cdfs_client ls /photos/summer/

Example Output:

my_vacation.jpg
another_pic.png

Deleting a file (`rm`)

To remove a file:

// File: main.c (simplified)
// ...
int main(int argc, char *argv[]) {
    // ... command parsing logic ...
    else if (strcmp(argv[1], "rm") == 0) {
        // Here, argv[2] is cdfs_path
        int32_t status = cdfs_rm((const uint8_t *)argv[2]);
        if (status == 0) printf("Deleted '%s'\n", argv[2]);
        else             printf("Failed to delete '%s' (file not found?)\n", argv[2]);
    }
    // ... other commands ...
    return 0;
}

Example Input:

cdfs_client rm /photos/summer/another_pic.png

Example Output:

Deleted '/photos/summer/another_pic.png'

Checking system status (`status`)

To get system health information:

// File: main.c (simplified)
// ...
int main(int argc, char *argv[]) {
    // ... command parsing logic ...
    else if (strcmp(argv[1], "status") == 0) {
        int32_t status = cdfs_status();
        if (status != 0) printf("Failed to get cluster status.\n");
    }
    // ... other commands ...
    return 0;
}

Example Input:

cdfs_client status

Example Output (simplified JSON):

{
  "metadata_server_status": "healthy",
  "storage_nodes": [
    { "id": "node_1", "ip": "192.168.1.10", "chunks_stored": 150 },
    { "id": "node_2", "ip": "192.168.1.11", "chunks_stored": 155 }
  ],
  "total_files": 25,
  "total_chunks": 305
}

What Happens Under the Hood? (Behind the Remote Control)

When you press a button on your cdfs-ton remote control (i.e., run a cdfs_client command), a lot of work happens automatically. The cdfs_client program, specifically the functions in client/dfs_client.c, handles all the heavy lifting.

Let's trace what happens when you use cdfs_put to upload my_vacation.jpg:

Preparation: The client reads my_vacation.jpg from your local computer and divides it into smaller pieces called "chunks". This is like breaking a big book into individual chapters. Each chunk is 16MB in size (CHUNK_SIZE).
Talk to the Control Tower: The client first connects to the Metadata Server. Think of the Metadata Server as the "control tower" of the cdfs-ton system. It doesn't store your actual file data, but it knows where everything is.
Get Chunk IDs: The client asks the Metadata Server for unique IDs for each chunk of my_vacation.jpg.
Find Storage Lockers: The client then asks the Metadata Server which "Storage Nodes" (the "storage lockers" of cdfs-ton where data is actually kept) are available to store the chunks.
Send Chunks to Storage Lockers: For each chunk, the client connects directly to a few selected Storage Nodes and sends the chunk data. It also sends a "checksum" to ensure the data arrives correctly. This process is repeated for each chunk, ensuring multiple copies (replicas) of each chunk are stored for safety.
Register the File: Once all chunks are safely stored, the client tells the Metadata Server: "Hey, I've just uploaded my_vacation.jpg, and here are all its chunk IDs and where each chunk is stored (which Storage Nodes have copies)." The Metadata Server updates its records.
Confirmation: Finally, the Metadata Server tells the client if the file was successfully registered. The client then shows you the "Successfully put..." message.

Here's a simplified sequence diagram to visualize this:

sequenceDiagram
    participant Client
    participant MetadataServer as Metadata Server
    participant StorageNode as Storage Node(s)

    Client->>MetadataServer: 1. Request chunk IDs
    MetadataServer-->>Client: 2. Provide chunk IDs
    Client->>MetadataServer: 3. Request active Storage Nodes
    MetadataServer-->>Client: 4. List of active Storage Nodes
    loop For each file chunk
        Client->>StorageNode: 5. Send chunk data
        StorageNode-->>Client: 6. Confirm chunk stored
    end
    Client->>MetadataServer: 7. Register file metadata (chunks, locations)
    MetadataServer-->>Client: 8. Confirm file registered

Diving into the `cdfs_put` Code

Let's look at some simplified parts of the cdfs_put function in client/dfs_client.c to see how these steps are implemented.

First, connecting to a server:

// File: client/dfs_client.c
static int32_t connect_to_server(const uint8_t *ip, int32_t port) {
    int32_t sock = 0;
    // ... setup network socket ...
    if (connect(sock, (struct sockaddr *)&serv_addr, sizeof(serv_addr)) < 0) {
        // ... error handling ...
        return -1;
    }
    return sock; // Returns a socket connection
}

This helper function is like dialing a phone number to connect to either the Metadata Server or a Storage Node. It establishes a network connection.

Now, inside cdfs_put, we first talk to the Metadata Server to allocate chunk IDs and get active Storage Nodes:

// File: client/dfs_client.c (simplified cdfs_put)
int32_t cdfs_put(const uint8_t *local_path, const uint8_t *cdfs_path) {
    // ... (file opening, chunk calculation) ...

    // 1. Connect to Metadata Server and ask for chunk IDs
    int32_t meta_sock = connect_to_server((const uint8_t *)config.meta_ip, config.meta_port);
    // ... (error handling) ...
    req_allocate_chunks_t alloc_req = { needed_chunks };
    send_exact(meta_sock, &alloc_hdr, sizeof(alloc_hdr)); // Send header
    send_exact(meta_sock, &alloc_req, sizeof(alloc_req)); // Send request data
    // ... (receive response, close socket) ...
    int32_t start_chunk_id = alloc_resp.start_chunk_id;

    // 2. Connect to Metadata Server again to get active Storage Nodes
    meta_sock = connect_to_server((const uint8_t *)config.meta_ip, config.meta_port);
    // ... (error handling) ...
    send_exact(meta_sock, &hdr_nodes, sizeof(hdr_nodes));
    send_exact(meta_sock, &req_nodes, sizeof(req_nodes));
    // ... (receive active_nodes list, close socket) ...

    // ... (more code for chunking and sending to Storage Nodes) ...
    return 0;
}

Notice how send_exact and recv_exact are used. These are helper functions (likely from common/serialization.h or common/protocol.h) that handle sending and receiving the exact number of bytes over the network, ensuring complete messages are sent and received.

The core of cdfs_put involves reading the file, chunk by chunk, and sending these chunks to multiple Storage Nodes:

// File: client/dfs_client.c (simplified cdfs_put - chunking and storing)
int32_t cdfs_put(const uint8_t *local_path, const uint8_t *cdfs_path) {
    // ... (previous steps: connect, get chunk IDs, get active nodes) ...

    while (1) { // Loop through each chunk of the file
        long chunk_start = ftell(fp); // Remember where this chunk starts in the file
        uint8_t mbuf[8192];
        size_t bytes = 0; // Bytes in current chunk
        // ... (read bytes from local file into mbuf, calculate checksum) ...
        if (bytes == 0) break; // End of file

        int32_t chunk_id = start_chunk_id + chunk_count;
        // ... (fill chunk_info_t structure) ...

        // Push to replicas: Send this chunk to several Storage Nodes
        for (int32_t i = 0; i < active_nodes.node_count; i++) {
            int32_t store_sock = connect_to_server(active_nodes.node_ips[i], active_nodes.node_ports[i]);
            if (store_sock < 0) continue; // Try next node if connection fails

            req_store_chunk_t req = { chunk_id, bytes, checksum };
            send_exact(store_sock, &hdr, sizeof(hdr)); // Send request header
            send_exact(store_sock, &req, sizeof(req)); // Send chunk metadata
            // ... (rewind file pointer and send the actual chunk data from mbuf) ...
            // ... (receive confirmation from Storage Node) ...
            close(store_sock);
        }
        chunk_count++;
        // ... (seek to next chunk in file) ...
    }
    // ... (finally, register file with Metadata Server) ...
    return 0;
}

This loop demonstrates how the client iterates through the local file, reads chunks, and then sends each chunk to multiple Storage Nodes for replication. This ensures data safety and availability, even if one Storage Node fails.

Finally, after all chunks are sent, the file's information is registered with the Metadata Server:

// File: client/dfs_client.c (simplified cdfs_put - register file)
int32_t cdfs_put(const uint8_t *local_path, const uint8_t *cdfs_path) {
    // ... (previous steps: connect, get chunk IDs, get active nodes, send chunks) ...

    // Register file with Metadata Server
    int32_t meta_sock = connect_to_server((const uint8_t *)config.meta_ip, config.meta_port);
    // ... (error handling) ...

    req_register_file_t reg_req;
    strncpy((char *)reg_req.filename, (const char *)cdfs_path, MAX_FILENAME - 1);
    reg_req.chunk_count = chunk_count;

    net_header_t meta_hdr = { OP_REGISTER_FILE, sizeof(reg_req) + chunk_count * sizeof(chunk_info_t) };
    send_exact(meta_sock, &meta_hdr, sizeof(meta_hdr));        // Send header
    send_exact(meta_sock, &reg_req, sizeof(reg_req));          // Send file details
    send_exact(meta_sock, chunks, chunk_count * sizeof(chunk_info_t)); // Send chunk locations
    // ... (receive status, close socket) ...
    return status;
}

This final step is crucial. The client tells the Metadata Server not just the filename, but also all the chunk_info_t details, including each chunk's ID, size, and which Storage Nodes are holding its copies. This information is the "metadata" about your file.

The other client commands (cdfs_get, cdfs_ls, cdfs_rm, cdfs_status) follow a similar pattern: they connect to the Metadata Server, send a specific request, and then receive a response. For cdfs_get, after getting the file's chunk information from the Metadata Server, the client then connects directly to Storage Nodes to download each chunk, just like cdfs_put uploads them.

Conclusion

The Client Application Interface is your user-friendly window into the cdfs-ton distributed file system. It provides simple commands like put, get, ls, rm, and status, allowing you to manage your files without worrying about the complex network communication, data chunking, or where your files are actually stored. Behind these simple commands, the client application intelligently interacts with the Metadata Server and Storage Nodes to fulfill your requests, acting as your reliable remote control.

Next, we'll dive deeper into the "control tower" itself: the Metadata Management system, to understand how it keeps track of all your files and their locations.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

01_client_application_interface.md

Chapter 1: Client Application Interface

Your Remote Control Buttons

How to Use Your Remote Control

Uploading a file (`put`)

Downloading a file (`get`)

Listing files (`ls`)

Deleting a file (`rm`)

Checking system status (`status`)

What Happens Under the Hood? (Behind the Remote Control)

Diving into the `cdfs_put` Code

Conclusion

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally

01_client_application_interface.md

Chapter 1: Client Application Interface

Your Remote Control Buttons

How to Use Your Remote Control

Uploading a file (put)

Downloading a file (get)

Listing files (ls)

Deleting a file (rm)

Checking system status (status)

What Happens Under the Hood? (Behind the Remote Control)

Diving into the cdfs_put Code

Conclusion

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally

Uploading a file (`put`)

Downloading a file (`get`)

Listing files (`ls`)

Deleting a file (`rm`)

Checking system status (`status`)

Diving into the `cdfs_put` Code