Skip to content

07_persisted_metadata_fsimage_edit_log.md

Jump to bottom
Lavnish edited this page Mar 31, 2026 · 1 revision

Chapter 7: Persisted Metadata (FSImage & Edit Log)

Welcome back, cdfs-ton adventurer! In our last chapter, System Configuration, we learned how to customize cdfs-ton to fit our needs. Now, let's tackle a really critical question: What if the brain of cdfs-ton, the Metadata Server, suddenly stopped working or had to restart? Would all our precious file information be lost forever?

Imagine the Metadata Server as a super-smart librarian who knows exactly where every single chapter (chunk) of every book (file) is located. If this librarian keeps all this knowledge only in their head, and then suddenly gets a bad case of amnesia (a server crash!), all that information is gone. Our cdfs-ton system would no longer know where your files are, making them effectively lost, even if the actual data chunks are still sitting on the Storage Nodes.

This is a huge problem for any reliable storage system! We need a way to ensure that the Metadata Server's memory – all its knowledge about files and chunks – is never lost, even if the server itself crashes or restarts.

The Solution: Saving the Brain's Memory to Disk!

To prevent data loss and ensure that the Metadata Server can always recover its full state, cdfs-ton uses a clever strategy: it continuously saves its entire "brain" to disk. It does this by using two main components:

  1. FSImage (File System Image): This is like taking a complete photograph or snapshot of the librarian's entire catalog at a specific moment in time. It's a full backup of all file and chunk information.
  2. Edit Log: This is like the librarian's daily diary. After taking the big photograph (FSImage), the librarian writes down every single change that happens afterward – a new book is added, an old book is removed, a new chunk ID is allocated.

When the Metadata Server needs to restart, it doesn't just start from scratch. Instead, it follows a precise recovery process:

  1. Load the FSImage: First, it loads the most recent "photograph" (FSImage) of its entire catalog. This brings it back to a known good state.
  2. Replay the Edit Log: Then, it reads through the "daily diary" (Edit Log) and applies every single change recorded since that photograph was taken. It's like replaying all the actions step-by-step.
  3. Fully Up-to-Date: After replaying all the changes, the server's memory is perfectly restored to the exact state it was in just before it crashed. Your files are safe!

This combination of FSImage and Edit Log ensures that the Metadata Server is highly durable, meaning its data (your file metadata!) is never lost.

Key Concepts

Let's look at these two crucial components more closely.

1. FSImage (fsimage.dat)

  • What it is: A complete snapshot of the Metadata Server's entire namespace (all registered files, their chunks, and the next_chunk_id value). It contains all the file_metadata_t structures we discussed in Chapter 2: Metadata Management.
  • When it's created: The FSImage is saved periodically, especially during a process called "compaction" (which we'll explain shortly).
  • File location: It's typically saved as a file named fsimage.dat in the Metadata Server's working directory.

2. Edit Log (edits.log)

  • What it is: A continuous, chronological record of every metadata-changing operation that occurs after the most recent FSImage was created.
  • When an entry is added:
    • When a file is uploaded (register_file).
    • When a file is deleted (delete_file).
    • When new chunk IDs are allocated (allocate_chunks).
  • File location: It's saved as a file named edits.log in the Metadata Server's working directory.

Why Both? The Power of Checkpointing (Compaction)

Why not just save a new FSImage every time a change happens? Because taking a full snapshot of the entire catalog can be slow! If you did it too often, the Metadata Server would spend all its time saving data instead of managing files.

The Edit Log is very fast because it only records the changes. However, if the Edit Log grows too big, replaying it during a restart can take a long time.

This is where Checkpointing (or Compaction) comes in:

  • Periodically, cdfs-ton performs a "compaction." During this process, it creates a brand-new FSImage that incorporates all the changes from the Edit Log.
  • Once the new FSImage is safely saved, the old Edit Log (which is now fully covered by the new FSImage) is truncated (cleared).
  • This keeps the Edit Log small and manageable, ensuring that restarts are always fast.

How Metadata Server Recovers from a Restart

Let's visualize the Metadata Server's recovery process:

sequenceDiagram
    participant MS as "Metadata Server"
    participant FSImageFile as "fsimage.dat"
    participant EditLogFile as "edits.log"

    MS->>MS: 1. Server starts
    MS->>FSImageFile: 2. Read latest FSImage file
    FSImageFile-->>MS: 3. Load full metadata snapshot (files, next_chunk_id)
    Note over MS: Server's memory is restored to the last full backup point.

    MS->>EditLogFile: 4. Read Edit Log file
    loop For each operation in Edit Log
        EditLogFile-->>MS: 5. Read operation type and data
        MS->>MS: 6. Apply operation (e.g., add new file, delete file, update next_chunk_id)
    end
    EditLogFile-->>MS: 7. All changes applied

    MS->>MS: 8. Server is now fully up-to-date and ready to serve requests.
Loading

Diving Deeper: Inside the Metadata Server's Persistence Code (metadata_server/metadata.c)

The persistence logic lives primarily within the metadata_server/metadata.c file. This file contains the functions that manage our files array, file_count, and next_chunk_id (our librarian's memory!), and ensures they are safely written to disk.

1. Recording Changes: The Edit Log

Every time an important change happens to the metadata, an entry is added to the edits.log file.

A. Appending File Changes (append_edit_log)

When a file is registered or deleted, the metadata.c module calls append_edit_log.

// File: metadata_server/metadata.c
void append_edit_log(file_metadata_t *file, uint8_t op) {
    FILE *fp = fopen("edits.log", "ab"); // "ab" for append binary
    if (fp) {
        fwrite(&op, sizeof(op), 1, fp); // Write operation code (1 for PUT, 2 for DELETE)
        fwrite(file, sizeof(file_metadata_t), 1, fp); // Write the entire file's metadata
        fclose(fp);
    }
}
  • fopen("edits.log", "ab"): Opens the edits.log file in "append binary" mode. This means new data will be added to the end of the file.
  • fwrite(&op, ...): Writes a small op code (like 1 for a file PUT or 2 for a file DELETE) to indicate what kind of operation occurred.
  • fwrite(file, ...): Writes the entire file_metadata_t structure. This contains the filename, chunk count, and chunk details.

Let's see how register_file (from Chapter 2: Metadata Management) uses this:

// File: metadata_server/metadata.c (simplified register_file)
int32_t register_file(const uint8_t *filename, const chunk_info_t* chunks, int32_t chunk_count) {
    pthread_mutex_lock(&namespace_mutex);
    // ... (logic to create a new file_metadata_t entry in 'files' array) ...
    
    append_edit_log(file, 1); // op=1 means PUT: a new file was added
    
    pthread_mutex_unlock(&namespace_mutex);
    return 0;
}

After safely adding a new file entry to its in-memory files array, the Metadata Server immediately records this PUT operation in the edits.log. The delete_file function works similarly, but uses op=2.

B. Appending Chunk ID Allocations (append_allocate_log)

When new chunk IDs are handed out, that's also a crucial change that needs to be recorded.

// File: metadata_server/metadata.c
void append_allocate_log(int32_t new_next_chunk_id) {
    FILE *fp = fopen("edits.log", "ab");
    if (fp) {
        uint8_t op = 3; // op=3 means ALLOCATE_CHUNKS
        fwrite(&op, sizeof(op), 1, fp); // Write operation code
        fwrite(&new_next_chunk_id, sizeof(new_next_chunk_id), 1, fp); // Write the new next_chunk_id
        fclose(fp);
    }
}

And how allocate_chunks uses it:

// File: metadata_server/metadata.c (simplified allocate_chunks)
int32_t allocate_chunks(int32_t count, int32_t *out_start_chunk_id) {
    pthread_mutex_lock(&namespace_mutex);
    *out_start_chunk_id = next_chunk_id;
    next_chunk_id += count;
    append_allocate_log(next_chunk_id); // Record the new next_chunk_id
    pthread_mutex_unlock(&namespace_mutex);
    return 0;
}

2. Saving the Full Snapshot: The FSImage (save_fsimage)

This function writes the current in-memory state of the entire file system (all the metadata) into fsimage.dat.

// File: metadata_server/metadata.c
void save_fsimage(void) {
    pthread_mutex_lock(&namespace_mutex); // Lock to ensure safe access
    FILE *fp = fopen("fsimage.dat", "wb"); // "wb" for write binary
    if (!fp) {
        perror("Failed to open fsimage.dat for writing");
        pthread_mutex_unlock(&namespace_mutex);
        return;
    }
    fwrite(&file_count, sizeof(file_count), 1, fp); // Save total file count
    fwrite(&next_chunk_id, sizeof(next_chunk_id), 1, fp); // Save next available chunk ID
    fwrite(files, sizeof(file_metadata_t), file_count, fp); // Save all file metadata
    fclose(fp);
    pthread_mutex_unlock(&namespace_mutex);
}
  • fwrite(&file_count, ...): Saves the current number of files.
  • fwrite(&next_chunk_id, ...): Saves the next_chunk_id counter.
  • fwrite(files, ...): This is the critical part; it writes the entire files array (which holds all file_metadata_t structures) to the disk.

3. Recovering State: Loading FSImage and Replaying Edit Log (load_fsimage)

This is the recovery function called when the Metadata Server starts up. It performs the two-step process: loading the fsimage.dat and then replaying edits.log.

// File: metadata_server/metadata.c (simplified load_fsimage)
void load_fsimage(void) {
    pthread_mutex_lock(&namespace_mutex);

    // --- Step 1: Load FSImage ---
    FILE *fp = fopen("fsimage.dat", "rb"); // "rb" for read binary
    if (!fp) { LOG_INFO("MD", "No fsimage.dat found, starting fresh.\n"); goto replay_edits; }

    fread(&file_count, sizeof(file_count), 1, fp); // Read file count
    fread(&next_chunk_id, sizeof(next_chunk_id), 1, fp); // Read next chunk ID
    fread(files, sizeof(file_metadata_t), file_count, fp); // Read all file metadata
    fclose(fp);
    LOG_INFO("MD", "Loaded fsimage.dat: %d files, next chunk ID: %d\n", file_count, next_chunk_id);
    
replay_edits:
    // --- Step 2: Replay Edit Log ---
    FILE *edits = fopen("edits.log", "rb");
    if (!edits) { LOG_INFO("MD", "No edits.log found or empty.\n"); goto end; }

    uint8_t op;
    file_metadata_t file_from_log; // Temporary struct to read log entries
    while (fread(&op, sizeof(op), 1, edits) == 1) { // Read op code
        if (op == 3) { // ALLOCATE_CHUNKS
            int32_t saved_next_id;
            if (fread(&saved_next_id, sizeof(saved_next_id), 1, edits) == 1) {
                next_chunk_id = saved_next_id; // Update next_chunk_id
            }
        } else if (fread(&file_from_log, sizeof(file_metadata_t), 1, edits) == 1) {
            if (op == 1) { // PUT (register new file)
                // Check if file already exists (could be in FSImage or earlier edit)
                int32_t found = 0;
                for (int32_t i = 0; i < file_count; i++) {
                    if (strcmp((char *)files[i].filename, (char *)file_from_log.filename) == 0) {
                        found = 1; break;
                    }
                }
                if (!found && file_count < MAX_FILES) {
                    files[file_count++] = file_from_log; // Add new file
                }
            } else if (op == 2) { // DELETE (remove file)
                for (int32_t i = 0; i < file_count; i++) {
                    if (strcmp((char *)files[i].filename, (char *)file_from_log.filename) == 0) {
                        // Shift entries to remove it
                        for (int32_t j = i; j < file_count - 1; j++) files[j] = files[j+1];
                        file_count--;
                        break;
                    }
                }
            }
        }
    }
    fclose(edits);
    LOG_INFO("MD", "Replayed edit log. New file_count: %d, next chunk ID: %d\n", file_count, next_chunk_id);
    
end:
    pthread_mutex_unlock(&namespace_mutex);
}
  • Loading FSImage: The code first attempts to open fsimage.dat. If found, it reads file_count, next_chunk_id, and the entire files array into memory.
  • Replaying Edit Log: Then, it opens edits.log. It reads each entry, starting with the op code.
    • If op == 3 (ALLOCATE): It updates the next_chunk_id with the value from the log.
    • If op == 1 (PUT): It adds the file_metadata_t from the log to the in-memory files array (making sure it doesn't add duplicates or exceed MAX_FILES).
    • If op == 2 (DELETE): It removes the file_metadata_t from the in-memory files array.
  • pthread_mutex_lock: Notice the mutex locks around the entire load_fsimage function. This is critical to ensure that no other threads try to read or modify the files array while it's being restored, preventing data corruption.

4. Compacting the Edit Log (compact_edit_log)

This function is called periodically (e.g., from the replication_monitor thread, as seen in Chapter 5: Fault Tolerance & Replication Monitoring) to merge the edits.log into a new fsimage.dat.

// File: metadata_server/metadata.c
void compact_edit_log(void) {
    LOG_INFO("MD", "Starting metadata compaction...\n");
    save_fsimage(); // 1. Save fsimage (captures current state, including all edits)
    
    pthread_mutex_lock(&namespace_mutex);
    FILE *fp = fopen("edits.log", "wb"); // "wb" to overwrite/truncate the file
    if (fp) {
        fclose(fp); // 2. Close, which truncates the file to 0 bytes
        LOG_INFO("MD", "Compacted edits.log successfully.\n");
    } else {
        LOG_ERR("MD", "Failed to truncate edits.log during compaction.\n");
    }
    pthread_mutex_unlock(&namespace_mutex);
}
  • save_fsimage(): This crucial step writes the current, up-to-date state of the Metadata Server's memory (which already includes all changes from the edits.log up to this point) into a new fsimage.dat file.
  • fopen("edits.log", "wb") and fclose(fp): By opening the edits.log file in "write binary" mode ("wb") and immediately closing it, the file is effectively truncated (emptied) to zero bytes. This is safe because all its contents are now captured in the new fsimage.dat.

Conclusion

The combination of FSImage and Edit Log is a cornerstone of cdfs-ton's reliability. It ensures that the vital metadata, the "brain" of our distributed file system, is never lost, even in the event of a server crash or restart. By taking periodic full snapshots (FSImage) and meticulously recording every change (Edit Log), cdfs-ton can always restore its complete state, providing a robust foundation for your distributed data.

Next, we'll shift our focus to understanding how cdfs-ton keeps track of everything that happens within the system, from errors to routine operations, by exploring System-wide Logging.

Clone this wiki locally