File Modification And Merging

Introduction

Managing files often requires modifying and merging content dynamically. Whether you're updating logs, editing configuration files, or combining multiple directories, the file-worker library provides utilities to modify and merge files and directories efficiently.

Imagine you need to clean up log files by deleting specific lines, merge all Markdown files in a documentation folder, or chain file operations fluently. This library helps automate such operations with minimal, readable code.

Features Overview

• Modify file content line by line.
• Delete specific lines from a file by index.
• Fluent API support: Methods returning File or Path objects for easy method chaining.
• Extension-based concatenation: Merge all files with a specific extension in a directory (optionally recursive).
• Merge directories while handling conflicts intelligently.
• Perform bulk modifications on files.

Getting Started

For installation and dependencies, refer to the Getting Started guide.

---

Modifying and Concatenating Files with ModifyFileExtensions

The ModifyFileExtensions class provides methods for updating file content efficiently and merging multiple files.

Example: Modifying a File Line by Line (Fluent API)

The library now supports a Fluent API, allowing you to chain operations by returning the modified Path or File.

import java.nio.charset.StandardCharsets;
import java.nio.file.Path;
import io.github.astrapi69.file.modify.ModifyFileExtensions;

Path filePath = Path.of("sample.txt");

// modifyFileAndGet returns the Path, enabling further operations
Path modifiedPath = ModifyFileExtensions.modifyFileAndGet(filePath, StandardCharsets.UTF_8, (index, line) -> line.toUpperCase());

System.out.println("File content converted to uppercase at: " + modifiedPath);

Example: Concatenating Files by Extension

A common task is merging all files of a specific type within a directory (e.g., combining all .md or .log files).

import java.io.File;
import io.github.astrapi69.file.modify.ModifyFileExtensions;

File sourceDirectory = new File("src/main/resources/docs");
File resultFile = new File("target/combined-docs.md");

// Concatenates all .md files in the directory alphabetically into the result file
ModifyFileExtensions.concatenateFilesWithExtension(sourceDirectory, ".md", resultFile);

System.out.println("All markdown files merged successfully into: " + resultFile.getName());

Note: There is also a concatenateFilesWithExtensionRecursive method available if you need to search through subdirectories.

Example: Concatenating a Specific List of Files

If you already have a specific list of files, you can concatenate them directly:

import java.io.File;
import java.util.List;
import io.github.astrapi69.file.modify.ModifyFileExtensions;

List<File> filesToConcatenate = List.of(new File("file1.txt"), new File("file2.txt"));
File resultFile = new File("merged.txt");

ModifyFileExtensions.concatenateAll(filesToConcatenate, resultFile);

Unit Test Coverage

The ModifyFileExtensionsTest class verifies this functionality, including the new extension-based concatenation:

@Test
public void testConcatenateFilesWithExtension() throws IOException {
    // Arrange
    File testDir = new File("test-dir");
    testDir.mkdirs();
    LineAppender.appendLines(new File(testDir, "a.txt"), new String[]{"Line 1"});
    LineAppender.appendLines(new File(testDir, "b.txt"), new String[]{"Line 2"});
    File resultFile = new File("result.txt");

    // Act
    File returnedFile = ModifyFileExtensions.concatenateFilesWithExtension(testDir, ".txt", resultFile);

    // Assert
    assertEquals(resultFile, returnedFile);
    String actual = ReadFileExtensions.fromFile(resultFile);
    assertEquals("Line 1\nLine 2\n", actual); // Files are processed in alphabetical order
    
    // Cleanup
    DeleteFileExtensions.delete(testDir);
    DeleteFileExtensions.deleteFile(resultFile);
}

---

Deleting Specific Lines with DeleteLinesByIndexInFile

The DeleteLinesByIndexInFile class allows you to remove specific lines from a file by leveraging the modification engine.

Example: Deleting Lines by Index

import java.nio.file.Path;
import java.util.List;
import io.github.astrapi69.file.modify.ModifyFileExtensions;
import io.github.astrapi69.file.modify.api.FileChangeable;
import io.github.astrapi69.file.modify.DeleteLinesByIndexInFile;

List<Integer> linesToDelete = List.of(1, 3); // 0-based or 1-based depending on implementation specifics
FileChangeable deleter = new DeleteLinesByIndexInFile(linesToDelete);

ModifyFileExtensions.modifyFile(Path.of("data.csv"), deleter);

System.out.println("Selected lines deleted from the file.");

---

Merging Directories with MergeDirectoryExtensions

The MergeDirectoryExtensions class helps combine multiple directories into a single target directory. It offers both a simple timestamp-based merge and advanced, strategy-driven merging.

Example 1: Simple Merge (Newest File Wins)

By default, the library resolves conflicts by keeping the most recently modified file.

import java.io.File;
import io.github.astrapi69.file.merge.MergeDirectoryExtensions;

File targetDir = new File("target/mergedDir");
File dir1 = new File("source/backup_2023");
File dir2 = new File("source/backup_2024");

// Uses the default "newest file wins" logic
File resultDir = MergeDirectoryExtensions.mergeAndGet(targetDir, dir1, dir2);

Example 2: Strategy-Driven Merge (Fluent API)

For more control, you can specify a MergeStrategy. This is particularly useful for synchronizing folders or cleaning up source directories after a successful merge.

import java.io.File;
import io.github.astrapi69.file.merge.MergeDirectoryExtensions;
import io.github.astrapi69.file.merge.strategy.MergeStrategy;

File targetDir = new File("target/master");
File sourceDir = new File("source/updates");

// TARGET_AS_MASTER: Copies new/changed files to target, then DELETES them from the source
File resultDir = MergeDirectoryExtensions.mergeAndGet(targetDir, MergeStrategy.TARGET_AS_MASTER, sourceDir);

System.out.println("Synced and cleaned source directory. Target is at: " + resultDir.getAbsolutePath());

Understanding the Strategies

• TARGET_AS_MASTER:

  • If a file is only in the source: Copy to target, then delete from source.
  • If a file is only in the target: Do nothing.
  • If a file exists in both with different content: Overwrite target with source, then delete from source.
  • If content is equal: Leave target unchanged, delete from source. (Ideal for "Sync and Cleanup" operations)

• SOURCE_TO_TARGET:

  • If a file is only in the source: Copy to target.
  • If a file is only in the target: Do nothing.
  • If a file exists in both with different content: Overwrite target with source.
  • If content is equal: Leave target unchanged. (Ideal for "Backup or Update" operations without modifying the source)

---

Merging File Content with MergeFileExtensions

While ModifyFileExtensions handles simple concatenation (appending files), the MergeFileExtensions class provides intelligent merge operations that go beyond simple concatenation. It supports deduplication, sorting, and various specialized merge strategies.

Merge Strategies Overview

The MergeFileStrategy enum defines exactly how files are combined. It offers both basic and advanced strategies:

• APPEND: Simple concatenation, no processing or deduplication.
• UNIQUE_LINES: Removes duplicate lines while preserving the order of their first occurrence.
• SORTED: Sorts all lines alphabetically (keeps duplicates).
• SORTED_UNIQUE: Removes duplicates AND sorts the result alphabetically.
• CSV_HEADER_MERGE: Merges multiple CSV files while ensuring the header row (first line) is only written once, from the very first file.
• MARKDOWN_SECTIONS: Combines Markdown documents and automatically inserts a horizontal rule (---) with proper spacing between them for a clean, unified document.
• BY_KEY: Merges files based on a unique key in each line. If a key appears multiple times, the last encountered line overwrites previous ones (ideal for data updates). Defaults to comma delimiter and first column.

Example: Merging Two Files with Deduplication

import java.io.File;
import io.github.astrapi69.file.merge.MergeFileExtensions;
import io.github.astrapi69.file.merge.strategy.MergeFileStrategy;

File file1 = new File("users_2023.txt");
File file2 = new File("users_2024.txt");
File result = new File("all_users_unique.txt");

// Merges and removes duplicate lines
MergeFileExtensions.mergeAndGet(file1, file2, result, MergeFileStrategy.UNIQUE_LINES);

System.out.println("Unique users merged into: " + result.getName());

Advanced Usage Examples

1. CSV Header Merge

Prevents duplicate headers when combining multiple CSV exports.

MergeFileExtensions.mergeAndGet(file1, file2, resultFile, MergeFileStrategy.CSV_HEADER_MERGE);

2. Markdown Sections

Creates a single, well-formatted documentation file from multiple chapters.

MergeFileExtensions.mergeAndGet(doc1, doc2, resultFile, MergeFileStrategy.MARKDOWN_SECTIONS);

3. Merge By Key (Relational Join)

Ideal for updating datasets where a specific column acts as a primary key.

Default Behavior: Assumes a comma (,) delimiter and uses the first column (index 0) as the key.

MergeFileExtensions.mergeAndGet(files, resultFile, MergeFileStrategy.BY_KEY);

Advanced Customization: For full control over the key column and delimiter, use the dedicated mergeByKey method:

import java.util.List;
import java.io.File;
import io.github.astrapi69.file.merge.MergeFileExtensions;

// Merges based on the 2nd column (index 1), using a semicolon (;) as delimiter
// Last encountered value for a key will overwrite previous ones
MergeFileExtensions.mergeByKey(files, resultFile, 1, ";");

---

Conclusion

The file-worker library provides robust, fluent, and highly customizable tools for modifying files, deleting specific content, concatenating files by extension, and merging directories efficiently. Whether you're processing logs, editing data, or managing large file structures, this library simplifies file handling while maintaining academic and professional code quality.

Start using it today to automate your file operations and enhance productivity!

---