Skip to content

Architecture Overview

Jump to bottom
Mikhail Deynekin edited this page Dec 23, 2025 · 1 revision

Architecture Overview

🏗️ System Architecture & Design

This page explains the internal architecture and design of Clean BOM Senior. Understanding how the script works will help you use it more effectively and contribute improvements.

📊 Application Flow Diagram

The following Mermaid diagram shows the complete execution flow:

flowchart TD
    Start([User Executes Script]) --> ParseArgs["Parse Command-Line Arguments
(--dry-run, --backup, --verbose, etc)"]
    
    ParseArgs --> ValidatePath["Validate Input Path
Check Permissions"]
    
    ValidatePath --> PathValid{Path Valid?}
    PathValid -->|No| ErrorPath["Error: Invalid Path
Exit Code 3"]
    PathValid -->|Yes| InitStats["Initialize Statistics
Counters & Logging"]
    
    InitStats --> ScanDir["Recursive Directory Scan
(with --exclude patterns)"]
    
    ScanDir --> ForEach["For Each File in Directory"]
    
    ForEach --> GetFile{Files Left?}
    GetFile -->|No| GenReport["Generate Report
Log Statistics"]
    GetFile -->|Yes| DetectType["Step 1: Detect File Type
by Extension/MIME/Content"]
    
    DetectType --> TypeOK{Type
Supported?}
    TypeOK -->|Binary/Skip| SkipFile["Skip File
Update Counters"]
    TypeOK -->|Yes| ReadFile["Step 2: Read File
into Memory"]
    
    SkipFile --> ForEach
    
    ReadFile --> CheckBOM["Step 3: Check for BOM
Detect UTF-8/UTF-16/UTF-32"]
    
    CheckBOM --> HasBOM{BOM
Detected?}
    
    HasBOM -->|No BOM| CheckCRLF["Step 4: Check Line Endings
CRLF vs LF"]
    HasBOM -->|BOM Found| RemoveBOM["Step 5: Remove BOM
Strip first 2-4 bytes"]
    
    RemoveBOM --> CheckCRLF
    
    CheckCRLF --> HasCRLF{CRLF
Found?}
    
    HasCRLF -->|No CRLF| NoChanges["No Changes Needed
Update Counters"]
    HasCRLF -->|CRLF Present| FixCRLF["Step 6: Convert Line Endings
CRLF to LF"]
    
    NoChanges --> ForEach
    
    FixCRLF --> DryRun{Dry-Run
Mode?}
    
    DryRun -->|Yes| LogChange["Log What Would Change
Do NOT Modify"]
    DryRun -->|No| Backup{Backup
Enabled?}
    
    LogChange --> UpdateStats["Update Statistics
(files_to_clean++)"]
    
    Backup -->|Yes| CreateBak["Create .bak Backup
Copy Original File"]
    Backup -->|No| WriteFile["Write Fixed Content
Back to File"]
    
    CreateBak --> WriteFile
    
    WriteFile --> UpdateStats2["Update Statistics
(files_cleaned++)"]
    UpdateStats2 --> ForEach
    
    UpdateStats --> ForEach
    
    GenReport --> UpdateGit{Fix
Gitattributes?}
    
    UpdateGit -->|Yes| AddGit["Add/Update .gitattributes
with eol=lf settings"]
    UpdateGit -->|No| PrintStats
    
    AddGit --> PrintStats["Print Summary Report
Cleaned/Skipped/Errors"]
    
    PrintStats --> CalcExit{Success?}
    
    CalcExit -->|All OK| Exit0["Return Exit Code 0
(Success)"]
    CalcExit -->|Some Errors| Exit2["Return Exit Code 2
(Errors)"]
    
    Exit0 --> End([End])
    Exit2 --> End
    ErrorPath --> End
Loading

🔧 Core Components

1. Argument Parser

  • Parses command-line flags: --dry-run, --backup, --verbose, --file-type, --exclude, etc.
  • Sets global configuration variables
  • Validates argument combinations
  • Key File: Main script header section

2. Path Validator

  • Validates that input path exists
  • Checks read/write permissions
  • Handles both files and directories
  • Returns: Exit code 3 if path invalid

3. Directory Scanner

  • Recursively scans directories using find command
  • Applies --exclude patterns for filtering
  • Respects . and .. special directories
  • Performance: Uses glob patterns for efficiency

4. File Type Detector

  • Three-level detection strategy:
    1. Extension (fastest) — checks file suffix
    2. MIME type — uses file command
    3. Content inspection — reads first bytes
  • Supports 50+ file types
  • Binary Detection: Automatically skips binary files

5. BOM Detector

  • Reads first 4 bytes of file
  • Detects UTF-8 BOM: EF BB BF (3 bytes)
  • Detects UTF-16 BE: FE FF (2 bytes)
  • Detects UTF-16 LE: FF FE (2 bytes)
  • Detects UTF-32: 00 00 FE FF or FF FE 00 00 (4 bytes)
  • Method: Uses od, hexdump, or xxd for detection

6. BOM Remover

  • Removes detected BOM bytes from file start
  • Preserves rest of file content
  • Handles all BOM types
  • Safe: Only removes if detected

7. CRLF Converter

  • Detects Windows (CRLF: 0D 0A) line endings
  • Converts to Unix (LF: 0A) format
  • Uses sed or tr for conversion
  • Option: --fix-crlf flag enables this

8. File Writer

  • Writes modified content back to file
  • Dry-Run Mode: Simulates without writing
  • Backup Mode: Creates .bak copies first
  • Preserves file permissions
  • Error Handling: Reports write failures

9. Statistics Tracker

  • Counts files processed
  • Counts BOM removals
  • Counts CRLF fixes
  • Counts errors/skipped files
  • Display: Shows summary at end

10. Report Generator

  • Generates human-readable summary
  • Shows counts by category
  • Lists problematic files in verbose mode
  • Provides exit codes for scripting

📈 Data Flow

┌─────────────────────────────────────────────────────────┐
│                    INPUT LAYER                          │
│  User Input → Command-Line Arguments → Config Variables │
└──────────────────────┬──────────────────────────────────┘
                       │
┌──────────────────────▼──────────────────────────────────┐
│                  PROCESSING LAYER                       │
│  Directory Scan → Type Detection → BOM/CRLF Detection   │
│                                                          │
│  For Each File:                                         │
│  ├─ Read File into Memory                              │
│  ├─ Check for BOM (UTF-8/16/32)                        │
│  ├─ Check for CRLF Line Endings                        │
│  ├─ Apply Fixes if Enabled                             │
│  └─ Write Back (or skip in dry-run)                    │
└──────────────────────┬──────────────────────────────────┘
                       │
┌──────────────────────▼──────────────────────────────────┐
│                   OUTPUT LAYER                          │
│  Statistics → Report → Exit Code → User Display        │
└─────────────────────────────────────────────────────────┘

🔄 Key Algorithms

BOM Detection Algorithm

IF first 4 bytes == 0x00 0x00 0xFE 0xFF THEN
    BOM_TYPE = UTF-32-BE
ELSE IF first 4 bytes == 0xFF 0xFE 0x00 0x00 THEN
    BOM_TYPE = UTF-32-LE
ELSE IF first 2 bytes == 0xFE 0xFF THEN
    BOM_TYPE = UTF-16-BE
ELSE IF first 2 bytes == 0xFF 0xFE THEN
    BOM_TYPE = UTF-16-LE
ELSE IF first 3 bytes == 0xEF 0xBB 0xBF THEN
    BOM_TYPE = UTF-8
ELSE
    BOM_TYPE = NONE
END IF

File Type Detection Algorithm

Extension = Get file suffix (e.g., .php, .json)

IF Extension IN SUPPORTED_TYPES THEN
    TYPE = extension-based
ELSE IF Use `file -i` for MIME type THEN
    IF MIME type matches known type THEN
        TYPE = mime-based
    END IF
ELSE IF Read first bytes match code patterns THEN
    TYPE = content-based
ELSE
    TYPE = unknown (skip file)
END IF

⚙️ Performance Characteristics

Metric Value Notes
Processing Speed 1000+ files/sec Depends on disk I/O
Memory Usage <10 MB Minimal overhead
File Size Limit Unlimited Processes line by line
Recursion Depth Unlimited Uses OS find limits
Parallel Support No Single-threaded

🛡️ Error Handling

Exit Codes

Code Meaning Action
0 Success Changes made successfully
1 No Changes No files needed cleaning
2 Errors Some files had issues
3 Invalid Path Path not found
4 Permission Denied Can't read/write files

Error Scenarios

  1. Path Not Found → Exit code 3, error message
  2. Permission Denied → Exit code 4, show which file
  3. Partial Success → Exit code 2, log failures
  4. File Write Failed → Skip file, continue processing
  5. Invalid Arguments → Show help, exit code 2

🔐 Safety Features

  1. Dry-Run Mode (--dry-run)

    • Previews changes without modifying files
    • Safe to test before actual run

  2. Backup Creation (--backup)

    • Creates .bak copies before changes
    • Easy rollback if needed

  3. Permission Preservation

    • Maintains original file permissions
    • Respects umask settings

  4. Read-Only Safety

    • Detects read-only files
    • Skips them with warning

  5. Atomic Operations

    • Writes to temp file first
    • Moves to original on success

🎯 Design Principles

  1. KISS (Keep It Simple, Stupid)

    • Pure Bash script, no dependencies
    • Straightforward logic

  2. Fail Safe

    • Never corrupts data
    • Always has backup option

  3. Efficiency

    • Minimal memory footprint
    • Fast processing

  4. Compatibility

    • Works on Linux, macOS, WSL
    • No external tool requirements

  5. Transparency

    • Verbose mode shows every step
    • Clear error messages

📚 Related Documentation

Understanding the architecture helps you use Clean BOM Senior effectively and debug issues.

Clone this wiki locally