multipart patches

Multi-Part Patch System

Overview

CyberPatchMaker automatically splits large patches into multiple parts when the patch size exceeds 4GB. This prevents memory exhaustion issues and makes large patches more manageable.

How It Works

Automatic Splitting

When generating a patch, the system:

1. Calculates total patch size before saving
2. Sorts operations by size with key file operations prioritized
3. Splits into 4GB parts if total size > 4GB
4. Generates multi-part metadata for verification
5. Saves parts as separate files (.01.patch, .02.patch, etc.)

File Naming Convention

patch-name.01.patch  ← Part 1 (contains key file + metadata + small files)
patch-name.02.patch  ← Part 2
patch-name.03.patch  ← Part 3 (if needed)
...

Part 1 Special Role

Part 1 (.01.patch) contains:

• Key file verification data (always in part 1)
• Multi-part metadata (total parts, part hashes)
• Small files (fill up to 4GB limit)
• Part verification hashes (SHA-256 of all parts)

Size Distribution Example

Given these files:

File 1: 100MB
File 2: 5.5GB
File 3 (keyfile): 450MB
File 4: 2.2GB
File 5: 50MB
File 6: 9.8GB
File 7: 1.6GB
File 8: 300MB
File 9: 7.4GB
File 10: 120MB

Sorting Order:

File 3 (keyfile): 450MB  ← Always first
File 5: 50MB
File 1: 100MB
File 10: 120MB
File 8: 300MB
File 7: 1.6GB
File 4: 2.2GB
File 2: 5.5GB
File 9: 7.4GB
File 6: 9.8GB

Part Distribution:

Part 1 (.01.patch): Files 3,5,1,10,8,7 = ~2.6GB

• Key file (File 3): 450MB
• Small files (5,1,10,8): 570MB
• Medium file (7): 1.6GB
• Multi-part metadata + hashes

Part 2 (.02.patch): File 4 = 2.2GB

Part 3 (.03.patch): File 2 = 5.5GB (exceeds 4GB, gets own part)

Part 4 (.04.patch): File 9 = 7.4GB (exceeds 4GB, gets own part)

Part 5 (.05.patch): File 6 = 9.8GB (exceeds 4GB, gets own part)

Generating Multi-Part Patches

CLI Generator

# Automatically splits if needed (no extra flags required)
patch-gen.exe --from-dir "C:\Version1" --to-dir "C:\Version2" --output patches

# Output:
# Patch size (27.5 GB) exceeds 4GB limit, splitting into multiple parts...
# Split patch into 5 parts
#   Part 1: 6 operations, ~2.6GB
#   Part 2: 1 operation, ~2.2GB
#   Part 3: 1 operation, ~5.5GB
#   Part 4: 1 operation, ~7.4GB
#   Part 5: 1 operation, ~9.8GB
# ✓ Multi-part patch saved: 5 parts

Custom Split Sizes

The CLI automatically detects and splits large patches. Use --splitsize to customize the part size:

# Auto-split with default 4GB limit
patch-gen.exe --from-dir "C:\v1" --to-dir "C:\v2" --output patches

# Custom split size (2GB parts)
patch-gen.exe --from-dir "C:\v1" --to-dir "C:\v2" --output patches --splitsize 2G

# Small parts for limited upload space
patch-gen.exe --from-dir "C:\v1" --to-dir "C:\v2" --output patches --splitsize 500MB --bypasssplitlimit

Progress indicators show:

• Total patch size
• Number of parts
• Part generation progress

Applying Multi-Part Patches

Automatic Detection

The applier automatically detects multi-part patches:

# Just specify part 1 - all parts loaded automatically
patch-apply.exe --patch "patches\Version1-to-Version2.01.patch" --current-dir "C:\MyApp"

# Output:
# Detected multi-part patch: 5 parts
# Loading part 2: Version1-to-Version2.02.patch
#   ✓ Part 2 hash verified
# Loading part 3: Version1-to-Version2.03.patch
#   ✓ Part 3 hash verified
# Loading part 4: Version1-to-Version2.04.patch
#   ✓ Part 4 hash verified
# Loading part 5: Version1-to-Version2.05.patch
#   ✓ Part 5 hash verified
# ✓ Loaded 15 total operations from 5 parts

User-Friendly Behavior

• Specify any part: If you run .02.patch, applier redirects to .01.patch
• Hash verification: All parts verified before applying
• Missing parts detected: Error if any part is missing or corrupted
• Progress feedback: Clear status for each part loaded

Technical Details

Part Metadata Structure

{
  "MultiPart": {
    "IsMultiPart": true,
    "PartNumber": 1,
    "TotalParts": 5,
    "MaxPartSize": 4294967296,
    "PartHashes": [
      {
        "PartNumber": 1,
        "Checksum": "abc123...",
        "Size": 2684354560
      },
      {
        "PartNumber": 2,
        "Checksum": "def456...",
        "Size": 2200000000
      }
      // ... more parts
    ]
  }
}

Verification Process

1. Load Part 1 first (contains metadata)
2. Check PartHashes array for total parts
3. Load each remaining part sequentially
4. Verify SHA-256 hash before accepting part
5. Merge operations from all parts
6. Apply combined patch as single unit

Memory Efficiency

Multi-part patches solve the memory exhaustion problem:

• Single 18GB patch: Tries to allocate 64GB+ for JSON parsing → OOM crash
• Multi-part (5x 3-4GB): Loads parts sequentially, max ~4GB per part → Success

Chunk Sidecar System (Very Large Patches)

For patches where individual parts exceed the configured split size, CyberPatchMaker uses a chunk sidecar system to further break down parts into smaller chunk files.

When Chunking Occurs

• Chunking is opt-in: only activates when --splitsize is explicitly provided
• When a saved part file exceeds the split size, the system:
  1. Chunks the part data into smaller files (each sized to the --splitsize value)
  2. Creates a .chunks.json sidecar file with chunk metadata
  3. Reconstructs the part on-the-fly when loading
• Without --splitsize, parts are never chunked even if they exceed 4GB

Chunk File Structure

patch-name.01.patch                    ← Part 1 (small, contains metadata)
patch-name.02.patch                    ← Part 2 (regular, not chunked)
patch-name.part3.chunks.json           ← Sidecar for chunked Part 3
patch-name.part3.1.patch               ← Chunk 1 of Part 3
patch-name.part3.2.patch               ← Chunk 2 of Part 3
patch-name.part3.3.patch               ← Chunk 3 of Part 3

Sidecar JSON Format

{
  "part_number": 3,
  "chunks": [
    {
      "FileName": "patch-name.part3.1.patch",
      "PartNumber": 3,
      "ChunkNumber": 1,
      "Size": 536870912,
      "Checksum": "abc123def456..."
    },
    {
      "FileName": "patch-name.part3.2.patch",
      "PartNumber": 3,
      "ChunkNumber": 2,
      "Size": 536870912,
      "Checksum": "789ghi012jkl..."
    }
  ]
}

Chunk Reconstruction Process

When applying a patch with chunked parts:

1. Detect sidecar: If <part>.chunks.json exists, part is chunked
2. Load sidecar: Read chunk metadata (file names, offsets, checksums)
3. Read chunks: Load each chunk file and verify SHA-256 checksum
4. Reassemble: Combine chunks in order into temporary file
5. Verify reconstructed hash: Ensure full part matches expected hash
6. Load part: Parse reconstructed part as normal patch data

Benefits of Chunking

• Handles any patch size: Even 50GB+ patches can be processed
• Memory-safe: Never loads more than one chunk at a time
• Parallel download ready: Chunks can be downloaded independently
• Checksum verification: Each chunk verified individually

Self-Contained Executables with Chunks

When creating self-contained executables with very large patches:

• Chunk sidecar embedded in the executable
• Chunk files distributed separately alongside the .exe
• Executable auto-detects chunks and reassembles during application

Example distribution:

MyApp-Patch-1.0.3.exe                  ← ~50MB (contains applier + patch metadata)
MyApp-Patch-1.0.3.exe.chunks.json     ← Chunk manifest
MyApp-Patch-1.0.3.exe.part1.1.patch    ← First chunk file
MyApp-Patch-1.0.3.exe.part1.2.patch    ← Second chunk file

Binary Sidecar Embedding Format

When creating a self-contained executable with --create-exe, the generator embeds chunk sidecar JSON files directly into the executable binary. The layout on disk is:

+---------------------------+
| Applier EXE (stub)       |  ← patch-apply.exe binary
+---------------------------+
| Patch Data (part 01)      |  ← Compressed .patch content
+---------------------------+
| Sidecar Blob (optional)   |  ← Embedded chunk sidecar data
+---------------------------+
| 128-byte Header           |  ← Metadata trailer
+---------------------------+

Sidecar Blob Format (binary, little-endian):

Field	Type	Description
Count	uint32	Number of embedded sidecar files
For each sidecar:
Name Length	uint16	Length of sidecar filename in bytes
Name	[]byte	Sidecar filename (e.g., mypass.part2.chunks.json)
Data Length	uint64	Length of sidecar data in bytes
Data	[]byte	Raw sidecar JSON content

128-byte Header Layout:

Offset	Size	Field
0	8 bytes	Magic: CPMPATCH
8	4 bytes	Version (uint32, currently 1)
12	8 bytes	Stub size (uint64, size of applier EXE)
20	8 bytes	Data offset (uint64, same as stub size)
28	8 bytes	Data size (uint64, size of patch data)
36	16 bytes	Compression type string (e.g., "zstd")
52	32 bytes	SHA-256 checksum of patch data
84	1 byte	Flags (bit 0: silent mode)
85	43 bytes	Reserved

At runtime, the applier reads the 128-byte header from the end of the executable, locates and extracts any embedded sidecar files, writes them to the same directory as the executable, and uses them to reconstruct chunked parts before applying the patch. Temporary files (extracted sidecars and part 01 data) are cleaned up after loading.

Compatibility

Backward Compatibility

• Single-part patches continue to work exactly as before
• No version detection needed - format is self-describing
• Legacy appliers will fail gracefully on multi-part patches (missing metadata error)

Forward Compatibility

• MaxPartSize is configurable in metadata (adjustable via --splitsize flag)
• PartHashes array allows adding new part entries without format changes

Best Practices

For Patch Creators

1. Let the system auto-split - don't try to manually split
2. Keep all parts together in same directory
3. Distribute all parts when sharing patches
4. Test with part 1 - verify all parts present
5. Include README explaining multi-part patches to users

For End Users

1. Download all parts before applying
2. Keep parts in same folder during application
3. Run part 1 - system loads others automatically
4. Don't rename parts - breaks automatic detection
5. Verify all parts present - check for .01, .02, .03, etc.

Troubleshooting

"Part X missing" Error

Cause: Not all parts downloaded or incorrect directory

Solution: Ensure all .XX.patch files in same directory as .01.patch

"Hash mismatch for part X"

Cause: Part file corrupted during download/transfer

Solution: Re-download the corrupted part file

"Failed to load multi-part patch"

Cause: Part 1 doesn't contain metadata (corrupted or wrong file)

Solution: Re-download part 1 (.01.patch file)

Out of Memory Despite Multi-Part

Cause: Individual operations still very large despite multi-part splitting

Solution: Use --splitsize with a smaller value to enable chunk sidecar splitting, which further subdivides oversized parts. The chunk sidecar system handles files of any size.

Performance Considerations

Patch Generation

• Sorting overhead: Minimal (< 1 second for 100k files)
• Part calculation: Linear time O(n) operations
• Hash generation: Proportional to patch size
• Disk I/O: Main bottleneck (same as single-part)

Patch Application

• Sequential loading: Each part loaded/verified in order
• Memory usage: Peak = largest single part (~4GB)
• Verification time: Proportional to number of parts
• Overall time: Similar to single-part (I/O bound)

Limitations

Current Limitations

1. Configurable part size - default 4GB, adjustable via --splitsize flag
2. Sequential loading - parts loaded one at a time (could be parallelized)
3. No compression across parts - each part compressed independently
4. Metadata only in Part 1 - other parts don't have standalone info

Known Edge Cases

1. Single file > 4GB: Gets own part (working as designed)
2. Key file > 4GB: Part 1 = just key file + metadata
3. Many tiny files: May create very unbalanced parts
4. Rename detection: Currently none (user must not rename)

Future Enhancements

Planned Features

1. Parallel part loading - speed up multi-part application
2. Delta compression - cross-part deduplication
3. Smart part sizing - better balance for many small files
4. Integrity validation - verify parts before generation completes
5. Resume support - restart failed multi-part downloads
6. Part compression - optimize individual part sizes further

FAQ

Q: Why 4GB default limit? A: Balances between manageable part sizes and avoiding too many parts. Prevents 32-bit integer overflow issues. The limit is configurable via --splitsize (e.g., --splitsize 2G).

Q: Can I combine parts manually? A: No - the format requires proper metadata merging. Use the applier to load all parts.

Q: Do I need all parts to view patch info? A: Yes - Part 1 contains metadata but operations span all parts.

Q: Can I create executables from multi-part patches? A: Yes - use --create-exe with multi-part patches. The generator embeds part 01 data inside the executable. External parts (.02.patch, etc.) and chunk sidecar files must be distributed alongside the .exe. The embedded executable auto-detects and loads external parts at runtime.

Q: What if I lose Part 2 but have others? A: Application will fail - all parts required for complete patch.

Q: Can old appliers use multi-part patches? A: No - they'll error on missing MultiPart metadata. Upgrade to latest version.

Q: Is compression per-part or across all parts? A: Per-part - each .XX.patch file is independently compressed.

Q: How much disk space for multi-part patches? A: Same as single-part (slightly more due to metadata duplication).