Skip to content

testing guide

Jump to bottom
Cyber Official edited this page May 31, 2026 · 1 revision

Testing Guide

Guide to running and understanding CyberPatchMaker's comprehensive test suite.

Overview

CyberPatchMaker includes a comprehensive test suite with 61 tests (59 standard + 2 optional) that validate all core functionality including generation, application, verification, error handling, backup system, and advanced scenarios like multi-hop patching, bidirectional patching, downgrade testing, compression formats, and automatic rollback.

Key Feature: Test data is automatically generated on first run - no bloat files committed to the repository!

Test Suite

Advanced Test Suite

File: advanced-test.ps1 Shell: PowerShell 5.1 or later Platform: Windows (PowerShell) Tests: 61 comprehensive tests (59 standard + 2 optional) Test Data: Auto-generated on first run (1.0.0, 1.0.1, 1.0.2) Optional Test Flags: -run1gbtest, -runlargefile Command Visibility: Shows exact command-line for each operation (displayed in cyan) Bidirectional Testing: Includes upgrade/downgrade cycle verification

Running Tests

Windows (PowerShell)

# Run the comprehensive test suite
.\advanced-test.ps1

# First run output (auto-generates test data):
Checking for test versions...
Version 1.0.0 not found, creating...
  Creating version 1.0.0...
  Version 1.0.0 created (3 files, 2 directories)
Version 1.0.1 not found, creating...
  Creating version 1.0.1...
  Version 1.0.1 created (4 files, 2 directories)
Version 1.0.2 not found, creating...
  Creating version 1.0.2...
  Version 1.0.2 created (11 files, 6 directories, 3 levels deep)

Created 3 test version(s)

# Then runs all tests with command visibility:
Running CyberPatchMaker Advanced Test Suite
========================================

Testing: Generate complex patch (1.0.1 → 1.0.2) with zstd
  Generating patch from 1.0.1 to 1.0.2 with zstd compression...
  Command: patch-gen.exe --versions-dir .\testdata\versions --from 1.0.1 --to 1.0.2 --output .\testdata\advanced-output\patches --compression zstd
  Patch generated (zstd): 1627 bytes
✓ PASSED: Generate complex patch (1.0.1 → 1.0.2) with zstd

Testing: Apply zstd patch to complex directory structure
  Copying version 1.0.1 to test-zstd...
  Applying zstd patch...
  Command: patch-apply.exe --patch .\testdata\advanced-output\patches\1.0.1-to-1.0.2.patch --current-dir .\testdata\advanced-output\test-zstd --verify
  Zstd patch applied successfully
✓ PASSED: Apply zstd patch to complex directory structure

...
(All 61 tests with command visibility)
========================================
Advanced Test Results
========================================
Passed: 61
Failed: 0

✓ All advanced tests passed!

Would you like to clean up test data now? (Y/N): Y
Cleaning up test data...
✓ Test data removed successfully

Note: The test script automatically generates test versions on first run. Subsequent runs will use the existing test data unless you delete the testdata/versions/ directory.

Test Data Cleanup Management

The test suite includes an intelligent cleanup system to help manage test data:

Interactive Cleanup Prompt:

  • After all tests complete, you'll be prompted to clean up test data
  • Press Y to immediately delete the testdata/ directory
  • Press N to keep test data for inspection

Auto-Delete Behavior:

  • If you choose to keep test data (N), a .cleanup-deferred state file is created
  • On the next test run, the system automatically detects deferred cleanup
  • Previous test data is automatically deleted before creating fresh test data
  • This ensures clean test runs while giving you time to inspect results

Example Workflow:

# First run - Choose to keep data for inspection
.\advanced-test.ps1
# ... tests complete ...
# Would you like to clean up test data now? (Y/N): N
# Test data kept in testdata/ directory (cleanup deferred to next run)

# Inspect test data manually
ls testdata/versions/
ls testdata/patches/

# Second run - Auto cleanup of old data
.\advanced-test.ps1
# Previous test data detected (cleanup was deferred)...
# Removing old test data...
# ✓ Old test data removed
# ... tests run with fresh data ...
# Would you like to clean up test data now? (Y/N): Y
# Cleaning up test data...
# ✓ Test data removed successfully

State File:

  • Location: testdata/.cleanup-deferred
  • Purpose: Tracks that cleanup was deferred
  • Behavior: Triggers auto-delete on next run
  • Cleanup: Automatically removed when test data is deleted

Test Suite Overview

The advanced test suite includes 61 comprehensive tests organized into several categories:

Build & Setup Tests (Tests 1-3)

  1. Verify executables exist - Confirms patch-gen.exe and patch-apply.exe build successfully
  2. Setup advanced test environment - Creates the test output directory structure
  3. Verify test versions exist - Confirms all three test versions (1.0.0, 1.0.1, 1.0.2) are present

Patch Generation Tests (Tests 4-8)

  1. Generate complex patch (1.0.1 -> 1.0.2) with zstd - zstd compression
  2. Generate same patch with gzip compression - gzip compression
  3. Generate same patch with no compression - uncompressed
  4. Compare compression efficiency - Verify all three methods produce valid patches
  5. Dry-run complex patch application - Preview mode without changes

Patch Application Tests (Tests 9-15)

  1. Apply zstd patch to complex directory structure - Apply zstd-compressed patch
  2. Apply gzip patch to complex directory structure - Apply gzip-compressed patch
  3. Apply uncompressed patch to complex directory structure - Apply uncompressed patch
  4. Verify complex directory structure after patching - Verify file tree integrity
  5. Verify new files added in nested paths - Check added files in subdirectories
  6. Verify modified files match version 1.0.2 - Check modified files
  7. Verify all compression methods produce identical results - Cross-compression validation

Multi-Hop & Downgrade Tests (Tests 16-20)

  1. Test multi-hop patching scenario - Sequential patching (1.0.0 -> 1.0.1 -> 1.0.2)
  2. Generate downgrade patch (1.0.2 -> 1.0.1) - Downgrade patch generation
  3. Apply downgrade patch to revert version - Downgrade patch application
  4. Verify downgrade results match version 1.0.1 - Verify downgrade correctness
  5. Test bidirectional patching cycle - Complete upgrade/downgrade cycle

Error Detection Tests (Tests 21-22)

  1. Verify patch rejection for wrong source version - Wrong version rejection
  2. Verify detection of corrupted files in source - Corruption detection

Backup System Tests (Tests 23-27b)

  1. Verify backup creation and mirror structure - Backup with mirror directory structure
  2. Verify selective backup (only modified/deleted files) - Only changed files backed up
  3. Verify backup is preserved after successful patching - Backup preserved post-patch
  4. Verify manual rollback from backup works - Manual restore from backup
  5. Verify backup handles complex nested paths - Deep path backup 27b. Verify deleted directories are backed up with all contents - Deleted directory backup

Performance & Custom Paths Tests (Tests 28-35)

  1. Verify patch generation performance - Speed benchmarking
  2. Test custom paths mode with different directories - Custom --from-dir/--to-dir
  3. Apply patch generated with custom paths mode - Apply custom paths patch
  4. Test custom paths with complex nested directories - Complex custom paths
  5. Verify version number extraction from directory names - Version from dir name
  6. Test compression options with custom paths - Compression with custom paths
  7. Test error handling for non-existent directories - Error handling
  8. Verify backward compatibility with legacy --versions-dir mode - Legacy mode

Self-Contained Executable Tests (Tests 36-40)

  1. Test CLI self-contained executable creation - --create-exe flag
  2. Verify CLI executable structure and header - Magic bytes verification
  3. Test batch mode with CLI executable creation - Batch exe creation
  4. Test 1GB bypass with large patch creation (optional: -run1gbtest) - Large patches
  5. Verify CLI executables use CLI applier - CLI applier verification

File Exclusion & Silent Mode Tests (Tests 41-46)

  1. Verify backup.cyberpatcher directories are excluded - Backup exclusion
  2. Verify .cyberignore file pattern matching - Cyberignore patterns
  3. Verify self-contained executable --silent flag for automation - Silent mode
  4. Verify silent mode generates timestamped log files - Log file generation
  5. Verify generator --silent flag embeds silent mode in created executables - Embedded silent
  6. Verify --crp flag creates reverse patches for downgrades - Reverse patches

Scan Cache Tests (Tests 47-53)

  1. Verify scan cache basic functionality with --savescans - Cache creation
  2. Verify scan cache custom directory with --scandata - Custom cache dir
  3. Verify force rescan with --rescan flag - Force rescan
  4. Verify scan cache performance improvement - Performance comparison
  5. Verify scan cache works with custom paths mode - Cache + custom paths
  6. Verify scan cache file structure and content - Cache file validation
  7. Verify scan cache invalidation on file changes - Cache invalidation

Simple Mode Tests (Tests 54-58)

  1. Verify Simple Mode struct field and encoding - SimpleMode field in Patch struct
  2. Verify runSimpleMode function exists - Simplified applier interface
  3. Verify Simple Mode workflow infrastructure - End-to-end (field exists, function callable)
  4. Verify Simple Mode documentation and feature completeness - Feature validation
  5. Verify Simple Mode addresses real-world use cases - Use case scenarios

Note: The SimpleMode field and runSimpleMode() function exist in the codebase but no generator code path currently sets SimpleMode = true. These tests verify the infrastructure exists.

.cyberignore Advanced & Large File Tests (Tests 59-60)

  1. Verify .cyberignore absolute path pattern support - Absolute path patterns
  2. Verify large file chunked processing and memory optimization (optional: -runlargefile) - Chunked processing

Test Data Structure

Auto-generated test versions:

Version 1.0.0 (Baseline - 3 files, 2 directories):

Expected result: All directories and files present

testdata/versions/1.0.0/
├── program.exe         # "Test Program v1.0.0\n"
├── data/
│   └── config.json     # JSON config file
└── libs/
    └── core.dll        # "Core Library v1.0.0\n"

Version 1.0.1 (Simple update - 4 files, 2 directories):

testdata/versions/1.0.1/
├── program.exe         # Modified: "Test Program v1.0.1\n"
├── data/
│   └── config.json     # Modified: updated features
└── libs/
    ├── core.dll        # Modified: "Core Library v1.5.0\n"
    └── newfeature.dll  # NEW: "New Feature v1.0.0\n"

Version 1.0.2 (Complex structure - 11 files, 6 directories, 3 levels deep):

testdata/versions/1.0.2/
├── program.exe                   # Modified: v1.0.2
├── data/
│   ├── config.json               # Modified: added features
│   ├── assets/images/            # NEW: 3 levels deep
│   │   ├── logo.png              # NEW: binary PNG file
│   │   └── icon.png              # NEW: binary PNG file
│   └── locale/                   # NEW: directory
│       └── en-US.json            # NEW: localization file
├── libs/
│   ├── core.dll                  # Modified: v2.5.0
│   ├── newfeature.dll            # Modified: v1.5.0
│   └── plugins/                  # NEW: directory
│       └── api.dll               # NEW: plugin file
└── plugins/                      # NEW: root-level directory
    ├── sample.plugin             # NEW: plugin file
    └── sample.json               # NEW: plugin config

Understanding Test Output

Success Output

Test 1: Build tools... ✓ PASS

Meaning: Test passed all checks

Failure Output

Test 5: Apply patch with verification... ✗ FAIL
  Expected exit code 0, got 1

Meaning: Test failed, explanation provided

What to do:

  1. Read the error message
  2. Check if test expectations are correct
  3. Debug the failing component
  4. Fix the issue
  5. Re-run tests

Detailed Failure Output

If a test fails, you may see additional details:

Test 8: Pre-verification failure... ✗ FAIL
  Expected exit code ≠ 0, got 0
  Expected file to remain corrupted, but it was restored

Manual Testing

Testing Patch Generation

# Create test versions
mkdir -p testdata/manual/1.0.0
mkdir -p testdata/manual/1.0.1
echo "Version 1.0.0" > testdata/manual/1.0.0/app.exe
echo "Version 1.0.1" > testdata/manual/1.0.1/app.exe

# Generate patch
./patch-gen.exe --versions-dir testdata/manual \
            --new-version 1.0.1 \
            --output testdata/manual/patches

# Verify patch exists
ls testdata/manual/patches/
# Should show: 1.0.0-to-1.0.1.patch

Testing Patch Application

# Create test application
mkdir -p testdata/manual/test-app
cp testdata/manual/1.0.0/app.exe testdata/manual/test-app/

# Apply patch
./patch-apply.exe --patch testdata/manual/patches/1.0.0-to-1.0.1.patch \
          --current-dir testdata/manual/test-app \
          --verify

# Verify version updated
cat testdata/manual/test-app/app.exe
# Should show: Version 1.0.1

Testing Dry-Run

# Reset to version 1.0.0
rm -rf testdata/manual/test-app
mkdir -p testdata/manual/test-app
cp testdata/manual/1.0.0/app.exe testdata/manual/test-app/

# Dry-run (no changes)
./patch-apply.exe --patch testdata/manual/patches/1.0.0-to-1.0.1.patch \
          --current-dir testdata/manual/test-app \
          --dry-run

# Verify version unchanged
cat testdata/manual/test-app/app.exe
# Should still show: Version 1.0.0

Testing Pre-Verification

# Create corrupted installation
mkdir -p testdata/manual/corrupted
echo "Corrupted Version" > testdata/manual/corrupted/app.exe

# Try to apply patch (should fail)
./patch-apply.exe --patch testdata/manual/patches/1.0.0-to-1.0.1.patch \
          --current-dir testdata/manual/corrupted \
          --verify

# Should see error: "key file checksum mismatch"
# No backup should be created
ls testdata/manual/corrupted.backup
# Should show: directory not found

Testing Downgrade Patches

Generating a downgrade patch:

# Generate downgrade patch (1.0.1 -> 1.0.0)
./patch-gen.exe --versions-dir testdata/manual \
            --from 1.0.1 \
            --to 1.0.0 \
            --output testdata/manual/patches

# Verify downgrade patch exists
ls testdata/manual/patches/
# Should show: 1.0.1-to-1.0.0.patch

Applying a downgrade patch:

# Create test installation with version 1.0.1
mkdir -p testdata/manual/test-downgrade
cp testdata/manual/1.0.1/app.exe testdata/manual/test-downgrade/

# Apply downgrade patch
./patch-apply.exe --patch testdata/manual/patches/1.0.1-to-1.0.0.patch \
          --current-dir testdata/manual/test-downgrade \
          --verify

# Verify version downgraded to 1.0.0
cat testdata/manual/test-downgrade/app.exe
# Should show: Version 1.0.0

Testing bidirectional patching cycle:

# Start with version 1.0.0
mkdir -p testdata/manual/test-bidirectional
cp testdata/manual/1.0.0/app.exe testdata/manual/test-bidirectional/

# Upgrade to 1.0.1
./patch-apply.exe --patch testdata/manual/patches/1.0.0-to-1.0.1.patch \
          --current-dir testdata/manual/test-bidirectional \
          --verify

cat testdata/manual/test-bidirectional/app.exe
# Should show: Version 1.0.1

# Downgrade back to 1.0.0
./patch-apply.exe --patch testdata/manual/patches/1.0.1-to-1.0.0.patch \
          --current-dir testdata/manual/test-bidirectional \
          --verify

cat testdata/manual/test-bidirectional/app.exe
# Should show: Version 1.0.0 (back to original)

Note: For comprehensive downgrade documentation, see the Downgrade Guide.

Optional Test Flags

The test suite supports optional flags for testing advanced features:

# Test large patches >1GB (requires significant disk space and time)
.\advanced-test.ps1 -run1gbtest

# Test chunked processing for 1.5GB file (memory optimization)
.\advanced-test.ps1 -runlargefile

# Combine flags
.\advanced-test.ps1 -run1gbtest -runlargefile

Test Data Management

Creating New Test Versions

# Create new version directory
mkdir testdata/versions/1.0.3

# Add test files
echo "Version 1.0.3" > testdata/versions/1.0.3/test-app.txt
echo "New feature data" > testdata/versions/1.0.3/feature.txt

# Generate patches
./patch-gen.exe --versions-dir testdata/versions \
            --new-version 1.0.3 \
            --output testdata/patches

Cleaning Test Data

# Remove generated patches
rm -rf testdata/patches/*.patch

# Remove test applications
rm -rf test-app
rm -rf testdata/1.0.0-test
rm -rf testdata/1.0.1-test

# Remove executables (rebuild from source)
rm patch-gen.exe patch-apply.exe

Troubleshooting Test Failures

Build Failures

Symptom: Build step fails with compilation errors

Solutions:

  1. Check Go version: go version (need 1.24.0+)
  2. Verify code compiles: go build ./...
  3. Check for syntax errors
  4. Update dependencies: go mod tidy

Directory Structure Failures

Symptom: Setup fails to find testdata

Solutions:

  1. Run tests from project root
  2. Check testdata/ directory exists
  3. Check version folders exist (1.0.0, 1.0.1, 1.0.2)
  4. Check test-app.txt files exist in each version

Generation Failures

Symptom: Generator command fails

Solutions:

  1. Check generator executable exists
  2. Verify testdata structure is correct
  3. Check disk space
  4. Run generator manually with verbose output
  5. Check error messages in test output

Application Failures

Symptom: Applier command fails

Solutions:

  1. Check applier executable exists
  2. Verify patch file exists
  3. Check test-app directory is correct
  4. Verify test-app has correct version
  5. Run applier manually with verbose output

Pre-Verification Failures

Symptom: Corruption detection test passes when it should report failure

Solutions:

  1. Verify corruption step actually modifies file
  2. Check that verification is enabled
  3. Check that pre-verification rejects corrupted installations
  4. Verify NO backup is created on failure
  5. This is the critical backup timing test!

Adding New Tests

To add new tests to the advanced test suite:

  1. Edit advanced-test.ps1
  2. Add new test function following the pattern: 
    function Test-NewFeature {
    Write-Host "Test N: New feature description... " -NoNewline
    
    # Setup
    # ... prepare test environment ...
    
    # Execute
    $result = # ... run command ...
    
    # Verify
    if ($result) {
        Write-Host "✓ PASS" -ForegroundColor Green
        return $true
    } else {
        Write-Host "✗ FAIL" -ForegroundColor Red
        Write-Host "  Error description"
        return $false
    }
}
  3. Add test to main execution block
  4. Update test count in summary section
  5. Test thoroughly before committing

Related Documentation

Clone this wiki locally