Skip to content

scan caching

Jump to bottom
Cyber Official edited this page May 31, 2026 · 1 revision

Scan Caching

CyberPatchMaker includes intelligent scan caching that dramatically speeds up subsequent patch generations. Instead of re-scanning large directories on every build, previously scanned versions can be loaded from cache in under a second.

Overview

When generating patches, the most time-consuming operation is scanning directory trees and calculating SHA-256 hashes for every file. For large projects with tens of thousands of files, this can take 15+ minutes.

Scan caching eliminates this bottleneck by storing the complete scan results (including manifest) to disk. On subsequent patch generations, the cached data is loaded and validated instantly.

Performance Impact

Real-World Example (War Thunder)

  • Files: 34,650 files
  • Size: ~56 GB
  • First scan (without cache): ~15 minutes
  • Cached scan load: <1 second
  • Speedup: 900x faster

Benefits

  • Instant patch generation for previously scanned versions
  • Key file hash validation ensures cache integrity
  • Location-based hashing prevents using wrong cached data
  • Transparent operation - works with all generation modes

How It Works

Cache Storage

Cached scans are stored in a dedicated cache directory (default: .data/):

.data/
├── scan_1.0.0_a1b2c3d4e5f6g7h8.json
├── scan_1.0.1_i9j0k1l2m3n4o5p6.json
└── scan_1.0.2_q7r8s9t0u1v2w3x4.json

Cache Filename Format

Each cache file is named: scan_<version>_<locationHash>.json

  • version: The version number (e.g., "1.0.0")
  • locationHash: First 16 characters of SHA-256 hash of the location path

The location hash ensures that cached scans from one directory cannot be mistakenly used for another directory, even if version numbers match.

Cache File Structure

{
  "version": "1.0.0",
  "location": "/path/to/versions/1.0.0",
  "key_file": {
    "Path": "game.exe",
    "Checksum": "abc123...",
    "Size": 15728640
  },
  "manifest": {
    "Version": "1.0.0",
    "KeyFile": {...},
    "Files": [...],
    "Directories": [...],
    "Timestamp": "2024-01-15T10:30:00Z",
    "TotalSize": 56987654321,
    "TotalFiles": 34650,
    "Checksum": "def456..."
  },
  "cached_at": "2024-01-15T10:30:05Z",
  "location_hash": "a1b2c3d4e5f6g7h8"
}

Usage

Enable Scan Caching

Use the --savescans flag when generating patches:

# First generation - creates cache
patch-gen --versions-dir ./versions --new-version 1.0.3 --output ./patches --savescans

Subsequent Generations

The cache is automatically used on subsequent runs:

# Second generation - uses cache (instant!)
patch-gen --versions-dir ./versions --new-version 1.0.4 --output ./patches --savescans

Custom Cache Location

Specify a custom cache directory with --scandata:

# Use shared cache for team collaboration
patch-gen --versions-dir ./versions --new-version 1.0.3 --output ./patches --savescans --scandata ./shared-cache

Force Rescan

If files have changed and you need to invalidate cache:

# Force fresh scan
patch-gen --versions-dir ./versions --new-version 1.0.3 --output ./patches --savescans --rescan

Cache Validation

Key File Hash Verification

When loading from cache, CyberPatchMaker verifies:

  1. Cached scan exists for the version
  2. Location hash matches (prevents wrong directory usage)
  3. Key file checksum matches the cached manifest

If the key file has changed, the cache is invalidated and a fresh scan is performed automatically.

Location-Based Isolation

Each location gets its own cache entries:

# Project A at /builds/project-a/versions/1.0.0
.data/scan_1.0.0_a1b2c3d4e5f6g7h8.json

# Project B at /builds/project-b/versions/1.0.0
.data/scan_1.0.0_x9y0z1a2b3c4d5e6.json

Even with identical version numbers, projects maintain separate caches.

Cache Management

List Cached Scans

View all cached scans programmatically:

cache := cache.NewScanCache(".data")
scans, _ := cache.ListCachedScans()

for _, scan := range scans {
    fmt.Printf("%s @ %s (%d files, %d bytes)\n",
        scan.Version, scan.Location, scan.TotalFiles, scan.TotalSize)
}

Delete Specific Cache Entry

Remove a cached scan:

cache.DeleteScan("1.0.0", "/path/to/versions/1.0.0")

Clear All Cache

Remove all cached scans:

cache.ClearCache()

Internal API

ScanCache Structure

type ScanCache struct {
    cacheDir string  // Directory containing cache files
}

Key Methods

Method Description
NewScanCache(cacheDir string) Create new cache manager (defaults to .data/)
SaveScan(version *utils.Version) Save scan to cache
LoadScan(versionNumber, location string) Load with validation
HasCachedScan(versionNumber, location string) Check if cache exists
DeleteScan(versionNumber, location string) Remove cached scan
ClearCache() Remove all cached scans
ListCachedScans() Get info about all cached scans
GetCacheDir() Returns the cache directory path as a string

Best Practices

For Development Workflows

  1. Enable caching by default in CI/CD pipelines
  2. Use shared cache location for team builds
  3. Version the cache directory alongside source code
  4. Clean cache periodically to remove old versions

For Production Builds

  1. Cache on first build of each version
  2. Reuse cache for all patches from that version
  3. Validate cache integrity with key file checksums
  4. Store cache separately from build artifacts

For Continuous Integration

# Mount cache volume in CI
docker run -v ./cache:/app/.data patch-gen \
  --versions-dir ./versions \
  --new-version 1.0.3 \
  --output ./patches \
  --savescans

Troubleshooting

Cache Not Working

  • Symptom: Scans take full time despite --savescans
  • Cause: Cache directory not writable or path incorrect
  • Solution: Check --scandata path and permissions

Stale Cache Data

  • Symptom: Patches include old files
  • Cause: Files changed without cache invalidation
  • Solution: Use --rescan flag to force fresh scan

Cache Location Mismatch

  • Symptom: Cache not found for known version
  • Cause: Location path changed (relative vs absolute)
  • Solution: Use consistent paths or --rescan

Version History

  • v1.0.10: Initial scan caching implementation
    • --savescans flag for cache creation
    • --scandata flag for custom cache location
    • --rescan flag for cache invalidation
    • Location-based hashing for isolation
    • Key file validation for integrity

Related Documentation

Clone this wiki locally