rulest_v3.2

rulest_v3.2

Changelog (GPU-Resident CELF)

This is a performance-focused community variant of rulest, concentrating on GPU CELF selection, memory usage and larger target sets. Output format remains unchanged.

Key Changes

• GPU-resident CELF: Replaces the in-memory/SQLite coverage matrix with GPU-side target hashing, a lazy-greedy heap, and on-demand exact rescoring using new OpenCL kernels.
• Major memory reduction: Replaced the fixed-width target-byte array with packed storage and offset metadata. For 14.37 million targets, the buffer dropped from ~6.85 GiB to ~114 MiB — approximately 61× less memory for this buffer.
• More scalable preprocessing: Chunked hashing, bounded batches and in-place target deduplication reduce temporary memory overhead.
• Adaptive batch sizing: CELF batch and scratch-memory budgets are derived from detected VRAM, with approximate tiers of 1024 (≥7 GB), 512 (≥3.5 GB) and 256 (below 3.5 GB). Actual batch size may be lower due to runtime constraints.
• Improved progress reporting: Adds a lightweight progress bar and heartbeat spinner for long GPU setup phases, plus more detailed timing and CELF profiling.

Removed

• Coverage-matrix infrastructure (coverage_store.py) and separate full-coverage/recovery passes.
• --local-search, --local-search-swaps, --local-search-pool, --local-search-refresh, and local_search_improve(). No post-CELF 1-swap refinement is performed.

Performance: (RTX 3060 Ti 8 GB, ~14.37M targets)

• Initial CELF processing dropped from ~40 minutes in v3.1a to ~1 minute in v3.2. For a 150k rule budget and ~700k candidates with 14.37 million targets, the new build reaches observed saturation in ~15 minutes, compared with ~40 minutes for the previous selection phase. Results are workload-specific and may vary by GPU and candidate pool.

Summary

The main goals are lower memory pressure and more scalable GPU CELF selection while preserving exact target-byte verification.