FastDL Processor

A high-performance C++ application for Source Engine game servers that monitors server directories and automatically creates compressed BZIP2 copies of game files for FastDL web hosting.

Overview

FastDL Processor is designed to be significantly faster than Python implementations, utilizing multi-threading and efficient C++ I/O operations. It preserves original files while creating compressed copies in a separate destination directory.

Features

• Multi-threaded processing - Auto-detects CPU cores for optimal performance
• Multiple server monitoring - Process several game servers simultaneously
• Directory structure preservation - Maintains relative paths from source to destination
• Incremental processing - Only processes new or changed files
• BZIP2 compression - Configurable compression level for optimal speed/size balance
• File stability checking - Ensures files aren't being written during compression
• Cross-platform support - Native support for Linux and Windows
• JSON configuration - Flexible configuration via config.json
• Configurable extensions - Customize which file types to process
• Persistent database - Tracks processed files to avoid redundant work

Supported Games

• Counter-Strike: Source (CS:S)
• Counter-Strike: Global Offensive (CS:GO)
• Team Fortress 2 (TF2)
• Left 4 Dead 2 (L4D2)
• Garry's Mod (GMod)
• Any Source Engine based game

Requirements

Requirement	Version
CMake	3.16+
C++ Compiler	C++17 compatible (GCC 8+, Clang 7+, MSVC 2019+)
libbz2	Development libraries
nlohmann-json	3.x

Building

Linux (Ubuntu/Debian)

sudo apt update
sudo apt install -y cmake g++ build-essential libbz2-dev nlohmann-json3-dev

mkdir build && cd build
cmake ..
make -j$(nproc)

Windows (Visual Studio 2022)

# Install vcpkg dependencies
vcpkg install bzip2 nlohmann-json

# Build with CMake
mkdir build
cd build
cmake .. -DCMAKE_TOOLCHAIN_FILE=[vcpkg-root]/scripts/buildsystems/vcpkg.cmake
cmake --build . --config Release

Configuration

Create config.json in the executable directory:

{
  "check_interval": 300,
  "chunk_size_mb": 8,
  "max_workers": 0,
  "stability_checks": 1,
  "stability_interval_ms": 100,
  "processed_files_db": "processed_files.txt",
  "extensions": [".nav", ".bsp", ".mdl", ".phy", ".vvd", ".vtf", ".vmt", ".wav", ".mp3", ".vtx", ".ani", ".dds", ".pcf", ".svg"],
  "multi_extensions": [".dx80.vtx", ".dx90.vtx", ".sw.vtx"],
  "servers": [
    {
      "name": "MyCSServer",
      "source": "/path/to/csserver/cstrike",
      "destination": "/var/www/fastdl/cstrike"
    }
  ]
}

Usage

# Run with default config.json
./FastDLProcessor

# Custom config file
./FastDLProcessor --config /path/to/config.json

# Quiet mode (errors only)
./FastDLProcessor --quiet

# Debug mode
./FastDLProcessor --debug

Performance

Metric	C++ vs Python
File scanning	10-50x faster
Compression	5-10x faster
Memory usage	Lower
CPU utilization	Better

Author

SyntX

License

MIT License - See LICENSE for details.