Skip to content

Home

Jump to bottom
Ian Ellis edited this page Jun 24, 2025 · 1 revision

ChunkWrap

A Python utility for splitting large files into manageable chunks, masking secrets, and wrapping each chunk with custom prompts for Large Language Model (LLM) processing.

Quick Start

# Install from PyPI
pip install chunkwrap

# Basic usage
chunkwrap --prompt "Analyze this code:" --file myscript.py

# Multiple files
chunkwrap --prompt "Review these files:" --file main.py utils.py config.json

What is ChunkWrap?

ChunkWrap solves a common problem when working with Large Language Models: input size limitations. When you have large codebases, documentation, or data files that exceed an LLM's context window, ChunkWrap intelligently splits them into manageable pieces while maintaining context and tracking progress.

Key Features

  • 🔄 Smart Chunking: Configurable chunk sizes with automatic file concatenation
  • 🔒 Secret Masking: Built-in security with TruffleHog-style regex patterns to redact sensitive data
  • 📋 Multiple Output Modes: Clipboard integration, stdout, or file output
  • 📊 Progress Tracking: Remembers where you left off between sessions
  • 🎯 Flexible Prompting: Different prompts for intermediate vs final chunks
  • ⚙️ Configurable: JSON configuration file for customizing default behavior

Why Use ChunkWrap?

Before ChunkWrap

  • Manually splitting large files
  • Losing track of which chunks you've processed
  • Accidentally exposing API keys or secrets to LLMs
  • Inconsistent prompting across chunks

After ChunkWrap

  • Automated, consistent chunking workflow
  • Automatic progress tracking
  • Built-in secret redaction
  • Standardized prompt formatting
  • Seamless clipboard integration

Installation

From PyPI (Recommended)

pip install chunkwrap

From Source

git clone https://github.com/magicalbob/chunkwrap.git
cd chunkwrap
pip install -e .

Basic Usage Examples

Single File Processing

chunkwrap --prompt "Analyze this Python code:" --file script.py

Multiple Files with Custom Chunk Size

chunkwrap --prompt "Review this codebase:" --file *.py --size 5000

Different Final Prompt

chunkwrap --prompt "Process chunk:" --lastprompt "Provide final summary:" --file large_document.txt

Secret Masking Setup

Create a truffleHogRegexes.json file in your working directory:

{
  "AWS": "AKIA[0-9A-Z]{16}",
  "Slack": "xox[baprs]-[0-9a-zA-Z]{10,48}",
  "GitHub": "ghp_[a-zA-Z0-9]{36}"
}

Output Modes

ChunkWrap supports three output modes:

Mode Command Description
Clipboard --output clipboard Copy to system clipboard (default)
stdout --output stdout Print to terminal
File --output file --output-file result.txt Write to specified file

Configuration

ChunkWrap creates a configuration file on first run:

  • Linux/macOS: ~/.config/chunkwrap/config.json
  • Windows: %APPDATA%\chunkwrap\config.json

Default Configuration

{
  "default_chunk_size": 10000,
  "intermediate_chunk_suffix": " Please provide only a brief acknowledgment that you've received this chunk. Save your detailed analysis for the final chunk.",
  "final_chunk_suffix": "Please now provide your full, considered response to all previous chunks."
}

Output Format

ChunkWrap wraps your content with clear delimiters:

Intermediate chunks:

Your prompt (chunk 2 of 4)
"""
[file content with secrets masked]
"""

Final chunk:

Your final prompt
"""
[remaining content]
"""

Common Workflows

Code Review Workflow

# Review a Python project
chunkwrap --prompt "Review this code for best practices:"\
  --lastprompt "Provide overall assessment and recommendations:"\
  --file src/*.py tests/*.py

Documentation Processing

# Process large documentation
chunkwrap --prompt "Summarize this section:"\
  --lastprompt "Create a comprehensive overview:"\
  --file docs/*.md --size 8000

Data Analysis Workflow

# Analyze large datasets
chunkwrap --prompt "Analyze this data chunk:"\
  --file data.csv logs.txt\
  --output file --output-file analysis_input.txt

State Management

ChunkWrap automatically tracks your progress:

  • Continue processing: Simply run the same command again
  • Reset progress: Use --reset to start over
  • Check progress: The tool shows current chunk position
# Reset and start over
chunkwrap --reset

# Show configuration location
chunkwrap --config-path

Advanced Features

Cross-Platform Support

  • Full clipboard integration on Windows, macOS, and Linux
  • SSH-friendly with --output stdout mode
  • Configurable output destinations

Security First

  • Automatic secret detection and masking
  • Configurable regex patterns
  • Safe handling of sensitive files

Developer Friendly

  • Comprehensive test suite (95%+ coverage)
  • Clean modular architecture
  • Extensive error handling
  • Type hints and documentation

Requirements

  • Python 3.11+
  • pyperclip (for clipboard functionality)

Getting Help

  • Configuration: chunkwrap --config-path
  • Version: chunkwrap --version
  • Reset state: chunkwrap --reset

Architecture Overview

ChunkWrap is built with a modular architecture:

  • Core Processing (core.py): Main orchestration logic
  • File Handling (file_handler.py): Multi-file reading and validation
  • Chunking (chunking.py): Text splitting algorithms
  • Security (security.py): Secret masking with regex patterns
  • Output (output.py): Multiple output mode handling
  • State (state.py): Progress persistence
  • Configuration (config.py): Settings management
  • CLI (cli.py): Command-line interface

Contributing

ChunkWrap is actively developed and welcomes contributions. The project maintains high code quality standards with comprehensive testing and clean architecture.

License

GNU General Public License v3.0 - see LICENSE for details.

Ready to streamline your LLM workflows? Get started with ChunkWrap today!

Clone this wiki locally