v1.3.3 - Chunking Support for Large Log Files

What's New in v1.3.3

🎉 Major Feature: Chunking Support for Large Log Files

This release adds intelligent chunking support for handling large log files when using the zen --apex command.

Key Features:

• Automatic Chunking: Files larger than 4.5 MB or containing more than 250 entries are automatically chunked
• Smart Chunk Creation: Each chunk respects size limits (2.5 MB max) and entry limits (250 entries max)
• Oversized Entry Handling: Gracefully handles single log entries that exceed chunk size limits
• Non-blocking Backend: Chunks are sent sequentially with progress feedback
• Backward Compatible: Small files continue to use the original non-chunked flow

Technical Details:

• CLI-side Implementation:

  • scripts/chunking_analyzer.py: Analyzes files to determine chunking strategy
  • scripts/chunk_creator.py: Creates chunks respecting size and entry limits
  • scripts/agent_cli.py: Integrated chunking into message sending flow

• Chunk Metadata: Each chunk includes comprehensive metadata for backend aggregation:

  • chunk_id: Unique identifier shared across all chunks of the same file
  • chunk_index: 0-based index of this chunk
  • total_chunks: Total number of chunks for the file
  • file_hash: Hash of the original file for integrity verification
  • entries_in_chunk: Number of log entries in this chunk
  • chunk_size_mb: Size of the chunk in megabytes

Bug Fixes:

• Fixed import paths to use relative imports for direct script execution (#26)
• Resolved ModuleNotFoundError when running agent_cli directly

Example Usage:

```bash

Chunking automatically activates for large files

zen --apex --logs-path /path/to/large-file.jsonl

Example output:

Chunking strategy: single_file_chunking

Chunking file: large-file.jsonl (10.23 MB, 1153 entries)

Sending 7 chunk(s)...

Chunk 1/7: 250 entries, 1.46 MB

Chunk 2/7: 250 entries, 0.83 MB

...

✓ All 7 chunk(s) sent successfully

```

Full Changelog

Implemented enhancements:

• feat: Implement CLI-side chunking for large log files (chunking_strategy branch)
• feat: Add chunking analyzer for file size and entry count analysis
• feat: Add chunk creator with intelligent size and entry limit handling
• fix: Change chunking imports to relative imports for direct script execution

Merged pull requests:

• #26: Chunking Implementation for Large Log Files

---

🤖 Generated with Claude Code