Enhance n-gram analyzer: configurable parameters, advanced tokenization, and large dataset support

.ai-context/00_bootstrap.md

@@ -0,0 +1,26 @@
+# Mango Tango CLI - Bootstrap Context
+
+## Project Identity
+
+Mango Tango CLI is a modular, terminal-based social media analytics platform designed for flexible, context-aware data exploration. It enables researchers and analysts to perform deep, adaptive analysis of social media datasets through a plugin-based analyzer architecture for coordinated inauthentic behavior (CIB) in datasets of online activity.
+
+## Tech Stack Essentials
+
+- Language: Python 3.12
+- Data Processing: Polars, Parquet
+- UI: Inquirer, Rich
+- Core Libraries: Dash, Shiny, Plotly
+
+## Architectural Pattern
+
+Dependency injection through context objects enables loose coupling between application layers, allowing seamless extension and testing of analysis modules with minimal interdependencies.
+
+## Primary Entry Points
+
+- `mangotango.py`: Application bootstrap
+- `main_menu()`: Interactive terminal workflow
+- Analyzer suite: Pluggable, declarative analysis modules
+
+## Behavioral Requirement
+
+CRITICAL: Apply brutal honesty - challenge assumptions, question designs, and provide direct, analytical feedback without reservation.

.ai-context/01_working_context.md

@@ -0,0 +1,126 @@
+# Working Context - Development Patterns
+
+## Core Architecture Pattern
+
+### Context-Based Dependency Injection
+
+The application uses context objects for loose coupling between layers:
+
+```python
+# Analysis execution pattern
+class AnalysisContext:
+    input_path: Path           # Input parquet file
+    output_path: Path          # Where to write results
+    preprocessing: Callable    # Column mapping function
+    progress_callback: Callable # Progress reporting
+    parameters: dict           # User-configured parameters
+```
+
+### Three-Layer Domain Model
+
+1. **Core Domain**: Application logic, UI components, storage
+2. **Edge Domain**: Data import/export, preprocessing
+3. **Content Domain**: Analyzers, web presenters
+
+## Essential Development Workflows
+
+### Analyzer Development Pattern
+
+```python
+# Declare interface first
+interface = AnalyzerInterface(
+    input=AnalyzerInput(columns=[...]),
+    outputs=[AnalyzerOutput(...)],
+    params=[AnalyzerParam(...)]
+)
+
+# Implement with context
+def main(context: AnalysisContext) -> None:
+    df = pl.read_parquet(context.input_path)
+    # Process data...
+    df.write_parquet(context.output_path)
+```
+
+### Tool Usage Strategy
+
+**Serena Semantic Operations** (symbol-level development):
+
+- `get_symbols_overview()` for file structure
+- `find_symbol()` for specific classes/functions
+- `find_referencing_symbols()` for dependency tracing
+- `replace_symbol_body()` for precise edits
+
+**Standard Operations** (known paths):
+
+- `Read` for specific file content
+- `Edit`/`MultiEdit` for file modifications
+- `Bash` for testing and validation
+
+### Data Processing Pattern
+
+**Parquet-Centric Flow**:
+
+1. Import (CSV/Excel) → Parquet files
+2. Primary Analysis → Normalized results
+3. Secondary Analysis → User-friendly reports
+4. Web Presentation → Interactive dashboards
+
+**Memory Management**:
+
+```python
+from app.utils import MemoryManager
+memory_mgr = MemoryManager()  # Auto-detects system capabilities
+```
+
+## Common Patterns
+
+### Logging Integration
+
+```python
+from app.logger import get_logger
+logger = get_logger(__name__)
+logger.info("Operation started", extra={"context": "value"})
+```
+
+### Progress Reporting
+
+```python
+# Modern Textual-based progress
+progress_manager.add_step("processing", "Processing data", total=1000)
+progress_manager.start_step("processing")
+progress_manager.update_step("processing", 500)
+progress_manager.complete_step("processing")
+```
+
+### Testing Approach
+
+```python
+from testing.context import TestPrimaryAnalyzerContext
+from testing.testers import test_primary_analyzer
+
+# Standardized analyzer testing
+test_primary_analyzer(
+    analyzer_module=your_analyzer,
+    test_context=TestPrimaryAnalyzerContext(...)
+)
+```
+
+## Key File Locations
+
+### Entry Points
+
+- `mangotango.py` - Application bootstrap
+- `components/main_menu.py:main_menu()` - UI entry point
+- `analyzers/__init__.py:suite` - Analyzer registry
+
+### Core Classes
+
+- `app/app.py:App` - Application controller
+- `storage/__init__.py:Storage` - Data persistence
+- `app/app_context.py:AppContext` - Dependency container
+
+### Development References
+
+- See `02_reference/` for detailed symbol information
+- See `@docs/dev-guide.md` for comprehensive development guide
+- See `@.serena/memories/` for deep domain knowledge

.ai-context/setup-guide.md → .ai-context/02_reference/advanced/setup-guide.md

@@ -69,20 +69,24 @@ Should output: "No-op flag detected. Exiting successfully."
 
 **Production Dependencies** (`requirements.txt`):
 
-- `polars==1.9.0` - Primary data processing
+- `polars==1.31.0` - Primary data processing (updated for performance)
 - `pydantic==2.9.1` - Data validation and models
 - `inquirer==3.4.0` - Interactive terminal prompts
 - `tinydb==4.8.0` - Lightweight JSON database
 - `dash==2.18.1` - Web dashboard framework
 - `shiny==1.4.0` - Modern web UI framework
 - `plotly==5.24.1` - Data visualization
 - `XlsxWriter==3.2.0` - Excel export functionality
+- `rich==14.0.0` - Terminal formatting and progress display
+- `python-json-logger==3.3.0` - Structured JSON logging
+- `regex==2025.7.34` - Advanced regex pattern matching
 
 **Development Dependencies** (`requirements-dev.txt`):
 
 - `black==24.10.0` - Code formatter
 - `isort==5.13.2` - Import organizer
 - `pytest==8.3.4` - Testing framework
+- `pytest-benchmark==5.1.0` - Performance testing and benchmarking
 - `pyinstaller==6.14.1` - Executable building
 
 ### Code Formatting Setup
@@ -183,6 +187,31 @@ pytest analyzers/hashtags/test_hashtags_analyzer.py::test_gini
 - Each analyzer should include its own test files
 - Tests use sample data to verify functionality
 
+### Performance Testing
+
+The project includes comprehensive performance testing and benchmarking:
+
+```bash
+# Run performance benchmarks
+pytest testing/performance/ -v
+
+# Run specific performance tests
+pytest testing/performance/test_chunking_optimization.py -v
+
+# Run benchmarks with detailed metrics
+python testing/performance/run_enhanced_benchmarks.py
+
+# Run integration validation tests
+pytest testing/performance/test_integration_validation.py -v
+```
+
+**Performance Test Categories**:
+
+- **Memory detection tests**: Validate auto-detection of system RAM
+- **Adaptive chunking tests**: Verify chunk size optimization
+- **System configuration tests**: Test behavior on different system configs
+- **Benchmarking framework**: Measure actual performance improvements
+
 ## Build Setup (Optional)
 
 ### Executable Building