Logstash to Vector Migration

This repository contains the complete migration of a Logstash pipeline to Vector, including configuration, documentation, testing framework, and validation tools.

Status

✅ Functionally Complete - All parsing logic implemented and validated
✅ 35/35 Unit Tests Passing - Complete test coverage
✅ Integration Testing Framework - Ready for baseline comparison
🔄 91% Overall Progress - Production deployment pending

Quick Start

Validate Configuration

vector validate impl/vector.yaml

Run Unit Tests

vector test impl/vector.yaml

Run Integration Tests

cd tests/integration
./run_all_tests.sh

Repository Structure

.
├── impl/
│   ├── vector.yaml              # Production Vector configuration (35 unit tests)
│   ├── vrl/                     # Externalized VRL parsing logic
│   │   ├── 01_enrich_static.vrl
│   │   ├── 02_parse_filename.vrl
│   │   ├── 03_parse_core_fields.vrl
│   │   ├── 04_parse_mask_group_id.vrl
│   │   ├── 05_parse_action.vrl
│   │   ├── 06_parse_mask_lot_id.vrl
│   │   ├── 07_parse_other_fields.vrl
│   │   └── 08_derive_and_cleanup.vrl
│   └── implementation-summary.md # Implementation status and summary
├── doc/
│   ├── requirements.md          # Detailed migration requirements
│   ├── todo.md                  # Task tracking and progress
│   └── testing-procedures.md    # Testing and validation procedures
├── tests/
│   └── integration/             # Complete integration testing framework
│       ├── README.md            # Comprehensive testing guide
│       ├── QUICKSTART.md        # Quick reference
│       ├── TESTING_SUMMARY.md   # Testing summary
│       ├── run_all_tests.sh     # Master test runner
│       └── data/                # Test data and baselines
├── sample/                      # Sample Logstash configs and log files
├── AGENTS.md                    # Agent guidelines and code style
├── INTEGRATION_TESTING_COMPLETE.md  # Integration testing status
└── README.md                    # This file

Key Features

Parsing Capabilities

• ✅ 12 business fields extraction with multiple grok pattern fallbacks
• ✅ Multiline event aggregation for ap_log type
• ✅ Conditional field removal (query phase logic)
• ✅ Field derivation (maskGroupId from product-layer)
• ✅ Type conversions (MaskListNo to integer)

Testing Infrastructure

• ✅ 35 embedded unit tests in Vector configuration
• ✅ Complete integration testing framework
• ✅ Logstash baseline comparison tools
• ✅ Elasticsearch document validation
• ✅ Test data for all scenarios (normal, query phase, malformed)

Documentation

For Developers

• AGENTS.md - Code style, guidelines, and development practices
• doc/requirements.md - Detailed migration requirements and parsing rules
• impl/implementation-summary.md - Implementation status

For Testing

• tests/integration/README.md - Comprehensive testing guide
• tests/integration/QUICKSTART.md - Quick testing reference
• doc/testing-procedures.md - Testing procedures

For Project Management

• doc/todo.md - Task tracking and progress (91% complete)
• INTEGRATION_TESTING_COMPLETE.md - Integration testing status

Configuration

Vector Configuration

• Location: impl/vector.yaml
• VRL Logic: Externalized in impl/vrl/ (8 files for modular parsing)
• Components: File source, remap transforms, Elasticsearch sink, console sink
• Tests: 35 embedded unit tests
• Status: Production-ready

Input

• Source: File-based (/app/log/web_*.log)
• Type: ap_log
• Multiline: Aggregates TRACE events starting with before SysUuid::set()

Output

• Production: Elasticsearch sink (elasticsearch-fz1.engmon.svc.cluster.local:9200)
• Local Testing: Console sink (outputs JSON to stdout)
• Index Pattern: {{ POD_NAMESPACE }}-%Y.%m.%d
• Format: Logstash format

Testing

Unit Tests

vector test impl/vector.yaml

Result: 35/35 tests passing ✅

Integration Tests

cd tests/integration
./run_all_tests.sh

Includes:

• Vector configuration validation
• Unit test execution
• Baseline comparison (requires Docker)
• Output validation

Test Coverage

• ✅ All 12 business fields
• ✅ Multiline processing
• ✅ Query phase conditional logic
• ✅ Field derivation
• ✅ Type conversions
• ✅ Error handling

Next Steps

1. Run Full Baseline Comparison - Validate 100% parity with Logstash
2. Performance Testing - Load testing with production volumes
3. Production Deployment - Deploy to production environment
4. Monitoring Setup - Configure alerting and observability

Requirements

• Vector CLI (v0.52.0+)
• Docker and Docker Compose (for full integration testing)
• Python 3.7+ with uv (for comparison scripts)

Support

• Issues: Check documentation in doc/ and tests/integration/
• Testing: See tests/integration/README.md for comprehensive guide
• Configuration: See doc/requirements.md for detailed specifications

---

Last Updated: 2026-01-29
Version: 1.0
Status: ✅ Functionally Complete, Ready for Production Validation