☂️ [Query] Service Implementation #6
Description
Overview
The Query Service provides query capabilities for observability data stored in Iceberg tables, implementing a Loki-compatible query interface with LogQL support and DataFusion-based optimizations.
Context
IceGate uses Apache DataFusion as its query engine, providing high-performance analytical queries over Arrow-backed data. The Query Service:
- Exposes a Loki-compatible HTTP API for logs queries
- Compiles LogQL queries to DataFusion logical plans
- Optimizes query execution for time-series workloads
- Supports both log record queries and metric aggregations from logs
Sub-Tasks
This parent issue tracks the following implementation tasks:
Core Implementation
- Loki Query Interface #7 - Implement Loki Query Interface
- LogQL Query Compiler (Records + Metrics) #8 - Implement LogQL Query Compiler (Records + Metrics)
- Optimize LogicalPlan Sort Operations #9 - Optimize LogicalPlan Sort Operations
- Optimize Datasource Node for Iceberg Tables #10 - Optimize Datasource Node for Iceberg Tables
LogQL Features (delegated from #8)
- LogQL Pipeline Parsers (json, logfmt, regexp, pattern, unpack) #13 - Implement LogQL Pipeline Parsers (json, logfmt, regexp, pattern, unpack)
- LogQL Pipeline Formatting (label_format, line_format, decolorize) #14 - Implement LogQL Pipeline Formatting (label_format, line_format, decolorize)
- LogQL Unwrap Aggregations (sum_over_time, avg_over_time, etc.) #15 - Implement LogQL Unwrap Aggregations (sum_over_time, avg_over_time, etc.)
- LogQL Binary Operations and Vector Matching #16 - Implement LogQL Binary Operations and Vector Matching
- LogQL Vector Aggregations: topk, bottomk, sort, sort_desc #17 - Implement LogQL Vector Aggregations: topk, bottomk, sort, sort_desc
- LogQL IP Filtering #18 - Implement LogQL IP Filtering
- Time Grid Gap Filling for Matrix Responses #19 - Implement Time Grid Gap Filling for Matrix Responses
- Loki Instant Query Endpoint (/loki/api/v1/query) #20 - Implement Loki Instant Query Endpoint (/loki/api/v1/query)
- Enhanced Label Management (drop/keep with regex matchers) #21 - Implement Enhanced Label Management (drop/keep with regex matchers)
Acceptance Criteria
- Query service accepts LogQL queries via HTTP API
- LogQL is correctly compiled to DataFusion logical plans
- Both logs queries and metric queries are supported
- Query plans are optimized for time-series patterns
- Datasource node efficiently reads from Iceberg tables
- Query performance meets SLA requirements
- Unit and integration tests cover all query types
- Documentation is updated with API examples
Dependencies
- Apache DataFusion 50.3.0 (already in Cargo.toml)
- Iceberg DataFusion 0.7.0 (already in Cargo.toml)
- HTTP server framework (axum)
- LogQL parser (ANTLR4)
Implementation Notes
Located in: src/bin/query.rs
Current status: Core implementation complete (~70%), feature gaps tracked in sub-tasks.