Summary
When partitions are converted to Parquet and uploaded to object storage, QuestDB extracts and stores metadata locally. This enables the query engine to plan and optimize queries without round-trips to remote storage.
How it works
- Partition converts to Parquet (via storage policy)
- Parquet file uploads to object storage (S3, GCS, Azure Blob)
- Metadata (row group info, column statistics, schema) stays local
- Query engine uses local metadata for planning and pruning
- Only required data is fetched from remote storage
Benefits
- Faster query planning — No network latency for metadata access
- Intelligent pruning — Skip row groups based on local min/max statistics
- Reduced costs — Fewer object storage API calls
- Lower latency — Metadata lookups are local disk reads, not remote fetches
Use case
Ideal for large-scale deployments where hot data stays local and cold data tiers to object storage. Queries spanning both hot and cold data remain performant because metadata is always local.
Parent feature
Sub-issue of #62 (Native Apache Parquet partition format)
Summary
When partitions are converted to Parquet and uploaded to object storage, QuestDB extracts and stores metadata locally. This enables the query engine to plan and optimize queries without round-trips to remote storage.
How it works
Benefits
Use case
Ideal for large-scale deployments where hot data stays local and cold data tiers to object storage. Queries spanning both hot and cold data remain performant because metadata is always local.
Parent feature
Sub-issue of #62 (Native Apache Parquet partition format)