Parquet Blocks Production Readiness

[annanay25]

Meta issue to track production readiness of the Apache Parquet block format.

References

• Design proposal Design proposal: Parquet blocks #1407
• Initial PR adding parquet support Add Parquet block format #1479

Tracking production readiness

• Fix FindTraceByID [@mdisibio]
  • Fix panic when scanning rowgroups with multiple pages
  • Current bug with find by ID logic? (Migrate TraceID column from string to byte slice #1531)
• Serverless build
  • Make a decision on Google Cloud Run vs. Cloud Functions
  • Swap serverless querying to vParquet
  • Rework our CI pipeline
• Caching [@annanay25]
  • Put metadata/footer in real cache.
• Fine tuning [@joe-elliott]
  • Row group size (100MB? 10000 traces?)
  • Read buffer size/count (1MB x 16 currently)
  • Search shard target bytes per job (30MB, 100MB?)
  • Sort inner trace content: Sort batches by span name - does this significantly improve the common case of searching for span by name?
• Compaction [@mdisibio]
  • Pre-fetch before combining?
  • Fix combiner metrics -- move combiner into versioned encodings
• Performance/Stability testing in highly multitenant environments
• Upstream issues: [@mdisibio]
  • Schema conversion fails with error "missing optional column <x>" segmentio/parquet-go#250
  • Reading of complex structs with []byte fields is incorrect near page boundaries segmentio/parquet-go#254

Nice to haves (non-blocking)

• CLI (prints schema & column sizes already) [@mdisibio]
  • iterate through a column and debug print all values Migrate TraceID column from string to byte slice #1531

Future

The following are out of scope for production readiness of parquet blocks, rather its a rough roadmap for the parquet block format.

• Cache specific resource attribute columns? cluster / namespace / service.name ..
• Switch WAL over to vParquet
• Dynamic column support
• Implement TraceQL over Parquet blocks.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days.
The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity.
Please apply keepalive label to exempt this Issue.

[mdisibio]

I would say at this point parquet is production ready :) And we implemented almost everything on this list, including all of the roadmap laid out here. Going to go ahead and close.