Skip to content

Benchmark RLI flow with a large table to improve performance #16952

@hudi-bot

Description

@hudi-bot

High level context

Benchmark RLI for tables on an existing table with large number of record keys (~100B). 
Incrementally ingest about 10GB of data, MoR table, partitioned with ~500 partitions. 

Use Hfile size of 2GB.

  • Ensure the bootstrap of RLI works as expected.
  • Measure the read and write latencies for the RLI index
  • Find and measure all bottlenecks
  • Report any issue with the core indexing or RLI or MDT DAGs.

JIRA info

Metadata

Metadata

Assignees

Labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions