Skip to content

Avoid using meta client in FGReader to avoid performance degradation #17154

@hudi-bot

Description

@hudi-bot

{}Goal{}: FGReader should take necessary information from the constructor/builder without having to do additional FS calls and scanning on the executor (e.g., loading table config, scanning timeline, reading schema, etc.) before reading the records.
 
Constructing meta client incurs loading table config, scanning timeline, reading schema, etc., which are very expensive operations, which must not happen per file group.

JIRA info

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions