Skip to content

Change byte size estimate for compressed files for FileBasedSources #19886

@damccorm

Description

@damccorm

Consider change the estimated byte size for file based sources for better initial autoscaling (on runners which support it; currently only Dataflow). Currently the estimate is based on total size, for compressed file we could add a factor multiplier (~5) for compressed files in a blob go produce a better estimate.

Imported from Jira BEAM-8495. Original Jira may contain additional context.
Reported by: altay.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions