-
Notifications
You must be signed in to change notification settings - Fork 800
[Feature] GCS download parallelism support #429
Copy link
Copy link
Closed
Labels
enhancementNew feature or requestNew feature or request
Milestone
Description
Describe the bug
Currently GCS data is downloaded in a single thread in Flytekit. gsutil has support for parallel downloads, that should be leveraged.
Expected behavior
Faster downloads
Flyte component
- Overall
- Flyte Setup and Installation scripts
- Flyte Documentation
- Flyte communication (slack/email etc)
- FlytePropeller
- FlyteIDL (Flyte specification language)
- Flytekit (Python SDK)
- FlyteAdmin (Control Plane service)
- FlytePlugins
- DataCatalog
- FlyteStdlib (common libraries)
- FlyteConsole (UI)
- Other
To Reproduce
NA
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request