What would you like to happen?
There are scenarios where we need to copy a large number of files between file systems, for example, from S3 to GCS.
We currently can do this via a custom composite transform that uses fileio.Match -> fileio.ReadFiles -> custom sink implementation that copies bytes from source to destination without parsing the file format but it's good to have an standard efficient utility in Beam for this.
Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
Issue Components
What would you like to happen?
There are scenarios where we need to copy a large number of files between file systems, for example, from S3 to GCS.
We currently can do this via a custom composite transform that uses
fileio.Match->fileio.ReadFiles->custom sink implementation that copies bytes from source to destination without parsing the file formatbut it's good to have an standard efficient utility in Beam for this.Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
Issue Components