Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File.copy_to should allow to upload/download files across 'remote' backends #8833

Closed
radeusgd opened this issue Jan 23, 2024 · 4 comments · Fixed by #9054
Closed

File.copy_to should allow to upload/download files across 'remote' backends #8833

radeusgd opened this issue Jan 23, 2024 · 4 comments · Fixed by #9054
Assignees
Labels
-libs Libraries: New libraries to be implemented

Comments

@radeusgd
Copy link
Member

We should give the user the ability to easily transfer files between the local drive, the Enso Cloud, and S3 (as well as other backends that may be added in the future).

To allow for this, we shall allow the File.copy_to method to take files from different backends, and also ensure that the copy_to method is also present on Enso_File and S3_File.

For example (File.new "foo.txt") . copy_to (S3_File.new "s3://foo/bar.txt") should allow to upload a file to S3, while (S3_File.new "s3://foo/bar.txt").copy_to (File.new "foo.txt") should allow to download it.

@radeusgd radeusgd added the -libs Libraries: New libraries to be implemented label Jan 23, 2024
@radeusgd
Copy link
Member Author

To be extensible for future 'file backends', we could introduce a type File_Like and define from conversions for all kinds of types (File, S3_File, Enso_File), so that the File.copy_to self (destination : File_Like) could still check that it got the valid argument. The File_Like 'interface' could also provide a generic interface for copying files across 'backends'.

It should be possible to also perform copying across 2 'remote' backends (even if it may actually download a temporary file under the hood), i.e. (S3_File.new "s3://foo/bar.txt").copy_to (Enso_File.root / "foo.txt"), or between different buckets of S3: (S3_File.new "s3://foo/bar.txt").copy_to (S3_File.new "s3://other_bucket/bar.txt") (which also may need to download the temporary file?).

@radeusgd radeusgd changed the title File.copy_to should allow to upload/download files across 'external' backends File.copy_to should allow to upload/download files across 'remote' backends Jan 23, 2024
@enso-bot
Copy link

enso-bot bot commented Feb 13, 2024

Radosław Waśko reports a new STANDUP for yesterday (2024-02-12):

Progress: Added tests for copying across and within S3 backend, implemented move_to. It should be finished by 2024-02-14.

Next Day: Next day I will be working on the same task. Implement File->S3 direction. Improve the logic to avoid unnecessary temp files.

@enso-bot
Copy link

enso-bot bot commented Feb 13, 2024

Radosław Waśko reports a new STANDUP for today (2024-02-13):

Progress: Fixed remaining tests for copying/moving, added some more. Cannot test Enso_File yet as writing is not enabled. At least added method stubs. It should be finished by 2024-02-14.

Next Day: Next day I will be working on the #9047 task. Fix impl after updating test for delete (it should fail if file did not exist). Work on updating Enso Cloud integration.

@enso-bot
Copy link

enso-bot bot commented Feb 15, 2024

Radosław Waśko reports a new STANDUP for yesterday (2024-02-14):

Progress: Fixing S3 delete edge cases. Profiling S3 operations performance. It should be finished by 2024-02-14.

Next Day: Next day I will be working on the #9047 task. Updating Enso Cloud integration.

@mergify mergify bot closed this as completed in #9054 Feb 16, 2024
mergify bot pushed a commit that referenced this issue Feb 16, 2024
- Closes #8833
- Tests for copying between S3 and `Enso_File` will only be added once we implement Enso_File writing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
-libs Libraries: New libraries to be implemented
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

1 participant