Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[C++] Decide if we implement GcsFileSystem::Move #30493

Closed
asfimport opened this issue Dec 2, 2021 · 4 comments
Closed

[C++] Decide if we implement GcsFileSystem::Move #30493

asfimport opened this issue Dec 2, 2021 · 4 comments

Comments

@asfimport
Copy link

asfimport commented Dec 2, 2021

There is no atomic move in GCS, so maybe we do not need to implement this.

Reporter: Carlos O'Ryan / @coryan
Assignee: Carlos O'Ryan / @coryan

Related issues:

PRs and other links:

Note: This issue was originally created as ARROW-14971. Please see the migration documentation for further details.

@asfimport
Copy link
Author

Weston Pace / @westonpace:
As far as I can tell Arrow does not rely on this operation internally. That being said, there is no documented guarantee I can find that the move must be atomic (and the move method in the Windows implementation of local fs is not truly atomic).

@asfimport
Copy link
Author

Antoine Pitrou / @pitrou:
Move need not be atomic, but it should probably be "cheap" (i.e. not imply streaming the whole file through the network). If that can't be done, then better to leave it unimplemented, IMHO.

See the S3 implementation:
https://github.com/apache/arrow/blob/master/cpp/src/arrow/filesystem/s3fs.cc#L2377-L2394

@asfimport
Copy link
Author

Carlos O'Ryan / @coryan:
For a single object (aka file) it is cheap, it just moves the bytes (and often just the metadata) on the server side.

For buckets is simply cannot be implemented, and for large directories, with many objects, it could be fairly slow (though we could parallelize it).

I have decided to implement the single object case and leave the rest alone.

@asfimport
Copy link
Author

Antoine Pitrou / @pitrou:
Issue resolved by pull request 11891
#11891

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant