Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Google cloud storage v1 #1

Merged
merged 4 commits into from
Mar 27, 2018
Merged

Google cloud storage v1 #1

merged 4 commits into from
Mar 27, 2018

Conversation

Zatte
Copy link
Owner

@Zatte Zatte commented Mar 25, 2018

A patch to add google cloud storage support to Afero
using GCS strong consistency guarantees to implement almost all afero intefaces.

V1 scope limitations:

  • No Chmod support - The GCS ACL could probably be mapped to *nix style permissions but that would add another level of complexity and is ignored in this version.
  • No Chtimes support - Could be simulated with attributes (gcs a/m-times are set implicitly) but that's is left for another version.
  • NOTE: Not thread safe - Also assumes all file operations are done through the same instance of the GcsFs. File operations between different GcsFs instances are not guaranteed to be consistent.
  • File modes will always bee 0755/0664 for folders/files when read back

Performance implications

  • Sequential reads are performant
  • Sequential writes are performant.
  • Seek + Read or ReadAt is performant after the initial seek.
  • Alternating reads/writes to the same file handler are highly inefficient. To get consistent FS behavior using an API that separates readers and writers we close any open readers before an write as well close open writers before a read (ensure the data is committed).
  • Seek + Write such as WriteAt, Truncate, Seek+Write will work as expected but with significant overhead. Doing a seek + write will in effect download the old file/object, overlay it with the new writes and save it back. This is done in a streaming fashion so large files will not clog the memory but will trigger a full download and upload of the file/object.

How to test:

  • Ensure you have a service account setup and $GOOGLE_APPLICATION_CREDENTIALS pointing to that file. Other methods for creating a client are supported by manually creating the gcloud client and using NewGcsFs(...) instead of NewGcsFsFromDefaultCredentials(...) as below
  • Update afero_test.go with read
var gcsFs = NewGcsFsFromDefaultCredentials(context.Background(), "[GCS_BUCKET_USED_FOR_TESTING]", string(filepath.Separator))
var Fss = []Fs{&MemMapFs{}, &OsFs{}, gcsFs }

Copy link

@cvik cvik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ser väl helt ok ut, vid en snabb genomläsning. Har inte testat eller verifierat någon funktionalitet, utan mest kikat på de estetiska bitarna etc. Så...
LGTM!

@@ -0,0 +1,86 @@
// Copyright © 2018 Mikael Rapp, github.com/zatte
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Om det ska mergas in i spf13/afero, så vill han kanske inte att du petar in din egen Copyright ovan.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tänker han får själv avgöra ifall apache licence ger ju hon alla rättigheter han behöver samt tror det finns andra contributions med liknande wordings i repot redan

@Zatte
Copy link
Owner Author

Zatte commented Mar 27, 2018

Cool; stort tack för feedback! Kommer ta bort denna och skapa en ny mot origin/repot isf =)

@Zatte Zatte merged commit b737c2a into master Mar 27, 2018
@Zatte Zatte deleted the google-cloud-storage-v1 branch March 27, 2018 15:34
tamalsaha pushed a commit to gomodules/afero that referenced this pull request Aug 9, 2020
* WIP

* The turd passes all tests

* Truncate performance improvement
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants