google cloud storage support #3021
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
|
@volth That looks like a pretty massive dependency (and it pulls in Java). Probably not something we want to depend on in Nix. As an aside we probably should move S3/GCS support into separate plugins to reduce Nix's closure size. |
edolstra
reviewed
Aug 12, 2019
|
|
||
| namespace nix { | ||
|
|
||
| class GCSBinaryCacheStore : public BinaryCacheStore |
Member
There was a problem hiding this comment.
This class and header file don't appear to be used anywhere except as a base for GCSBinaryCacheStoreImpl, so you could just get rid of them.
Member
|
@andir Can you make a PR to add |
Member
Author
|
Yes, will do during the day. I also have a few WIP changes locally that
handle large (slow) file uploads. I'll push them as well.
…On Mon, 12 Aug 2019, 12:16 Eelco Dolstra, ***@***.***> wrote:
@andir <https://github.com/andir> Can you make a PR to add
google-cloud-cpp to Nixpkgs?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3021?email_source=notifications&email_token=AAE365HUQPPK2EX24NVANU3QEEZ6PA5CNFSM4IIXKKL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4CDHFY#issuecomment-520369047>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAE365BYDHA2SO3RFHUVILDQEEZ6PANCNFSM4IIXKKLQ>
.
|
10 tasks
Member
Author
|
@edolstra I added those remarks, rebased onto master and added a bit of rough documentation. Let me know if there is anything else that has to be done. |
flokli
reviewed
Aug 29, 2019
Contributor
flokli
left a comment
flokli
left a comment
There was a problem hiding this comment.
Some comments.
We probably also want to add a doc/manual/packages/gcs-substituter.xml, and a line to doc/manual/release-notes/….
| { | ||
| PathSet paths; | ||
|
|
||
| // FIXME: is this really needed for binary caches? |
Contributor
There was a problem hiding this comment.
This should probably be replied before merging.
Contributor
|
Really excited to see this! I'll give the current code a test run on some GCS-based bucket later this week. |
flokli
reviewed
Aug 29, 2019
| <para>Nix has built-in support for storing and fetching store paths | ||
| from Google Cloud Storage.</para> | ||
| <para>For a GCS bucket with the name <literal>example-nix-cache</literal> the URI is <uri>gs://example-nix-cache</uri>.</para> | ||
| <para>Nix will use the environment variable <literal>GOOGLE_APPLICATION_CREDENTIALS</literal> to discover the credentials JSON file.</para> |
Contributor
There was a problem hiding this comment.
We should probably write we do authentication as documented at https://cloud.google.com/storage/docs/authentication#libauth - which also includes automatic credential management in Google App Engine and Google Cloud engine via their metadata server.
GOOGLE_APPLICATION_CREDENTIALS is another way to do it, but this reads as if it's the only way to do it. I'd still keep the example with GOOGLE_APPLICATION_CREDENTIALS however.
zimbatm
reviewed
Aug 29, 2019
zimbatm
reviewed
Aug 29, 2019
Member
zimbatm
left a comment
zimbatm
left a comment
There was a problem hiding this comment.
-
nix copy --to gs://bucketnameis working fine -
nix-build --substituters gs://bucketnameisn't supported
Member
Author
Is that a requirement? The way that is currently implement for S3 looks like an ugly hack. |
Member
|
I don't think it's a requirement. I was assuming that registering the store would make it work everywhere so I was surprised when it didn't work. Looking at the S3 code I see that it's not that clean. |
Contributor
|
I gave this a quick run on a CI pipeline, using the The derivations are 1.8MiB and 700KiB in size, and that's running from a GCP instance, so I highly doubt too slow transfer speeds. However, I'm not sure what else can be the issue here… |
Contributor
|
I noted a lot of CPU usage during However, after some tracing, this seems to be due to extensive polling inside google-cloud-sdk-cpp while uploading to GCS: I did a snapshot backtrace while |
This was referenced Oct 5, 2019
Contributor
|
With #2733 fixed by bumping to curl 7.68.0, this might be worth another try. |
Contributor
|
I propose closing this - this has bitrotten a bit, isn't actively being worked on, and @edolstra mentioned both the S3 and GCS support should eventually be moved into separate plugins. |
Member
|
Yes, though I think it would make sense to keep them in the |
Contributor
|
@edolstra can this be closed? |
flokli
added a commit
to NixOS/nixpkgs
that referenced
this pull request
Nov 11, 2020
This was only to enable NixOS/nix#3021, which has bitrotted sufficiently in the meantime.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I started adding support for Google Cloud Storage (GCS) to Nix.
This is very similar to the AWS S3 support. Unlike AWS there is no default path for the credentials. The environment variable
GOOGLE_APPLICATION_CREDENTIALSmust point to a standard Google credentials JSON file. The store URI must have the formatgs://bucketName.I have not thoroughly tested everything just yet. It seems to work fine so far. I uploaded a bunch of larger system closures (11-20GB) and some pandoc closures without problems.
Initially I did run into timeout issues but eventually switched to uploading larger files in chunks (37297df ).
This work depends on yet to be merged
nixpkgschanges that introduce the required dependencies. (https://github.com/andir/nixpkgs/tree/google-cloud-cpp).I will continue work on this during the coming days and welcome early feedback.
This change is