Add Google Cloud Storage support #237

Closed
wants to merge 4 commits into
from

Projects

None yet

4 participants

@deverant
Contributor

@fdr, let me know if I should squash this somehow. For now I wanted to keep the work done by different parties separate and have each commit fix only one thing.

This fixes #211.

mattupstate and others added some commits Aug 13, 2015
@mattupstate mattupstate Add Google Cloud Storage support
Original version by Matt Wright.

Heavily modified (to use the gcloud driver and gce conventions) by
Daniel Farina.

Errata: I am quite sure gs instance metadata is not working in this
patch.

Also, for whatever reason, I had a hard time getting blob
upload/download to work, so everything is done via signed URLs.
ee796c4
@deverant deverant Add thread safe Http client for google client
Since httplib2 is not thread safe we encapsulate the Http object so
that every request() call goes to a thread local Http instance.

This requires some funky calls to the internals for gcloud connection
instance to initialize the credentials correctly. There might be a
cleaner way to do it.
6b0492c
@deverant deverant Fix incorrect call to utcnow() 601dce6
@deverant deverant Add documentation about GCLOUD_PROJECT
This environment variable is needed for the GCS implementation to
work. Looks like the google api library needs it.
d3bcef1
@fdr
Member
fdr commented Apr 13, 2016

Thanks, that's helpful, I'll probably squash in the end but it's useful for review for me to see your deltas.

I will try to make a block of time over the next couple of weeks to apply this. It'll probably indicate a new release, probably WAL-E v1.0.

@fdr
Member
fdr commented Apr 29, 2016 edited

Okay, one question: do you use this/can you test this with GCE instance metadata, defined creds, or both? That's one errata in the patch that doesn't have clear resolution. Otherwise, I suppose I can remove mention of it for another interested party to fix.

@fdr
Member
fdr commented Apr 29, 2016

Another question: how would you feel about batching this with an upgrade to Python3?

I was waiting for Gevent to support Python3, and it is the last of my dependencies to do so: the time is finally here. But seeing as you pushed this over the finish line, I wanted to make a release that would suit your needs, all else being close to equal.

@deverant
Contributor

Hey,

Sorry for the late reply.

I don't currently run this on a GCE instance so I couldn't answer that. I might have a need later on to look at that but not right now. So feel free to remove mention of it for now.

I can have a look if I can reliably test this using Python3. Wouldn't mind switching to be honest.

@fdr
Member
fdr commented May 10, 2016

On Tue, May 10, 2016 at 1:59 AM Samuel Kohonen notifications@github.com
wrote:

Hey,

Sorry for the late reply.

I don't currently run this on a GCE instance so I couldn't answer that. I
might have a need later on to look at that but not right now. So feel free
to remove mention of it for now.

I can have a look if I can reliably test this using Python3. Wouldn't mind
switching to be honest.

Alright. I've actually done a couple of false starts on porting
WAL-E...it's mostly 2to3 compatible. I think it's tractable, though the
diff is large.

@deverant
Contributor

Hey!

Sorry, didn't really get too far with this. Trying to 2to3 the project wasn't as simple as I had hoped. Is there a chance that we can merge this already now? And when you have a branch somewhere with most of python3 work done for the core parts I am more than happy to test and make sure the GCS parts work and provide any fixes to that part of the codebase as needed.

@fdr
Member
fdr commented Jul 8, 2016

Yeah, I am going to merge this. I do have a bunch of python3 work in https://github.com/fdr/wal-e/tree/2to3wip

@fdr
Member
fdr commented Jul 12, 2016

Committed in c82114e though there is a problem with one integration test I'm trying to figure out...

@fdr fdr closed this Jul 12, 2016
@deverant
Contributor

Is the problem with the GCS tests? If so I can checkout your branch tomorrow and have a look at it.

@fdr
Member
fdr commented Jul 12, 2016

On Tue, Jul 12, 2016 at 4:10 PM Samuel Kohonen notifications@github.com
wrote:

Is the problem with the GCS tests? If so I can checkout your branch
tomorrow and have a look at it.

I actually found the bug: a parameter inversion. Steaming along, now. I'll
commit the fix shortly.

@deverant deverant deleted the deverant:gcs_support branch Jul 13, 2016
@TamalSaha

Thanks @deverant and @fdr for adding this feature. Can we get a new release of WAL-e so that we can install this with needing to use master directly?

@fdr
Member
fdr commented Jul 26, 2016

I'd like to do that, but the gcloud dependency has been giving me periodic trouble: GoogleCloudPlatform/google-cloud-python#2013, GoogleCloudPlatform/google-cloud-python#1979

I think I'm going to have to implement optional backends. Upstreams, save Azure and AWS, have packaging that is too unreliable to punish everyone for.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment