Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Synnefo/Pithos+ object store backend #3611

Merged
merged 5 commits into from Jul 11, 2017

Conversation

Projects
None yet
6 participants
@saxtouri
Copy link
Contributor

commented Feb 14, 2017

An implementation of the storage backend class (lib/galaxy/objectstore: ObjectStore) for Synnefo/Pithos+ object store ( https://www.synnefo.org/docs/synnefo/latest/pithos.html ) for Galaxy.

The implementation is using "kamaki", which is the Synnefo API client ( https://www.synnefo.org/docs/kamaki/latest/ ).

@galaxybot galaxybot added the triage label Feb 14, 2017

@galaxybot galaxybot added this to the 17.05 milestone Feb 14, 2017

@saxtouri saxtouri force-pushed the saxtouri:feature-pithos branch 2 times, most recently from 0fc12fd to f3608f6 Feb 15, 2017

@saxtouri

This comment has been minimized.

Copy link
Contributor Author

commented Feb 15, 2017

The tests do not fail on my system. It seems that the tests fail on travis on the "import kamaki" line, but I have added "kamaki" to "conditional-requirements.txt" and this seems to suffice on my tests.

Any suggestions on what went wrong?

@bgruening

This comment has been minimized.

Copy link
Member

commented Feb 16, 2017

@saxtouri I guess Galaxy needs a wheel to make this work. Can you create one with starforge? https://github.com/galaxyproject/starforge/blob/master/wheels/build/wheels.yml

martenson added a commit to galaxyproject/starforge that referenced this pull request Feb 20, 2017

@martenson martenson referenced this pull request Feb 20, 2017

Merged

build kamaki wheel #122

@martenson

This comment has been minimized.

Copy link
Member

commented Feb 20, 2017

The wheels for Kamaki are now available: https://depot.galaxyproject.org/starforge/wheels/build-165/

@@ -12,3 +12,6 @@ statsd
azure-storage==0.32.0
# PyRods not in PyPI
python-ldap==2.4.27

# Synnefo / Pithos+ object store client
kamaki

This comment has been minimized.

Copy link
@martenson

martenson Feb 20, 2017

Member

this also needs something like this:

def check_kamaki( self ):
        return 'pithos' in self.object_stores

around here

This comment has been minimized.

Copy link
@saxtouri

saxtouri Feb 21, 2017

Author Contributor

Done! Thanks!

@martenson
Copy link
Member

left a comment

please see above

@saxtouri saxtouri force-pushed the saxtouri:feature-pithos branch from f3608f6 to 35ec2a8 Feb 21, 2017

@martenson

This comment has been minimized.

Copy link
Member

commented Feb 21, 2017

The failing test is due to our (lack of) architecture for testing conditional dependencies. I am trying to work it out.

@saxtouri saxtouri force-pushed the saxtouri:feature-pithos branch from 3af15f3 to 569fbc8 Mar 3, 2017

@saxtouri saxtouri force-pushed the saxtouri:feature-pithos branch from 569fbc8 to 51391b0 Mar 13, 2017

@martenson

This comment has been minimized.

Copy link
Member

commented Mar 14, 2017

From my standpoint LGTM. However someone more fluent in object store should have a look here before this gets merged.

@bgruening

This comment has been minimized.

Copy link
Member

commented Apr 13, 2017

@dannon @natefoo would be great to get your input here :)
@saxtouri thanks, great work!

@martenson martenson modified the milestones: 17.09, 17.05 Apr 26, 2017

@dannon

This comment has been minimized.

Copy link
Member

commented May 1, 2017

@saxtouri This looks great to me, but prior to incorporation, would you mind dropping the copyright statement? It'd be fine to change it to an attribution statement/comment if you'd like (so, something like "This work originally developed by <people> at <institution>"), and add yourself to https://github.com/galaxyproject/galaxy/blob/dev/CONTRIBUTORS.md, but code contributions should fall under the general project copyright and licensing terms.

(and, apologies for not thinking about that bit until I was about to merge!)

@jmchilton

This comment has been minimized.

Copy link
Member

commented Jul 5, 2017

Ping @saxtouri again - is that copyright request reasonable?

@saxtouri

This comment has been minimized.

Copy link
Contributor Author

commented Jul 6, 2017

Sorry for the delay, of course it is reasonable, I'm updating this today.

saxtouri added some commits Dec 29, 2016

Initialize PithosObjectStore (Pithos backend)
Introduce galay.objectstore.pithos module
Create the PithosObjectStore class
Create auxiliary methods to:
- define/parse configuration for Pithos+
- authenticate with Synnefo
- initialize a Pithos+ client
Implement "exists" and "_construct_path" methods
The "_construct_path" constructs a path from an object and will also
be used in methods to follow.
Implement some cache-based methods + basic cache
Methods: exists, create, size, delete, get_data, get_filename
Also, implemented basic cache functionality: _get_cache_path,
    _in_cache, _fix_permissions, _pull_into_cache
Did not implement a "_pull_from_cache" method, since kamaki offers
    simple commands for this.
Implement methods for sync, share and quotas
Namely: update_from_file, get_object_url, get_store_usage_percent

@saxtouri saxtouri force-pushed the saxtouri:feature-pithos branch from 51391b0 to c9966df Jul 11, 2017

@saxtouri

This comment has been minimized.

Copy link
Contributor Author

commented Jul 11, 2017

OK, I removed the copyright statement. I apologize for the delay. GRNET didn't really have a problem with this, the statement was put there by scripts I use.

@dannon

This comment has been minimized.

Copy link
Member

commented Jul 11, 2017

Thank you @saxtouri! I'll run some final tests and merge this in a second.

@dannon dannon merged commit 45288cc into galaxyproject:dev Jul 11, 2017

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
@martenson

This comment has been minimized.

Copy link
Member

commented Jul 12, 2017

Thanks @saxtouri , very nice contribution!

mvdbeek added a commit to mvdbeek/starforge that referenced this pull request Jul 30, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.