Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cloud ObjectStore #4487

Merged
merged 30 commits into from
Dec 12, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
0516279
Added CloudBridge to the requirements list.
VJalili Jul 21, 2017
aaad001
Added CloudBridge to the dependencies check.
VJalili Jul 21, 2017
55c976c
(1) Added Cloud to ObjectStore; (2) updated CloudBridge dependencies.
VJalili Jul 22, 2017
4a7045c
Merge remote-tracking branch 'origin/release_17.05'
mvdbeek Jul 24, 2017
e4c00dd
- Updated `Cloud` to account for changes in `CloudBridge` interface.
VJalili Jul 26, 2017
ab34fc9
Updated `Babel` and `requests` packages version to meet the minimum
VJalili Jul 26, 2017
e8541a7
Reverted the changes on package versions, as these changes are
VJalili Jul 26, 2017
cde2ff8
Merge remote-tracking branch 'upstream/master'
VJalili Jul 27, 2017
7ad7d88
Changed boto import style.
VJalili Jul 28, 2017
9c57adb
Updated CloudBridge to its current latest version.
VJalili Aug 2, 2017
2589c74
Commented on boto import, and removed a `use_reduced_redundancy` config.
VJalili Aug 2, 2017
89be432
Removed boto import, and replaced all S3-specific exception catches with
VJalili Aug 2, 2017
fafb019
Consolidated two upload methods, i.e., upload from file and string,
VJalili Aug 2, 2017
fb25495
Removed a S3-related comment, and swapped an if/else condition.
VJalili Aug 2, 2017
7727c0b
Removed a comment section.
VJalili Aug 2, 2017
37efbd0
Changed indentation of two lines.
VJalili Aug 7, 2017
9a5016f
Update CloudBridge to its current latest
VJalili Aug 17, 2017
4e17c8e
Add Cloud configuration to object_store_conf.xml.sample
VJalili Aug 18, 2017
0ec02b5
Merge remote-tracking branch 'remotes/upstream/dev' into CloudObjectS…
VJalili Aug 23, 2017
a878436
Removed white spaces.
VJalili Aug 23, 2017
eaa5931
Updated requirements: sqlalchemy-migrate and pbr.
VJalili Aug 23, 2017
90c444d
Removed a white space between parenthesis and arguments
VJalili Aug 24, 2017
f6586a8
Surrounded CloudBridge import in a try-catch block.
VJalili Aug 24, 2017
26cbf04
Rework conditional dependency handling in galaxy.objectstore.cloud.
jmchilton Sep 5, 2017
cf56d1d
Merge pull request #3 from jmchilton/CloudObjectStore
VJalili Sep 5, 2017
6d65e90
Updated an error message.
VJalili Sep 5, 2017
cfce6c2
Removed `boto` logging level set.
VJalili Sep 5, 2017
b555837
Removed retries for getting a bucket.
VJalili Sep 5, 2017
5977056
Resolved a bug with uploading a dataset using Cloud (not differentiating
VJalili Sep 12, 2017
45ac767
Removed an unused variable (i.e., mb_size) from Cloud
VJalili Sep 12, 2017
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions config/object_store_conf.xml.sample
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,16 @@
</object_store>
-->

<!-- Sample Cloud Object Store
<object_store type="cloud">
<auth access_key="..." secret_key="..." />
<bucket name="..." use_reduced_redundancy="False" />
<cache path="database/object_store_cache" size="1000" />
<extra_dir type="job_work" path="database/job_working_directory_s3"/>
<extra_dir type="temp" path="database/tmp_s3"/>
</object_store>
-->


</backends>
</object_store>
3 changes: 3 additions & 0 deletions lib/galaxy/dependencies/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,9 @@ def check_python_ldap(self):
def check_azure_storage(self):
return 'azure_blob' in self.object_stores

def check_cloudbridge(self):
return 'cloud' in self.object_stores

def check_kamaki(self):
return 'pithos' in self.object_stores

Expand Down
1 change: 1 addition & 0 deletions lib/galaxy/dependencies/conditional-requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ graphitesend
azure-storage==0.32.0
# PyRods not in PyPI
python-ldap==2.4.27
cloudbridge==0.3.3

# Synnefo / Pithos+ object store client
kamaki
4 changes: 2 additions & 2 deletions lib/galaxy/dependencies/pinned-requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -58,11 +58,11 @@ psutil==4.1.0
pulsar-galaxy-lib==0.7.0.dev5

# sqlalchemy-migrate and dependencies
sqlalchemy-migrate==0.10.0
sqlalchemy-migrate==0.11.0
decorator==4.0.2
Tempita==0.5.3dev
sqlparse==0.1.16
pbr==1.8.0
pbr==2.0.0

# svgwrite and dependencies
svgwrite==1.1.6
Expand Down
3 changes: 3 additions & 0 deletions lib/galaxy/objectstore/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -722,6 +722,9 @@ def build_object_store_from_config(config, fsmon=False, config_xml=None):
elif store == 's3':
from .s3 import S3ObjectStore
return S3ObjectStore(config=config, config_xml=config_xml)
elif store == 'cloud':
from .cloud import Cloud
return Cloud(config=config, config_xml=config_xml)
elif store == 'swift':
from .s3 import SwiftObjectStore
return SwiftObjectStore(config=config, config_xml=config_xml)
Expand Down
Loading