Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feat/cleversafe url #744

Merged
merged 18 commits into from
Mar 11, 2020
Merged

Feat/cleversafe url #744

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
74179b2
feat(cleversafe): allow specifying s3-compliant url for a bucket in c…
Avantol13 Dec 18, 2019
40c336a
rebuild
Dec 18, 2019
d88dcf7
Merge branch 'master' of github.com:uc-cdis/fence into feat/cleversaf…
Dec 18, 2019
81a1dea
Merge branch 'master' into feat/cleversafe-url
Avantol13 Dec 18, 2019
14acf99
fix(endpoint-url): fix getting the endpoint url
Dec 18, 2019
0c6a1d7
Merge branch 'feat/cleversafe-url' of github.com:uc-cdis/fence into f…
Dec 18, 2019
a0c4a39
fix(endpoint-url): fix getting the endpoint url
Dec 18, 2019
2671e14
fix(endpoint-url): fix getting the endpoint url
Jan 6, 2020
5b8f159
fix(endpoint-url): fix getting the endpoint url
Jan 6, 2020
a72bf6e
fix(endpoint-url): fix getting the endpoint url
Jan 6, 2020
27fd27d
Merge branch 'master' into feat/cleversafe-url
Avantol13 Jan 16, 2020
554a238
chore(tests): configure tests to have a bucket with a different s3 en…
Avantol13 Mar 5, 2020
246359a
Update fence/blueprints/data/indexd.py
Avantol13 Mar 10, 2020
fa3286b
Merge branch 'master' into feat/cleversafe-url
Avantol13 Mar 10, 2020
eacd20a
fix(indexd): use bucket name instead of bucket config for function
Avantol13 Mar 10, 2020
1062432
fix(index): fix var name
Avantol13 Mar 10, 2020
bbbbd05
feat(cfg): ensure we don't make unecessary call to get region when en…
Avantol13 Mar 10, 2020
7d78f17
Update fence/blueprints/data/indexd.py
paulineribeyre Mar 10, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Fence
# Fence

[![Build Status](https://travis-ci.org/uc-cdis/fence.svg?branch=master)](https://travis-ci.org/uc-cdis/fence)

Expand Down
19 changes: 15 additions & 4 deletions fence/blueprints/data/indexd.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -594,13 +594,24 @@ def get_signed_url(
aws_creds = get_value(
config, "AWS_CREDENTIALS", InternalError("credentials not configured")
)

http_url = "https://{}.s3.amazonaws.com/{}".format(
self.parsed_url.netloc, self.parsed_url.path.strip("/")
s3_buckets = get_value(
config, "S3_BUCKETS", InternalError("buckets not configured")
)

bucket = self.bucket_name()
current_bucket = s3_buckets.get(self.bucket_name())
Avantol13 marked this conversation as resolved.
Show resolved Hide resolved

if current_bucket and current_bucket.get("endpoint_url"):
http_url = current_bucket["endpoint_url"].strip("/") + "/{}/{}".format(
paulineribeyre marked this conversation as resolved.
Show resolved Hide resolved
self.parsed_url.netloc, self.parsed_url.path.strip("/")
)
else:
http_url = "https://{}.s3.amazonaws.com/{}".format(
self.parsed_url.netloc, self.parsed_url.path.strip("/")
)

credential = S3IndexedFileLocation.get_credential_to_access_bucket(
self.bucket_name(), aws_creds, expires_in
bucket, aws_creds, expires_in
)

# if it's public and we don't need to force the signed url, just return the raw
Expand Down
2 changes: 2 additions & 0 deletions fence/config-default.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -487,6 +487,8 @@ AWS_CREDENTIALS:
S3_BUCKETS:
bucket1:
cred: 'CRED1'
# optionally you can manually specify an s3-compliant endpoint for this bucket
endpoint_url: 'https://cleversafe.example.com/'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you still need a bucket region when using cleversafe?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it looks like the code is using the region, so i'm not sure how it's working in prod... maybe they configured it with a fake region field. if not, it would fail trying to get the region from aws

bucket2:
cred: 'CRED2'
region: 'us-east-1' #optional but if specified avoids a call to GetBucketLocation which you may lack the AWS ACLs for.
Expand Down
4 changes: 2 additions & 2 deletions tests/conftest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -461,8 +461,8 @@ def indexd_client(app, request):
"baseid": "",
"rev": "",
"size": 10,
"file_name": "file1",
"urls": ["s3://bucket1/key"],
"file_name": "file2",
"urls": ["s3://bucket2/key"],
"hashes": {},
"acl": ["phs000178", "phs000218"],
"form": "",
Expand Down
1 change: 1 addition & 0 deletions tests/test-fence-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -360,6 +360,7 @@ S3_BUCKETS:
cred: 'CRED1'
bucket2:
cred: 'CRED2'
endpoint_url: 'https://cleversafe.example.com/'
bucket3:
cred: 'CRED1'
bucket4:
Expand Down