Skip to content

Use CMR metadata to get S3FS sessions and authenticate queries to CMR using top level APIs#296

Merged
betolink merged 9 commits intomainfrom
wip-s3credentials
Sep 18, 2023
Merged

Use CMR metadata to get S3FS sessions and authenticate queries to CMR using top level APIs#296
betolink merged 9 commits intomainfrom
wip-s3credentials

Conversation

@betolink
Copy link
Contributor

@betolink betolink commented Sep 12, 2023

This PR allows earthacces.search_data() and earthaccess.search_dataset() use the auth object to send bearer tokens to CMR, usually this is not required but in the case of restricted datasets or early release data CMR will only return results if we are in the access control lists for a given dataset.

with this PR earthaccess will be able to use the metadata from CMR to grab S3 credentials, perhaps we need to simplify this even more but for now this works.

import earthaccess

earthaccess.login()

results = earthaccess.search_data(
    short_name='SWOT_L2_HR_RIVERSP_1.0',
    cloud_hosted=True,
    bounding_box=(-10, 20, 10, 50),
    temporal=("1999-02", "2019-03"),
    count=10
)

# now earthaccess will use that the S3 credentials if they are present in the metadata if not it will grab the DAAC's
earrthaccess.open(results)

This code was failing because the SWOT mission has its own S3 credentials that are not from PODAAC.

another way of getting the authenticated sessions would be:

if len(results) > 0:
    print(f"Actual granules returned: {len(results)}")    
    fs_s3 = earthaccess.get_s3fs_session(results=results)

And then we could use the storage_options to create authenticated sessions, this is the part that can be simplified:

import fiona
from fiona.session import AWSSession

with fiona.Env(
    session=AWSSession(
        aws_access_key_id=fs_s3.storage_options["key"],
        aws_secret_access_key=fs_s3.storage_options["secret"],
        aws_session_token=fs_s3.storage_options["token"]
    )
):
    # some code  with gpd.read_file() etc

@github-actions
Copy link

github-actions bot commented Sep 12, 2023

Binder 👈 Launch a binder notebook on this branch for commit a85f531

I will automatically update this comment whenever this PR is modified

Binder 👈 Launch a binder notebook on this branch for commit ca6590f

Binder 👈 Launch a binder notebook on this branch for commit c4321c0

@betolink betolink marked this pull request as ready for review September 12, 2023 18:45
@betolink betolink changed the title WIP: use CMR metadata to get S3FS sessions and authenticate queries to CMR using top level APIs Use CMR metadata to get S3FS sessions and authenticate queries to CMR using top level APIs Sep 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant