Accessing an s3Credential endpoint that's not officially in EarthAccess package

[wildintellect]

We're trying to test out using EarthAccess to get to a special bucket at ASF for some simulated data that is unlikely to ever end up in CMR.

I couldn't find a way to pass the credential endpoint to EarthAccess, https://nisar.asf.alaska.edu/s3credentials it only wanted to let me use the built in options, this NISAR url is not the official distribution point for ASF since it's a different bucket than usual.

As an example, with direct code it can work:

token = ....
import requests
import boto3
import os

resp = requests.get(
    "https://nisar.asf.alaska.edu/s3credentials",
    headers={"app-name": "veda",
            "Authorization": f"Bearer {token}"}
    )
creds = resp.json()


boto3_session = boto3.Session(
            aws_access_key_id=creds['accessKeyId'],
            aws_secret_access_key=creds['secretAccessKey'],
            aws_session_token=creds['sessionToken']
    )

s3 = boto3_session.client('s3')

bucket='sds-n-cumulus-prod-nisar-sample-data'
# browsed with s3.list_objects(Bucket=bucket)
prefix = 'GCOV/ALOS1_Rosamond_20081012/NISAR_L2_PR_GCOV_001_005_A_219_4020_SHNA_A_20081012T060910_20081012T060926_P01101_F_N_J_001.h5'

# Download the file
download_path = os.path.basename(prefix)
s3.download_file(bucket, prefix, download_path)

cc: @betolink @jjfrench

[jhkennedy]

@wildintellect can you tell me what you're trying to do with EarthAccess here that you can't or don't want to do directly? Psudo code of what you'd like to do even if it doesn't work would be helpful

[betolink]

Hi @wildintellect I think what you are looking for would be doable with earthaccess, ideally this would be the code:

fs = earthaccess.get_s3fs_session(endpoint="https://nisar.asf.alaska.edu/s3credentials")

which is basically a wrapper of the code you posted, the Auth class accepts custom endpoints for credentials but this has not been surfaced to the top level methods since all the results earthaccess deals with are coming from CMR.

I tried testing the S3 crdential endpoint and ran into an SSL certificate issue with the subdomain. I think we could work on this and release have it ready for the next version, most likely before the end of the month.

[wildintellect]

fyi @chuckwondo @jjfrench this is what we need.

[jhkennedy]

@wildintellect @chuckwondo @jjfrench note the distribution endpoint for NISAR data has been moved to per NASA: https://nisar.asf.earthdatacloud.nasa.gov/

[chuckwondo]

Currently, this works with released versions of earthaccess (not sure what the earliest release is where this works, but certainly works for 0.9.0):

earthaccess.__store__.get_s3fs_session(endpoint="https://nisar.asf.earthdatacloud.nasa.gov/s3credentials")

However, I've created this issue: #602