Skip to content

Updates to user defined buckets #3102

Merged
AndrewPlayer3 merged 20 commits into
user-defined-bucketsfrom
user-bucket-permissions
May 20, 2026
Merged

Updates to user defined buckets #3102
AndrewPlayer3 merged 20 commits into
user-defined-bucketsfrom
user-bucket-permissions

Conversation

@jhkennedy
Copy link
Copy Markdown
Contributor

@jhkennedy jhkennedy commented May 19, 2026

I was able to deploy this branch to https://hyp3-jhk-sandbox.asf.alaska.edu/ and run a few burst jobs that write their files to a private bucket in the LAVAs account, as well as a couple autorift jobs that failed expectedly to check the failure path as well.

The example jobs:
https://hyp3-jhk-sandbox.asf.alaska.edu/jobs?name=test-user-bucket&user_id=jhkennedy

or

import hyp3_sdk as sdk

hyp3 = sdk.HyP3('https://hyp3-jhk-sandbox.asf.alaska.edu')
jobs = hyp3.find_jobs(name='test-user-bucket', user_id='jhkennedy')

This PR:

  • Add the bucket-policy endpoint to the OpenAPI spec
  • changes the bucken name example to "my-example-bucket" to make the bucket-policy endpoint example a little more clear
  • compacts the returned bucket policy and returns it as a dict->JSON instead of string
  • fixes the sts.get_caller_identity() usage for JSON output from AWS APIs
  • Compacts and updates the permissions of:
    • the batch compute task for external s3 buckets
    • get-files so it can get files from external buckets
    • upload-logs so it can get logs from external buckets
  • the step function to provide job_id, bucket, and bucket_prefix to get-files and upload-logs so we don't have to make a dynamo call
  • updates get-files to handle user bucket download URLs and missing expiration times (user's may not have a lifecycle policy on their buckets)
  • upload-log to always write the log file as {job_id}.log irrespective of the bucket and bucket_prefix provided
  • make sure every job spec is using Ref::bucket Ref::bucket_prefix instead of ! Ref bucket and Ref::job_id

With these changes, I think we're ready to merge to develop and take it for a spin in a broader set of environments, particularly EDC.

@jhkennedy jhkennedy requested a review from AndrewPlayer3 May 19, 2026 19:28
@jhkennedy jhkennedy requested review from a team as code owners May 19, 2026 19:28
Comment thread requirements-apps-api.txt
@@ -1,4 +1,5 @@
boto3==1.42.97
pydantic==2.12.5
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was needed because a different version kept getting installed and uploaded, which pip was warning about...

I'd like to test removing all these after we merge to test, because we really shouldn't need to specify them.

@AndrewPlayer3 AndrewPlayer3 merged commit 1142386 into user-defined-buckets May 20, 2026
11 checks passed
@AndrewPlayer3 AndrewPlayer3 deleted the user-bucket-permissions branch May 20, 2026 17:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants