Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/issue 7 - Remove wait for GITC response #30

Merged
merged 67 commits into from
Jun 17, 2024
Merged

Feature/issue 7 - Remove wait for GITC response #30

merged 67 commits into from
Jun 17, 2024

Conversation

torimcd
Copy link
Contributor

@torimcd torimcd commented Jun 5, 2024

Github Issue: closes #7

Description

Waiting for the GITC response currently throttles the number of image sets we can send. Change to remove the wait period and complete successfully when we complete the send. Handle the GITC response outside of the state machine.

Overview of work done

  • remove the wait for task token during the sendToGITC lambda
  • change to using the image name + granule concept id for the CNM identifier
  • save CNM message sent to GITC in audit bucket with granuleUR + submission time as filename
  • move the umm-json lookup to the utils module so it can be reused out of state machine
  • handle the GITC response separately by moving to a regular lambda (not a cumulus process) and save the CNM-R in the same audit location with same filename (+cnm-r.json) to enable easy reconciliation

Overview of verification done

No new unit tests, existing tests pass

Overview of integration done

Deployed to SIT. Workflow succeeds when invoked manually and both CNM & CNM-R messages are saved in the audit bucket under collection_var prefix

PR checklist:

  • Linted
  • Updated unit tests
  • Updated changelog
  • Integration testing

See Pull Request Review Checklist for pointers on reviewing this pull request

torimcd added 30 commits May 1, 2024 13:48
@torimcd
fix type in readme
28d710c
@torimcd
change task token to uuid in send to gitc
f398c3a
@torimcd
remove wait for task token & gitc response handler from state machine
1f9cf71
@torimcd
change gitc response handler to invoke save cma message
c587bb0
@torimcd
fix typo
42a8706
@torimcd
fix context parameter
53c6d1e
@torimcd
convert uuid to string
168013e
@torimcd
fix json typo
d264ee3
@torimcd
remove uuid lib to use stdlib uuid
fc48c27
@torimcd
change identifier to image set name instead of uuid
f3ddcae
@torimcd
remove uuid import
49f0af2
@torimcd
fix json formatting of cma invoke
191f76d
@torimcd
add granule concept ID to identifier
86492da
@torimcd
update image set name with granule conceptid
ca46a16
@torimcd
remove whitespace
5953f73
@torimcd
move get umm json to utils and save gitc outgoing cnm
2969c68
@torimcd
remove unused import
e84af9f
@torimcd
try setting cmr query env based on stage
da12a51
@torimcd
remove task token from tests
e4fb9d7
@torimcd
fix granule index
921e057
@torimcd
change how image set name defined
5f62390
@torimcd
add region to ssm client in utils
3865176
@torimcd
add ssm parameters to gitc lambdas
7f28ba9
@torimcd
change region reference
cf90d3d
@torimcd
update how granule name referenced in send to gitc
8ee95ee
@torimcd
remove unused cmr var
46a9395
@torimcd
fix case
8944b35
@torimcd
fix cnm parsing
bf86736
@torimcd
reformat save cnm to separate step
b146dcc
@torimcd
update tf vars for save cnm
ad8ed6e
@torimcd torimcd added the enhancement New feature or request label Jun 5, 2024
frankinspace
frankinspace requested changes Jun 5, 2024
View reviewed changes
Copy link
Member

@frankinspace frankinspace left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great, just a few comments.

bignbit/handle_gitc_response.py Outdated Show resolved Hide resolved
terraform/lambda_functions.tf Outdated Show resolved Hide resolved
bignbit/save_cma_message.py Outdated Show resolved Hide resolved
@torimcd torimcd requested a review from frankinspace June 5, 2024 16:09
@frankinspace frankinspace merged commit b95dbf3 into develop Jun 17, 2024
7 checks passed
@frankinspace frankinspace deleted the feature/issue-7 branch June 17, 2024 17:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@frankinspace frankinspace frankinspace approved these changes

@voxparcxls voxparcxls Awaiting requested review from voxparcxls

Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
Add support for OPERA_L3_DSWX-S1_V1
Development

Successfully merging this pull request may close these issues.

Remove the wait for GITC response
2 participants
@torimcd @frankinspace