Validate and Load all PDS4 MESSENGER data products with Nucleus

[jordanpadams]

💡 Description

• Document benchmark metrics for execution time, cost, ?

[tloubrieu-jpl]

@ramesh-maddegoda focuses on MSGRMDS_4001 and MESSDEM_1001 that need to be loaded in the registry first so that they can be used in ticket NASA-PDS/search-api-notebook#24

[tloubrieu-jpl]

Blocked because AWS Airflow is unavailable on NGAP

[tloubrieu-jpl]

Unblocked since Ramesh work on MCP. He is now testing the ECS task called by the nucleus workflow.

[jordanpadams]

Status: @ramesh-maddegoda working on improving Terraform deployments

[tloubrieu-jpl]

@ramesh-maddegoda is deploying everything needed on MCP, from scratch.

[tloubrieu-jpl]

@ramesh-maddegoda will test nucleus to validate its robustness with a bigger dataset.

[ramesh-maddegoda]

Some of the files in the s3://asc-pds-messenger failed to copy to the PDS Nucleus staging bucket with a permission issue.

aws s3 cp s3://asc-pds-messenger/MSGRMDS_8001/RTM/MDIS_RTM_N01/2013_228/MDIS_RTM_N01_006974_4644396_1.IMG s3://pds-nucleus-staging/messenger-data/MSGRMDS_8001/RTM/MDIS_RTM_N01/2013_228/

copy failed: s3://asc-pds-messenger/MSGRMDS_8001/RTM/MDIS_RTM_N01/2013_228/MDIS_RTM_N01_006974_4644396_1.IMG to s3://pds-nucleus-staging/messenger-data/MSGRMDS_8001/RTM/MDIS_RTM_N01/2013_228/MDIS_RTM_N01_006974_4644396_1.IMG An error occurred (AccessDenied) when calling the GetObjectTagging operation: Access Denied

[tloubrieu-jpl]

A new parameter enable to copy all the metadata .

[tloubrieu-jpl]

@ramesh-maddegoda identified a bug while doing that test. The lambda reading the data sync report is now taking more that 15 minutes. Now there will be a single lambda call per report.

[tloubrieu-jpl]

The upgrade worked on a small dataset and @ramesh-maddegoda is now testing on the messenger dataset.

[tloubrieu-jpl]

Now Ramesh is loading data to the registry on JPL AWS. Last step for this task.

[nutjob4life]

20,000 processed! 8 directories ran! Found 2 errors:

• "Capacity not available at the moment", related to ECS, seems to have run out; need to have "capacity providers" but cost is a concern; see related bug in Nucleus
• Role error: have seen it before, error occured by reading data file. An error in validate‽ Trying to isolate it with the same Docker image and reproduce it without Nucleus. Is it Airflow or validate?

[tloubrieu-jpl]

• capacity issue is solved with a retry configuration.
• validate error is a bug in validate: Exception when trying to handle missing_constant == Infinity validate#844

[tloubrieu-jpl]

@ramesh-maddegoda is experimenting with SQS to send to new records to the mysql database and avoid the time out he was experiencing with direct insertion.

[tloubrieu-jpl]

SQS now mostly works, but another lambda had a time out.

[tloubrieu-jpl]

We now integrate the copy from S3 to EFS as a nucleus step in the DAGs. We give up DataSync which comes with risks of overlapping copies and complication to remove files from EFS.

@ramesh-maddegoda will also write a note in a wiki for a future design where we don't need to use EFS at all.

[jordanpadams]

This work has been paused as we focus on Catalina Sky Survey. Will move to B15.0 release plan to complete work.

[jordanpadams]

📆 05/2024 status: Delayed several sprints due to delays in #93. This is an operations activity. No impact on build.

[jordanpadams]

📆 06/2024 status: Delayed several sprints due to delays in #93. This is an operations activity. No impact on build.