[registry-harvest] Start testing new scalable Harvest Tools

[jordanpadams]

🟢 Issues Ready For I&T

• Parent theme: B12.1 Registry Tools Performance Improvements nasa-pds-engineering-node/pds-registry-app#184

For testing, will most likely want to start with nasa-pds-engineering-node/pds-registry-app#200 to spin up all the components using Docker.

3 new repos:

• https://github.com/NASA-PDS/registry-crawler-service
• https://github.com/NASA-PDS/registry-harvest-cli
• https://github.com/NASA-PDS/registry-harvest-server

Not sure where usage documentation resides? will need to follow-up with @tloubrieu-jpl and @tdddblog

NOTE: Most, if not all requirements that have been verified for Registry ingestion functionality should also be verified with these tools. Those requirements should be available through github:

• https://github.com/NASA-PDS/pds-registry-app/issues?q=is%3Aissue+is%3Aclosed+label%3Arequirement
• https://github.com/NASA-PDS/harvest/issues?q=is%3Aissue+is%3Aclosed+label%3Arequirement

@tloubrieu-jpl and @jordanpadams can help try to pull out requirements specific to ingestion if needed, but we just want to be sure to support the obvious ones at minimum

📜 Test Plan / Procedure

⚙️ Automated Tests

[gxtchen]

I have tested nasa-pds-engineering-node/pds-registry-app#200 and posted the test results in testrail, https://cae-testrail.jpl.nasa.gov/testrail/index.php?/runs/view/14268&group_by=cases:section_id&group_order=asc
Noticed the registry api's default return format is set to be csv, is this the decision made? I was expecting pds4/json.

[viviant100]

@tloubrieu-jpl @jordanpadams see above

[tloubrieu-jpl]

Thanks @gxtchen , you are right, I created a ticket as a 'nice to have' to have application/json or application/pds4+json as the default mime type in the swagger-ui (NASA-PDS/registry-api#88)