As a manager, I want the registry to have regression tests included in CI

[jordanpadams]

Motivation

...so that I can ensure our software product is accurate and robust.

Additional Details

Acceptance Criteria

Given a feature request or bug identified in the registry-mgr
When I perform an update to the code and merge a pull request
Then I expect continuous integration to execute with regression tests to catch if the fix breaks an assumption.

Given a feature request or bug identified in harvest
When I perform an update to the code and merge a pull request
Then I expect continuous integration to execute with regression tests to catch if the fix breaks an assumption.

Engineering Details

This does not need to include a huge swath of regression tests, but it should at least include the base infrastructure and orchestration necessary to add regression tests over time.

• Maybe we pull out the docker install info from https://github.com/NASA-PDS/registry-api-service into it's own "registry + API" test repo that this can utilize?
• Should POSTMAN or cucumber be used?

here are some details on how to spin up the Docker container: https://github.com/NASA-PDS/pds-registry-app#docker

there is also another Docker container here: https://github.com/NASA-PDS/registry-api-service/#docker

there are also a couple test scripts generated by @al-niessner for the API as well: https://github.com/NASA-PDS/registry-api-service/tree/master/verify

Tasks

• Create Docker image to build Registry components: Harvest, Registry Manager, API
• Find out how to run Elasticsearch from a GitHub action
• Create Docker image to harvest and load test data into Elasticsearch
• Create Docker image to install and run Registry API server
• Create Docker image to test Registry API
• Create main CI GitHub action

[al-niessner]

Since you stuck my name on this, the scripts that I developed are actually the acceptance criteria translated from the ticket to python in my case.

One item that takes a ticket much longer to close is that the acceptance criteria drifts when the ticket enters review. Case in point is pds-api 54. While the changes matched the acceptance criteria of the ticket, the reviewer applied further tests beyond what was in the acceptance. In doing so, found some extra bugs. From a process perspective, should these items have first been in the acceptance criteria or should it have generated another ticket with the new criteria being its acceptance to fix the bugs? If the desire is to limit the life of any ticket, then need to have fixed criteria and new criteria should generate a new ticket. Otherwise, expect estimates to be way off because the acceptance criteria can change after the estimate has been made.

If acceptance criteria is encoded into a test (before or after a ticket is opened) then it can be used to ensure that functionality persists -- aka, regression testing.

[jordanpadams]

thanks @al-niessner good point.

@tdddblog i copied some design thoughts from @nutjob4life on how we want to implement here: https://github.com/NASA-PDS/nasa-pds.github.io/wiki/PDS-EN-CI:-A-Design-Memoir

[tloubrieu-jpl]

@tdddblog for the API integration test, I am maintaining a postman test suite that would be god to have called as part of the integration test. One version of it is available on https://raw.githubusercontent.com/NASA-PDS/registry-api-service/master/src/test/resources/postman_collection.json
It can be called in command line with a tool called newman: https://support.postman.com/hc/en-us/articles/115003710329-What-is-Newman-

I have an updated version for the latest version of the API, but it also needs to be updated depending on the test dataset that we have loaded (I am using the one loading in Al's containers).

I don't know if this should be maintained in the registry-api-service as it is now or rather in the pds-registry-app. I would lean toward the later since the test only make sense if the data is loaded in the elasticsearch.

[tloubrieu-jpl]

@eugene can start from the docker images that Al developed before for the second task in the list.