Create a registry docker for developer testing

[tloubrieu-jpl]

Since the registry is a central component of PDS system, many developers use it as an interface for their own development.

To spare them some time, we want to provide them with a docker image providing the elasticsearch data filled with some test data.

The docker image will:

• instantiate the elasticsearch server
• install and run harvest and registry-manager on test data as described on https://nasa-pds.github.io/pds-registry-app/install/index.html

The github repositories which are at stake are:

• https://github.com/NASA-PDS/pds-registry-app (the application package)
• https://github.com/NASA-PDS/harvest
• https://github.com/NASA-PDS/pds-registry-mgr-elastic

The docker resources will be manage in this pds-registry-app repository, in a specific directory (docker).

Acceptance criteria:

Instruction for running the registry in docker containers are provided in the pds-registry-app repostory README.

[al-niessner]

I am having problems with mvn linkcheck:linkcheck. Error is:

[INFO] Invoking the maven-site-plugin to ensure that all files are generated...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:07 min
[INFO] Finished at: 2021-02-04T19:13:01Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-linkcheck-plugin:1.2:linkcheck (default-cli) on project pds-registry-app: Execution default-cli of goal org.apache.maven.plugins:maven-linkcheck-plugin:1.2:linkcheck failed: An API incompatibility was encountered while executing org.apache.maven.plugins:maven-linkcheck-plugin:1.2:linkcheck: java.lang.ExceptionInInitializerError: null
[ERROR] -----------------------------------------------------

I have googled it and it is not obvious on how to make it available like mvn install linkcheck but rather some changes to the POM. Therefore, I need to ask should I install it or is the README.md in this repo out of date?

[tloubrieu-jpl]

@al-niessner this step is not necessary for the build you are working on.

linkcheck is to check links in the mvn site that we publish on github. It is known to only work with jdk1.8 unlike the rest of the registry which works with jdk11, so you can give up this step in the docker image.

Some information on that is available in the readme is your scroll right:
mvn linkcheck:linkcheck ## check the link in the maven site and create a report, this requires jdk1.8 to work, later version fail.

[tloubrieu-jpl]

The application is going to be composed of 2 docker images (registry tools + elastic search)

[al-niessner]

Where can I find a block of test data for the harvest command?

The paragraphs above suggest using some PDS records, but does not detail the commands to do it.

I have the archive directory set up but nothing in it and got this result:

harvest -c /var/local/harvest/conf/examples/registry.xml -o ~/tmp
4[INFO] Output directory: /root/tmp
[INFO] Output format: json
[INFO] Reading configuration from /var/local/harvest/conf/examples/registry.xml
[INFO] Summary:
[INFO] Skipped files: 0
[INFO] Processed files: 0

which is when I realized I need some data. It seems like there may be a block of it somewhere (cannot find it into the harvest-3.3.3 release).

[jordanpadams]

@tloubrieu-jpl @tdddblog can we point @al-niessner to the directory for the data. I believe it is in your home directory @tdddblog ?

Btw, I would really like this moved to some.consistent location for test data for all our software, if at all possible. If we need more.diak space or whatever, let's just make it happen so everyone always knows where to go for PDS test data so we can document appropriately.

[al-niessner]

I can add it to the docker image if it does not change that much. At the least, it will be a default test data set that the user can then override with their own.

…

On Sat, Feb 6, 2021 at 12:03 AM Jordan Padams ***@***.***> wrote: @tloubrieu-jpl <https://github.com/tloubrieu-jpl> @tdddblog <https://github.com/tdddblog> can we point @al-niessner <https://github.com/al-niessner> to the directory for the data. I believe it is in your home directory @tdddblog <https://github.com/tdddblog> ? Btw, I would really like this moved to some.consistent location for test data for all our software, if at all possible. If we need more.diak space or whatever, let's just make it happen so everyone always knows where to go for PDS test data so we can document appropriately. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#129 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIUBIWOD2C4CGOHWKAGVKDS5TZUTANCNFSM4WYAKP5Q> .

[tloubrieu-jpl]

@al-niessner @jordanpadams the pds-registry-app package pulls a small dataset from https://pds.nasa.gov/datastandards/documents/examples/ when you run mvn package.

The data is not in the github source, it is pulled in a few minutes when the package is created. I believe this mvn package should be part of the docker image.

If we want a bigger dataset we can work with the one in Eugene home directory. The original datasets are published online. These datasets (insight cameras, orex) are much bigger than the test dataset so it takes longer to download and ingest in the registry (days I think)