Dockerize API Service

[jordanpadams]

No description provided.

[tloubrieu-jpl]

Both service and UI need to be dockerized, unless there is a AWS deployment strategy which does not need that

[tloubrieu-jpl]

@collinss-jpl made progress on this ticket.

Question on how to manage secrets, @nutjob4life @al-niessner will have answers.

[nutjob4life]

Here's my approach to credentials etc. with a containerized application, but you can extend this to configuration in general:

First, clearly, Docker Secrets are the correct way to go, but oh my goodness what a mental strain they are, and then as you've identified you've got the Docker Swarm complexity to deal with. Add to that a tight deadline and it's just too much—an example of the cost exceeding the reward ratio.

On the other hand, you can
put your secrets into a file that's outside of the container and then mount the directory of that file as a volume in the container. That works, but presents some other hurdles:

• For security reasons, you may want to limit the volumes a container can access (in the case of an untrusted image), so you end up creating more complex subdirectory layouts just to hold the configuration.
• It may be unclear when the configuration is actually used: at container startup? Is the file scanned for modification times and dynamically loaded? That's additional documentation to both write and process.
• It requires the user to understand the syntax of the file; is it an with bracketed sections? If so does it support the macro-replacement extension? Is it actually an exectuable .tcl file? XML?
• A good "use pattern" being established with containers is to prefer volumes for persistence purposes only and to pass as much of configuration via other means. By adhering to this pattern your image is easier to adopt by others who are familiar with the pattern; the PostgreSQL and MySQL images do this for example.

Okay, so what's the other means?

First, for some more complex applications out there (like OpenLDAP server), the configuration is part of the persistence mechanism, so the app starts up with a reasonable set of defaults (and a mounted volume to store the configuration) and you use typical client tools (like ldapmodify for an LDAP server) to change the configuration. Same goes for Solr, except it includes its own "Solr control script" in the image:

$ docker container run --detach --publish 8983:8983 --name pds-solr-server solr:8  # Start Solr
$ docker container exec --interactive --tty pds-solr-server solr:8 create_core …  # Update its config

Alternatively, you might even just surf to http://localhost:8983 and do the configuration through the web.

Now, for the rest of us, environment variables to the rescue.

If you've got a containerized app that needs something like a username and password to talk to talk to a database or to provide privileged access to its own state, environment variables are directly supported by Docker and Docker Composition and just need a little bit of caution to use well.

You don't, of course, want to do this:

$ docker container run --detach --publish 8080:80 --name pds-demo-server --env DB_USERNAME=pds-db-user --env DB_PASSWORD=s3cr3t pds-image:latest  # ❌

since that puts the password in the process list which anyone can view with ps auxww. So, naturally, set those in the shell before executing docker container run.

But even better: use a Docker Composition:

services:
    db:
        image: postgres:12.4
        volumes: …
        ports: …
        networks: …
        environment:
            # Empty value here inherits from host's env:
            POSTGRES_PASSWORD:
    pds-demo-service:
        image: pds-image:latest
        ports: …
        networks: …
        environment:
            # Empty value here inherits from host's env:
            POSTGRES_PASSWORD:
networks: …
version: …

Now you can set the password for both the database and the service without having to repeat yourself. You may need to make sure the ENTRYPOINT for pds-image knows to look for POSTGRES_PASSWORD (or map it to another name in the yaml) or whatever and set things up appropriately, of course—but you may have some other startup things you're doing anyway. Don't like setting the environment in the shell? Use the .env file or the env_file: setting.

There's one other alternative I've seen that looks pretty interesting: it's the meta container approach. The ESGF does this with their "Compute WPS" and other projects, and I've seen it someplace else (forget where right now). Essentially it looks like this (in this hypothetical example):

$ alias ls=ls -F
$ pwd
/home/kelly
$ mkdir pds-demo
$ cd pds-demo
$ ls
$ curl --location --remote-name --silent https://pds.nasa.gov/turnkey/pds-demo-startup.sh
$ ls
pds-demo-startup.sh
$ cat pds-demo-startup.sh
#!/bin/sh
docker container run --rm --interactive --tty --volume ${PWD}:/pds pds-demo-setup:1.2.3
exit 0
$ sh pds-demo-startup.sh
🪐 PDS Demo Service 😅

Welcome to the PDS Demo Service setup. To accept the PDS defaults in brackets,
just hit Return (might be labeled Enter on some keyboards due to IBM changing
the conventional meaning of "carriage return" which itself had been the term
transferred over from typewriters into printing and video terminals and using
"enter" instead).

First, we need the PDS username [pds]: ⮐
Next the PDS password: s3cret⮐
Now the OSTI endpoint [http://osti.whatever/end/point]: ⮐
Mother's maiden name [Steinbeck]: Richardson⮐
Last four digits of SSN [4196]: ⮐
First three digits of SSN: 192⮐
Middle two digits of SSN [33]: 34⮐

Writing PDS configuration…done!
If you need to do this PDS setup again, just run bin/setup
To start it: bin/startup

$ ls
README.txt   bin/   docs/    etc/    var/
$ echo oh wow it set up the entire directory and even cleaned up the bootstrap .sh file
oh wow it set up the entire directory and even cleaned up the bootstrap .sh file
$ ls bin
setup*      start*     status*    stop*
$ ls etc
base.cfg   certs/   crontab.cfg   credentials.cfg   docker-compose.yaml    logging.cfg    versions/
$ ls var
hsql/   mongodb/   postgresql/
$ du -sh var
421G var
$ echo there is even a ton of data
there is even a ton of data
$ cat bin/startup
#!/bin/sh
docker-compose --project-name pds-demo --file /home/kelly/pds-demo/etc/docker-compose.yaml up --detach
$ echo okay looks good lets get it going
okay looks good lets get it going
$ bin/start

[gxtchen]

@jordanpadams Hi Jordan, where can I find the document for this task?

[jordanpadams]

@gxtchen https://github.com/NASA-PDS/pds-doi-service#running-with-docker . But @tloubrieu-jpl can correct me if I'm wrong

[jordanpadams]

@gxtchen you can also check out the pull request for more details. #185

[tloubrieu-jpl]

@gxtchen I confirm you should use the section https://github.com/NASA-PDS/pds-doi-service#running-with-docker for the docker deployment. We would like to improve that but as it is for build 12.0, that is the resource to be used. Thanks