NIFI-5438 volume support for the NiFi docker containers

[pepov]

Create all volume directories for chowning properly, then create proper volumes for persistence. Notes:

• Create conf instead of conf/templates since it will be created automatically.
• Create database_repository instead of data, since that is the default configuration.
• Create state volume used to store local state by default.

mvn integration-test -P docker ran fine from the nifi-docker folder against both modules

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

For all changes:

• Is there a JIRA ticket associated with this PR? Is it referenced
  in the commit message?

• Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

• Has your PR been rebased against the latest commit within the target branch (typically master)?

• Is your initial contribution a single, squashed commit?

For code changes:

• Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
• Have you written or updated unit tests to verify your changes?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
• If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
• If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

• Have you ensured that format looks appropriate for the output in which it is rendered?

Note:

Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

[MikeThomsen]

@pepov can you share some examples of how to test this?

[pepov]

@MikeThomsen sure! If you run the integrations tests you will have the following images available:

docker images | grep nifi | grep 1.8.0-SNAPSHOT
apache/nifi                                              1.8.0-SNAPSHOT-dockerhub     a670a54ee676        30 hours ago        1.74GB
apache/nifi                                              1.8.0-SNAPSHOT-dockermaven   0f1e3ae32879        30 hours ago        3GB

Now you run one of those, for example:

docker run -d --name nifi apache/nifi:1.8.0-SNAPSHOT-dockermaven

You can inspect the volumes it created using the following command:

docker inspect nifi | jq '.[].Mounts | map([.Source, .Destination])'
[
  [
    "/var/lib/docker/volumes/139a9e99e6542165f061923168944ca184a6340b330aec3aa20efe1f70fd22a3/_data",
    "/opt/nifi/nifi-1.8.0-SNAPSHOT/conf"
  ],
...

You won't be able to access that directory from your host on a Mac for example, but you can mount it and examine it using another docker container. Lets mount the whole /var/lib/docker folder and check one of the volumes:

docker run -ti --rm -v /var/lib/docker:/var/lib/docker debian bash -c "ls -al /var/lib/docker/volumes/139a9e99e6542165f061923168944ca184a6340b330aec3aa20efe1f70fd22a3/_data"
total 88
drwxr-xr-x 3 1000 1000  4096 Jul 19 15:20 .
drwxr-xr-x 3 root root  4096 Jul 19 15:19 ..
drwxr-xr-x 2 1000 1000  4096 Jul 19 15:20 archive
-rw-rw-r-- 1 1000 1000 20135 Jun  1 14:21 authorizers.xml
-rw-rw-r-- 1 1000 1000  2326 Jun  1 14:21 bootstrap-notification-services.xml
-rw-rw-r-- 1 1000 1000  3509 Jun  1 14:21 bootstrap.conf
-rw-r--r-- 1 1000 1000   271 Jul 19 15:20 flow.xml.gz
-rw-rw-r-- 1 1000 1000  8095 Jun  1 14:21 logback.xml
-rw-rw-r-- 1 1000 1000  6449 Jun  1 14:21 login-identity-providers.xml
-rw-rw-r-- 1 1000 1000 11114 Jul 19 15:19 nifi.properties
-rw-rw-r-- 1 1000 1000  8669 Jun  1 14:21 state-management.xml
-rw-rw-r-- 1 1000 1000  1437 Jun  1 14:21 zookeeper.properties

You can combine this with bind mounting a volume from your host system and you are then ready to make backups from a running or even from an already removed container for example.

After removing the nifi container forcefully:

docker rm -f nifi

You will still be able to list the volumes and mount it to an arbitrary container as in the previous example:

docker volume list
DRIVER              VOLUME NAME
local               11f81374f20c3b80234f5662898ac93eb7dcea9210c9c84e70782025351d9915
local               139a9e99e6542165f061923168944ca184a6340b330aec3aa20efe1f70fd22a3
local               182188a75419e8ed029b910ac5ca73675250675eac033056e11b0e1f9b7a5b84
local               3967c839cc69c373adc78efd3d1ad1d626ad068b2c7e82c09902167f243edcd1
local               8b021167e1d5b4b58d708b9c08019a6dd5a890d27bc72535085335cda5b28fc7
local               a80d1be309ddde9d76aa405642ca036df477e3bd0670e07d5de37f5521a92bd7
local               a95d23b02768fe8858937da94b41342052722589a9cf0a609f023c147eb1ee66
local               f63966beb26225b0fe146257c0d649a9a04d94b4fc2daedb192e63ca4cf0a28f

[apiri]

Looks good to me

[apiri]

hey @MikeThomsen,

Didn't know if you wanted to check this out a bit more. If not, I am happy to close it out and merge it in.

Thanks!

[MikeThomsen]

Go for it. I'm not very knowledgeable on the subject, so I'll defer.

[apiri]

Sounds good. Thanks for confirming.