Container restart fails when the admin user password has changed using Splunk WEB

[gibban69]

Steps to reproduce

1. docker run --name splunk -d -p 8000:8000 -e "SPLUNK_PASSWORD=helloworld" -e "SPLUNK_START_ARGS=--accept-license" splunk/splunk:7.2.5
2. Login to <http://:8000> and change password for admin user
3. docker stop splunk
4. docker start splunk

Step 4 fails and container stops

included: /opt/ansible/roles/splunk_standalone/tasks/../../splunk_common/tasks/check_for_required_restarts.yml for localhost Monday 08 April 2019 12:57:45 +0000 (0:00:00.087) 0:00:30.655 ********** TASK [splunk_standalone : Check for required restarts] ************************* fatal: [localhost]: FAILED! => {"changed": false, "content": "", "msg": "Status code was -1 and not [200, 404]: Connection failure: [Errno 104] Connection reset by peer", "redirected": false, "status": -1, "url": "http://127.0.0.1:8089/services/messages/restart_required?output_mode=json"}

How can I change the password for admin user?

[nwang92]

I don't believe this is something we support at this current time. Ideally, the password defined at container-start time is the expected password used throughout the lifetime of the container. As long as /opt/splunk/etc is not backed by any persistent volume, you should be able to change the password via the env var SPLUNK_PASSWORD or in your default.yml.

[gibban69]

This blog post https://www.splunk.com/blog/2018/10/24/announcing-splunk-on-docker.html
and the release note https://docs.splunk.com/Documentation/Splunk/7.2.5/ReleaseNotes/MeetSplunk indicates that Splunk 7.2 docker deployment is production ready at least in the S1 setup.

Isn't a realistic scenario for production that /opt/splunk/etc and /opt/splunk/var are mounted on persistent storage and that they thereby survive removal/recreation of the Splunk container and throughout the lifetime of the production setup, rather than just throughout the lifetime of the container? I think that, at some point during the production period, which could be years, the password might need to be changed and in the worst case even be reset.
Password change for the admin user, just have to be supported.

Having the password visible in clear text, either via docker inspect or in default.yml, should not be mandatory considering the security aspects. It should be enough having the password encrypted as in /opt/splunk/etc/passwd.

A suggestion would be that the password given as an environment argument to the container or in default.yml should be used only for setting the initial password. Even if not giving this password, a default known password could be set and be prompted for change or even be forced to change at first login.

The checks done (with ansible) as admin user towards the Management API during container startup to see if the splunk service need to restarted and other checks are a good service, but they should not prevent a startup of the container if they fail due to a changed admin password. Some of the checks do ignore errors, but others don't.

Actually in earlier version 7.2.3 it was, at least, possible to make the container start again after an admin password had changed. Like this:

docker run --name splunk --mount type=bind,source=/some_dir/var,target=/opt/splunk/var  --mount type=bind,source=/some_dir/etc,target=/opt/splunk/etc -p 8000:8000 -p 9997:9997 -d -e "SPLUNK_START_ARGS=--accept-license" -e "SPLUNK_PASSWORD=password" splunk/splunk:7.2.3
# wait for a complete initialization and start up
# change password in Splunk WEB

docker stop splunk; docker rm -v splunk

docker run --name splunk --mount type=bind,source=/some_dir/var,target=/opt/splunk/var --mount type=bind,source=/some_dir/etc,target=/opt/splunk/etc -p 8000:8000 -p 9997:9997 -d -e "SPLUNK_START_ARGS=--accept-license" -e "SPLUNK_PASSWORD" splunk/splunk:7.2.3
#In this older version 7.2.3, start up is actually successful

Notice the SPLUNK_PASSWORD given without assigning a password to it.

[nwang92]

This is great feedback, thanks! For the security note you pointed out, the use of SPLUNK_PASSWORD as plaintext is mostly for the dev workflow. In practice, depending on your orchestrator, I do believe we recommend you roll with docker secrets or kubectl secrets to persist the default.yml or variables in a non-plaintext-manner.

For the password workflow, I suppose there are some differing ideologies in terms of how we expect this gets used vs. how it actually gets used? My assumption was that most users would want to define the "state" of their Splunk environment (including all configs, apps, methods of getting-data-in, etc.) via environment variables or a default.yml. When the container starts up, it will converge towards those desired configurations.

The password change might be something we want to handle differently, although I suppose then it becomes a question of where the line is drawn. For instance, if a user starts a Splunk container with HEC enabled but then disables it via the web UI, the next time the container restarts the internal provisioning will try to enable HEC (because that's what was defined as part of the runtime definition). Then of course, if you introduce the complexity of changing passwords, that particular stage could also fail because changing a setting with an out-of-date password will result in an unauthorized failure.

Regarding your example above, I believe the difference in behavior was exactly the latter case mentioned above - some newer actions taken by the bootstrapping process do require the password, and if this changes between the time the container runs vs. the next container restart, it is likely to fail.