Updates to nssdb will cause image build to fail.

[GrahamDumpleton]

Within /opt/app-root/src/.pki there is a directory nssdb which is a database for something, possibly related to:

export LD_PRELOAD=libnss_wrapper.so

If when Python packages are installed they trigger some operation (network??) which causes that database to be updated, the ownership of the database directory is ending up as:

bash-4.2$ ls -las .pki/
ls: cannot access .pki/nssdb: Permission denied
total 8
4 drwxrwxrwx 4 default default 4096 Aug 28 07:21 .
4 drwxrwxrwx 5 default default 4096 Aug 28 07:22 ..
? d????????? ? ?       ?          ?            ? nssdb

If that occurs, when the assemble script tries to do:

chmod -R og+rwx /opt/app-root

it will fail with the following error as it doesn't have permission to change the ownership of the database as at that point it is running as the unprivileged default user.

E0828 17:25:31.252773 06387 sti.go:419] chmod: chmod: cannot access '/opt/app-root/src/.pki/nssdb'cannot access '/opt/app-root/src/.pki/nssdb': Permission denied: Permission denied

This error the causes the image build to fail due to use of set -e.

I don't know what the database is for and whether it might best be setup to be placed somewhere not in that location, but a workaround is to use instead of the chmod -R:

find /opt/app-root -name .pki -prune -o -exec chmod og+rwx {} \;

This presumes that a .pki file/directory isn't going to exist elsewhere in the users code which should be changed.

[GrahamDumpleton]

The reason you likely are not experiencing this is that the use of 'squash' on the image prevents it from happening.

Go into the 2.7 directory and build the image manually:

docker build -t python-27-centos7-orig .

Then it will fail:

$ s2i build /tmp/example python-27-centos7-orig my-image
W0106 12:30:54.046535 43492 strategies.go:51] An error occurred when pulling python-27-centos7-orig: unable to get python-27-centos7-orig:latest. Attempting to use local image.
I0106 12:30:54.461348 43492 sti.go:446] ---> Copying application source ...
E0106 12:30:54.479268 43492 util.go:85] chgrp: cannot access './.pki/nssdb': Permission denied
E0106 12:30:54.481080 43492 util.go:85] chmod: cannot access './.pki/nssdb': Permission denied
E0106 12:30:54.483790 43492 util.go:85] find: './.pki/nssdb': Permission denied

The problem then is that if people simply follow the example of what to do from the 'Dockerfile' and directory of files and ignores the 'Makefile' in the top directory which does the 'squash', they will have problems.

Also, it made no difference that 'fix-permissions' is now used in assemble in place of the existing chmod.

# set permissions for any installed artifacts
fix-permissions /opt/app-root

As to why it doesn't fail when 'squash' is used, it appears to changes the ownership/permissions the 'nssdb' file as a side effect.

$ docker run --rm -it python-27-centos7-orig bash
bash-4.2$ ls -lasR .pki/
.pki/:
ls: cannot access .pki/nssdb: Permission denied
total 8
4 drwxrwxrwx 4 default root 4096 Jan  6 01:29 .
4 drwxrwxrwx 4 default root 4096 Jan  6 01:29 ..
? d????????? ? ?       ?       ?            ? nssdb
ls: cannot open directory .pki/nssdb: Permission denied
bash-4.2$ exit
exit

$ docker run --rm -it openshift/python-27-centos7 bash
bash-4.2$ ls -lasR .pki/
.pki/:
total 12
4 drwxrwxrwx 3 default root 4096 Sep 29 15:53 .
4 drwxrwxrwx 3 default root 4096 Sep 29 15:53 ..
4 drwxrwxrwx 2 default root 4096 Sep 29 15:53 nssdb

.pki/nssdb:
total 8
4 drwxrwxrwx 2 default root 4096 Sep 29 15:53 .
4 drwxrwxrwx 3 default root 4096 Sep 29 15:53 ..

This isn't really a good situation as is guaranteed that people will not follow to the letter the model you set up for how to do an S2I builder. There probably should be a deliberate action to remove the '.pki' directory from the image using:

RUN rm -rf .pki

Now, what is the '.pki' directory for? And why does 'squash' change the ownership/permissions?

[GrahamDumpleton]

Actually, this is really warped. Those permissions do show it as owned by that user, yet after 'USER 1001' is used in the 'Dockerfile', it becomes inaccessible.

No idea why. Appears it just needs to be removed.

[GrahamDumpleton]

And one reason why people would ignore the Makefile and build scripts, is that they will not work on older bash versions. First because of the 'test -v' option.

• http://unix.stackexchange.com/questions/56837/how-to-test-if-a-variable-is-defined-at-all-in-bash-prior-to-version-4-2-with-th

Second because of assumptions of where per user directory is for Python.

${HOME}/.local/bin/docker-scripts squash -f $base ${IMAGE_NAME}

They will not therefore work on MacOS X.

[mfojtik]

@GrahamDumpleton the docker build should definitely work without squashing... if it is not, then it is a bug.

[GrahamDumpleton]

I can still replicate this with current master.

[GrahamDumpleton]

What is odd though is that if one doesn't use docker on local system, but have OpenShift build it as:

oc new-build https://github.com/openshift/sti-python.git --context-dir=2.7 --name=python27-centos7

then when used to create an application using:

oc new-app python27-centos7~https://github.com/GrahamDumpleton/django-hello-world-v2.git

it all works fine.

Does OpenShift do am implicit squash of images before they are put in the OpenShift registry?

[jawnsy]

I was having this problem, too (for s2i-go), but not anymore. I believe @csrwng has fixed it in this changeset: sclorg/s2i-base-container@1135b0d

@GrahamDumpleton can you try replicating again with master?

[GrahamDumpleton]

Will check when get a chance.

BTW, I have come to the belief that $HOME set to being /opt/app-root/src is possibly a bad idea. Right now I believe it would have been a much better idea to have $HOME being /opt/app-root.

I don't believe the src directory should be the dumping ground for runtime files created by anything being run and which uses $HOME for that. This becomes an issue when using s2i with Docker outside of OpenShift and you want to as part of development/debugging volume mount your local source directory from your machine on top of /opt/app-root/src in the image. All those runtime files then get dumped back into your local system directory.

My philosophy is that the src directory should almost be treated as read only. Anything the application temporarily creates that is for that run of the container should be written elsewhere. Set $HOME to /opt/app-root and say use that and avoids some of the issues. This wouldn't change that the src directory would be the current working directory of the application though to cope with those who use relative paths against best practice.

[jawnsy]

@GrahamDumpleton I think that warrants a separate discussion (maybe as a mailing list discussion or issue against sti-base.) Unfortunately I doubt we can change these things due to the potential of breaking others' s2i images. As an s2i builder image author myself I've also found those things to be strange, so I agree with you completely.

[GrahamDumpleton]

I know it will never change, so don't see a great reason to raise it further. Just wanted to record my opinion somewhere. :-)

[PI-Victor]

@GrahamDumpleton can you confirm that this is no longer an issue and close this?

[GrahamDumpleton]

Closing this on the basis that have not seen the AUFS permission bug crop up with nssdb file for a while now.