(Selfsinged) Certificates can't be installed

[Someone894]

Hello,

some days ago I found your wonderful ml-workspace image.
You're doing great work with this image :-)

While testing it out I found a bug.
I suppose that this is a operating-system issue, where you can't really do anything about it, but I'd still like to report it.

When I create an container via:

docker run -d \
	-p 8080:8080 \
	--gpus all \
	--name "ml-workspace" \
	--env WORKSPACE_SSL_ENABLED="true" \
	-v /data/Docker/ML-tooling/workspace:/workspace \
	--env AUTHENTICATE_VIA_JUPYTER="secret" \
	--shm-size 1G \
	--restart always \
	mltooling/ml-workspace-gpu:0.12.1

As you can see I'm using selfsinged certificate mode, since no certificate volumn is mapped.

I get this log file:

2021-01-13 13:23:37,335 [INFO] Starting...
2021-01-13 13:23:37,409 [INFO] Start Workspace
2021-01-13 13:23:37,409 [INFO] Copy tutorials to /workspace folder
2021-01-13 13:23:37,520 [INFO] Running config backup restore.
2021-01-13 13:23:37,520 [INFO] Nothing to restore. Config backup folder is empty.
2021-01-13 13:23:37,526 [INFO] Configure ssh service
2021-01-13 13:23:37,576 [INFO] Creating new SSH Key (id_ed25519)
Agent pid 41
Identity added: /root/.ssh/id_ed25519 (root@3aeea7d289d1)
2021-01-13 13:23:37,626 [INFO] Configure nginx service
Generating self-signed certificate for SSL/HTTPS.
ERROR: usage: test [options] [file_or_dir] [file_or_dir] [...]
test: error: unrecognized arguments: -e
  inifile: None
  rootdir: /etc/ssl/certs
ERROR: usage: test [options] [file_or_dir] [file_or_dir] [...]
test: error: unrecognized arguments: -e
  inifile: None
  rootdir: /etc/ssl/certs
ERROR: usage: test [options] [file_or_dir] [file_or_dir] [...]
test: error: unrecognized arguments: -e
  inifile: None
  rootdir: /etc/ssl/certs

    ...

ERROR: usage: test [options] [file_or_dir] [file_or_dir] [...]
test: error: unrecognized arguments: -e
  inifile: None
  rootdir: /etc/ssl/certs
ERROR: usage: test [options] [file_or_dir] [file_or_dir] [...]
test: error: unrecognized arguments: -e
  inifile: None
  rootdir: /etc/ssl/certs
Warning: there was a problem reading the certificate file /etc/ssl/certs/cert.pem. Message:
  /etc/ssl/certs/cert.pem (No such file or directory)
/resources/scripts/setup-certs.sh: 26: /resources/scripts/setup-certs.sh: cannot create /opt/conda/envs/python2/lib/python2.7/site-packages/certifi/cacert.pem: Directory nonexistent
2021-01-13 13:28:31,325 [INFO] Configure tools
2021-01-13 13:28:31,389 [INFO] Initialize filebrowser database.
2021-01-13 13:28:31,423 [INFO] Create filebrowser admin with generated password: jzwdoikaqyspxumgnvhb
2021-01-13 13:28:31,540 [INFO] Configure cron scripts
2021-01-13 13:28:31,710 [INFO] Scheduling cron check xfdesktop task with with cron: 0 * * * *
2021-01-13 13:28:31,715 [INFO] Running cron jobs:
2021-01-13 13:28:31,715 [INFO] @hourly /opt/conda/bin/python '/resources/scripts/check_xfdesktop_leak.py' check> /proc/1/fd/1 2>/proc/1/fd/2
2021-01-13 13:28:31,784 [INFO] Starting configuration backup.
2021-01-13 13:28:31,876 [INFO] Scheduling cron config backup task with with cron: 0 * * * *
2021-01-13 13:28:31,882 [INFO] Running cron jobs:
2021-01-13 13:28:31,882 [INFO] @hourly /opt/conda/bin/python '/resources/scripts/check_xfdesktop_leak.py' check> /proc/1/fd/1 2>/proc/1/fd/2
2021-01-13 13:28:31,882 [INFO] @hourly . /resources/environment.sh; /opt/conda/bin/python '/resources/scripts/backup_restore_config.py' backup> /proc/1/fd/1 2>/proc/1/fd/2
2021-01-13 13:28:31,899 [INFO] Configure and run custom scripts
2021-01-13 13:28:32,148 INFO Included extra file "/etc/supervisor/conf.d/cron.conf" during parsing
2021-01-13 13:28:32,148 INFO Included extra file "/etc/supervisor/conf.d/filebrowser.conf" during parsing
2021-01-13 13:28:32,148 INFO Included extra file "/etc/supervisor/conf.d/glances.conf" during parsing
2021-01-13 13:28:32,148 INFO Included extra file "/etc/supervisor/conf.d/jupyter.conf" during parsing
2021-01-13 13:28:32,148 INFO Included extra file "/etc/supervisor/conf.d/netdata.conf" during parsing
2021-01-13 13:28:32,148 INFO Included extra file "/etc/supervisor/conf.d/nginx.conf" during parsing
2021-01-13 13:28:32,148 INFO Included extra file "/etc/supervisor/conf.d/novnc.conf" during parsing
2021-01-13 13:28:32,149 INFO Included extra file "/etc/supervisor/conf.d/rsyslog.conf" during parsing
2021-01-13 13:28:32,149 INFO Included extra file "/etc/supervisor/conf.d/sshd.conf" during parsing
2021-01-13 13:28:32,149 INFO Included extra file "/etc/supervisor/conf.d/sslh.conf" during parsing
2021-01-13 13:28:32,149 INFO Included extra file "/etc/supervisor/conf.d/ungit.conf" during parsing
2021-01-13 13:28:32,149 INFO Included extra file "/etc/supervisor/conf.d/vncserver.conf" during parsing
2021-01-13 13:28:32,149 INFO Included extra file "/etc/supervisor/conf.d/vscode.conf" during parsing
2021-01-13 13:28:32,149 INFO Included extra file "/etc/supervisor/conf.d/xrdp.conf" during parsing
2021-01-13 13:28:32,149 INFO Set uid to user 0 succeeded
2021-01-13 13:28:32,159 INFO RPC interface 'supervisor' initialized
2021-01-13 13:28:32,159 CRIT Server 'inet_http_server' running without any HTTP authentication checking
2021-01-13 13:28:32,159 INFO RPC interface 'supervisor' initialized
2021-01-13 13:28:32,159 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2021-01-13 13:28:32,159 INFO supervisord started with pid 1717
2021-01-13 13:28:33,162 INFO spawned: 'rsyslog' with pid 1719
2021-01-13 13:28:33,164 INFO spawned: 'nginx' with pid 1720
2021-01-13 13:28:33,166 INFO spawned: 'sslh' with pid 1721
2021-01-13 13:28:33,169 INFO spawned: 'sshd' with pid 1722
2021-01-13 13:28:33,171 INFO spawned: 'jupyter' with pid 1723
2021-01-13 13:28:33,173 INFO spawned: 'vncserver' with pid 1727
2021-01-13 13:28:33,175 INFO spawned: 'cron' with pid 1728
2021-01-13 13:28:33,177 INFO spawned: 'filebrowser' with pid 1729
2021-01-13 13:28:33,179 INFO spawned: 'glances' with pid 1732
2021-01-13 13:28:33,181 INFO spawned: 'netdata' with pid 1734
2021-01-13 13:28:33,183 INFO spawned: 'novnc' with pid 1736
2021-01-13 13:28:33,185 INFO spawned: 'ungit' with pid 1743
2021-01-13 13:28:33,187 INFO spawned: 'vscode' with pid 1745
2021-01-13 13:28:33,189 INFO spawned: 'xrdp' with pid 1747
2021-01-13 13:28:34,212 INFO success: rsyslog entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-01-13 13:28:34,212 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-01-13 13:28:34,212 INFO success: sslh entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-01-13 13:28:34,212 INFO success: sshd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-01-13 13:28:34,212 INFO success: jupyter entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-01-13 13:28:34,212 INFO success: vncserver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-01-13 13:28:34,212 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-01-13 13:28:34,212 INFO success: filebrowser entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-01-13 13:28:34,212 INFO success: glances entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-01-13 13:28:34,212 INFO success: netdata entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-01-13 13:28:34,212 INFO success: novnc entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-01-13 13:28:34,212 INFO success: ungit entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-01-13 13:28:34,212 INFO success: vscode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-01-13 13:28:34,212 INFO success: xrdp entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-01-13 13:28:36.904169: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2021-01-13 14:00:01,277 [INFO] Starting configuration backup.

As you can see the update-ca-certificates causes a crash. After about 5 min the system keeps going and runs into the nex issue sine the path /opt/conda/envs/python2/lib/python2.7 does not exsist.
I suppose this old python 2.7 path is no longer valied since you're using Python 3.X so I'm not that bothered by it.
But the update-ca-certificates-Bug leads to me beeing unable to install e.g. RStudio within the container.
I Hope that using another Ubuntu base image will fix this issue.

B.t.w. is it a lot of work for you to upgrade yout R 3.6.1 to the R 4.0.3 Version?

Thanks for you wunderfull Software.

[LukasMasuch]

@Someone894 Thanks for reporting this issue! I will try to fix this as soon as possible.

B.t.w. is it a lot of work for you to upgrade yout R 3.6.1 to the R 4.0.3 Version?

Probably not a lot if it works fine with tools like RStudio. I will check this and maybe update it in the next release.

[Someone894]

I took some time and started to research on this topic.
I found one odd behavior that I want to show you:

As you can see below, it makes a difference if you run it on an root bash or on an root bash via sudo:

(base)
root:/workspace
▶ update-ca-certificates
Updating certificates in /etc/ssl/certs...
ERROR: usage: test [options] [file_or_dir] [file_or_dir] [...]
test: error: unrecognized arguments: -e
  inifile: None
  rootdir: /etc/ssl/certs

ERROR: usage: test [options] [file_or_dir] [file_or_dir] [...]
test: error: unrecognized arguments: -e
  inifile: None
  rootdir: /etc/ssl/certs

ERROR: usage: test [options] [file_or_dir] [file_or_dir] [...]
test: error: unrecognized arguments: -e
  inifile: None
  rootdir: /etc/ssl/certs

ERROR: usage: test [options] [file_or_dir] [file_or_dir] [...]
test: error: unrecognized arguments: -e
  inifile: None
  rootdir: /etc/ssl/certs

^C
-------------------------------------------------------------------------------
test 8 <module>

sys.exit(main())

__init__.py 143 main
config = _prepareconfig(args, plugins)

__init__.py 318 _prepareconfig
config = pluginmanager.hook.pytest_cmdline_parse(

hooks.py 286 __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)

manager.py 93 _hookexec
return self._inner_hookexec(hook, methods, kwargs)

manager.py 84 <lambda>
self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(

callers.py 203 _multicall
gen.send(outcome)

helpconfig.py 100 pytest_cmdline_parse
config: Config = outcome.get_result()

callers.py 80 get_result
raise ex[1].with_traceback(ex[2])

callers.py 187 _multicall
res = hook_impl.function(*args)

__init__.py 1003 pytest_cmdline_parse
self.parse(args)

__init__.py 1283 parse
self._preparse(args, addopts=addopts)

__init__.py 1172 _preparse
self.pluginmanager.load_setuptools_entrypoints("pytest11")

manager.py 290 load_setuptools_entrypoints
for ep in dist.entry_points:

metadata.py 240 entry_points
return EntryPoint._from_text(self.read_text('entry_points.txt'))

metadata.py 100 _from_text
config.read_string(text)

configparser.py 723 read_string
self.read_file(sfile, source)

configparser.py 718 read_file
self._read(f, source)

configparser.py 1110 _read
self._join_multiline_values()

configparser.py 1117 _join_multiline_values
all_sections = itertools.chain((defaults,),

KeyboardInterrupt
(base)
root:/workspace                                                                                                          ⍉
▶ sudo update-ca-certificates
Updating certificates in /etc/ssl/certs...
4 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...

Replacing debian:ACCVRAIZ1.pem
Replacing debian:AC_RAIZ_FNMT-RCM.pem
Replacing debian:Actalis_Authentication_Root_CA.pem
Replacing debian:AffirmTrust_Commercial.pem
done.
done.
(base)
root:/workspace

---

As you can see here:

root:/
▶ ls -la
total 80
drwxr-xr-x    1 root root  4096 Jan 21 06:42 .
drwxr-xr-x    1 root root  4096 Jan 21 06:42 ..
drwxr-xr-x    1 root root    41 Jan 10 23:02 bin
drwxr-xr-x    2 root root     6 Apr 24  2018 boot
drwxr-xr-x    5 root root   440 Jan 13 13:28 dev
-rwxr-xr-x    1 root root     0 Jan 13 13:23 .dockerenv
drwxr-xr-x    1 root root   171 Jan 15 11:49 etc
drwxr-xr-x    2 root root     6 Apr 24  2018 home
drwxr-xr-x    1 root root    27 Jan 15 11:48 lib
drwxr-xr-x    2 root root  4096 Jan 15 11:48 lib32
drwxr-xr-x    1 root root    34 Jan 10 02:47 lib64
drwxr-xr-x    2 root root     6 Nov 19 13:07 media
drwxr-xr-x    2 root root     6 Nov 19 13:07 mnt
drwxr-xr-x    1 root root    19 Jan 10 02:52 opt
dr-xr-xr-x 1099 root root     0 Jan 13 13:23 proc
drwsrwsrwx    1 root root    88 Jan 13 13:28 resources
drwsrws---    1 root root  4096 Jan 21 11:13 root
drwxr-xr-x    1 root root   137 Jan 13 13:28 run
drwxr-xr-x    1 root root  4096 Jan 10 02:53 sbin
drwxr-xr-x    2 root root     6 Nov 19 13:07 srv
dr-xr-xr-x   13 root root     0 Aug 10 09:44 sys
-rwxr-xr-x    1 root root 24064 Apr 19  2020 tini
drwx------    1 root root  4096 Jan 21 11:13 tmp
drwxrwxr-x    1 4011 4011    94 Jan 15 11:48 usr
drwxr-xr-x    1 root root    80 Jan 10 02:53 var
drwxr-xr-x    7 1010 6000   185 Jan 20 09:36 workspace

the usr folder has strange permissions, when you change the via chmod -R root:root /usr the update-ca-certs works fine.
Since in the base ubuntu:18.04 the permissions are fine (root:root) i suppose you changed them by accident.

[raethlein]

Hey @Someone894, thanks for reporting this!

So, the root cause seems to be line 181 in script /usr/sbin/update-ca-certificates, the find $ETCCERTSDIR -type l ! -exec test -e {} \; part. The find -exec command does not use Linux's test command there, but /opt/conda/bin/test, which is of course wrong and throws the error with the wrong argument -e passed.
You can verify this by running find $ETCCERTSDIR -type l ! -exec which test \; which outputs

/opt/conda/bin/test
/opt/conda/bin/test
/opt/conda/bin/test
...

A quick fix in your workspace is to execute either:

1. sed -i 's@exec test@exec /usr/bin/test@g' /usr/sbin/update-ca-certificates, which replaces the test call to the specific Linux native call.
2. or export PATH=/usr/bin/:$PATH, although I don't know whether this will have an effect on other programs

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 14 days

[xiaoxiaoyu93]

a temporary hack solution

add a hook before install ca-certificates

RUN echo "post-invoke=sed -i 's@exec test@exec /usr/bin/test@g' /usr/sbin/update-ca-certificates" > /etc/dpkg/dpkg.cfg.d/update-ca-certificates

it works on my build

thanks for @raethlein 's idea

[LukasMasuch]

With the most recent workspace version the certificate problem seems to be solved.

[shism2]

I still ran into the same problem.

[raethlein]

Hey @shism2, which workspace version do you use?
I think @LukasMasuch refers to mltooling/ml-workspace:0.13.0 which is not added as a GitHub release here.

[shism2]

@raethlein Ah okay makes sense. I'm using the latest public one. Nevermind then. @raethlein Where do I add "RUN echo "post-invoke=sed -i 's@exec test@exec /usr/bin/test@g' /usr/sbin/update-ca-certificates" > /etc/dpkg/dpkg.cfg.d/update-ca-certificates"? I tried adding into the dockerfile but that just led to a couple hours of trying to figure out why docker wouldn't build it.

[raethlein]

Alright, so maybe it is fixed :) In case it is fixed, you should be able to just pull the newest version and build that (which basically is version 0.13.0) without the need to add anything. If the problem still is there, I think the line belongs before the RUN with the comment # Install basics or within that same RUN block but before the apt-get install ... ca-certificates part (and without the extra RUN then)

[shism2]

@raethlein Thanks. Do you know when 0.13.0 will be released?

[raethlein]

@shism2 Sorry for the delayed answer, but I am glad to point out now that @LukasMasuch released version 0.13.2 tonight 🙂 In case you find some time to try out the new version, please let us know in case you still encounter the certificate issue.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 14 days