Crash Loop back off in OpenShift with supervisord

[RGregulla]

Meta -

Image(s):

node-chrome & hub
Docker-Selenium Image Version(s):
latest (04.10.2018)
Docker Version:
OpenShift 3.7
OS:
RedHat

Expected Behavior -

Actual Behavior -

Running selenium in a OpenShift Environment results in a crash loop back off in Openshift after the changes to supervisord. The version before that change was running perfectly.

This is the log from the container

Traceback (most recent call last):
  File "/usr/bin/supervisord", line 9, in <module>
    load_entry_point('supervisor==3.2.0', 'console_scripts', 'supervisord')()
  File "/usr/lib/python2.7/dist-packages/supervisor/supervisord.py", line 367, in main
    go(options)
  File "/usr/lib/python2.7/dist-packages/supervisor/supervisord.py", line 377, in go
    d.main()
  File "/usr/lib/python2.7/dist-packages/supervisor/supervisord.py", line 77, in main
    info_messages)
  File "/usr/lib/python2.7/dist-packages/supervisor/options.py", line 1388, in make_logger
    stdout = self.nodaemon,
  File "/usr/lib/python2.7/dist-packages/supervisor/loggers.py", line 346, in getLogger
    handlers.append(RotatingFileHandler(filename,'a',maxbytes,backups))
  File "/usr/lib/python2.7/dist-packages/supervisor/loggers.py", line 172, in __init__
    FileHandler.__init__(self, filename, mode)
  File "/usr/lib/python2.7/dist-packages/supervisor/loggers.py", line 98, in __init__
    self.stream = open(filename, mode)
IOError: [Errno 13] Permission denied: '/var/log/supervisor/supervisord.log'

Steps to reproduce -

[diemol]

Fixed in the referenced commit and will be released soon.

[arnaud-deprez]

Hi,
This does not seem to resolve the problem, it can't still run with random user id.

I have this now with selenium/hub:3.14.0-gallium:

2018-10-06 18:40:05,137 WARN Included extra file "/etc/supervisor/conf.d/selenium-hub.conf" during parsing
2018-10-06 18:40:05,139 INFO supervisord started with pid 6
2018-10-06 18:40:06,141 INFO spawned: 'selenium-hub' with pid 9
supervisor: couldn't setuid to 1000: Can't drop privilege as nonroot user
supervisor: child process was not spawned
2018-10-06 18:40:06,145 INFO success: selenium-hub entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2018-10-06 18:40:06,145 INFO exited: selenium-hub (exit status 127; not expected)

The nodes are saying the same thing (example with selenium/node-chrome:3.14.0-gallium):

2018-10-06 18:40:18,157 WARN Included extra file "/etc/supervisor/conf.d/selenium.conf" during parsing
2018-10-06 18:40:18,213 INFO supervisord started with pid 7
2018-10-06 18:40:19,222 INFO spawned: 'xvfb' with pid 10
2018-10-06 18:40:19,225 INFO spawned: 'selenium-node' with pid 11
2018-10-06 18:40:19,226 INFO exited: xvfb (exit status 127; not expected)
supervisor: couldn't setuid to 1000: Can't drop privilege as nonroot user
supervisor: child process was not spawned
2018-10-06 18:40:19,227 INFO success: selenium-node entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2018-10-06 18:40:19,227 INFO exited: selenium-node (exit status 127; not expected)

The result is the hub cannot be started because it's considered unhealthy:

⇒  oc get po -l release=selenium
NAME                                         READY     STATUS    RESTARTS   AGE
selenium-selenium-chrome-68c6596f55-ksjqm    1/1       Running   0          19h
selenium-selenium-firefox-84c599884f-zh4ds   1/1       Running   0          19h
selenium-selenium-hub-5b57c7d784-5gdpv       0/1       Running   6          19h

[diemol]

This is unexpected, we tried to run Supervisor as a non root user to respect the same concept the previous scripts were using, but it seems that to drop privileges Supervisor must be started as root, and seluser can be specified in the configuration file.

We will do those changes, and release during the week. Thanks for testing and reporting this.

[diemol]

@RGregulla do you have a sample yaml that I can try? I have some changes locally and I am testing them in the OpenShift free tier, and it seems to work, but I would like to try your config to be sure before releasing again.

[arnaud-deprez]

Sure, I'm actually using the helm chart here https://github.com/helm/charts/tree/master/stable/selenium with this my-values.yaml:

selenium:
  hub:
    ## The repository and image
    image: "selenium/hub"
    ## The tag for the image
    tag: "3.14.0-gallium"
    serviceType: "ClusterIP"
  chrome:
    enabled: true
    ## The repository and image
    image: "selenium/node-chrome"
    ## The tag for the image
    tag: "3.14.0-gallium"
  firefox:
    enabled: true
    image: "selenium/node-firefox"
    tag: "3.14.0-gallium"

Then on minishift, I do:

namespace=cicd
oc new-project $namespace
helm template --name selenium -f my-values.yaml --set ingress.enabled=true --set ingress.hostname="hub-cicd.$(minishift ip).nip.io" charts/selenium | oc apply -f -
oc get po -w

And once all pods are Running, you should be able to reach hub at http://hub-cicd.$(minishift ip).nip.io and of course there should be no error in the logs of the hub and the nodes.

[diemol]

Just tried it and seems to work fine, I'll merge to master and release soon.