Default user ID is always set to 8888

[adriangonz]

Describe the bug

Since 1.2, the Seldon Core operator will always inject by default a securityContext at the pod level which runs every container as user 8888. This is in line with OpenShift's best practices. The user can always change the default user ID through the defaultUserID property of Seldon Core's Helm chart, or by overriding the securityContext at the container level.

Seldon Core's images (i.e. executor and Python wrapper) will always run with this user. However when using a custom Docker image the user ID could change, in which case the user would run into permissions problems.

We should document this change, how it can affect the user and how to change it. Alternatively, we could always set the defaultUserID to empty and start using the runAsNonRoot field. I.e. injecting the following securityContext by default:

securityContext:
  runAsNonRoot: true

That would mean that containers can only be ran as non-root users (unless overriden), and that K8s will use the user ID in the Docker's image metadata.

To reproduce

1. Install SC with the default parameters.
2. Deploy a SeldonDeployment using SC 1.2.
3. Check that the pod has a securityContext with runAsUser: 8888.

Expected behaviour

We should make it clear to the user that containers will run by default as user 8888. Alternatively, we should change the operator so that it doesn't enforce a particular user ID and instead uses the container's user from the image's metadata (the default if you don't specify a runAsUser).

Environment

• Seldon Core 1.2

[ramanNarasimhan77]

@adriangonz
I am currently facing this issue. My team is trying to change all our docker images to run as a non-root appuser

I have a base python image that has USER set to appuser with uid 1000 and entry point tini

While wrapping my model using docker approach, I was installing seldon-core, tensorflow etc using pip with --user flag which would result in the python packages being installed to /home/appuser/.local/bin

I was able to build the image and serve locally but everytime I used it in a seldonDeployment, it would start pods which would then go into Crashloopbackoff state. The pod logs showed exec seldon-core-microservice permission denied

After a whole lot of debugging, I found out that in the securityContext userid is being defaulted to 8888 and then on searching for 8888 in the issues list, I found this issue.

I am using this example deep-mnist.json as reference

Can you please share an example showing how to specify security context explicitly in a SeldonDeployment ? In my case, I have to run as 1000.

[ukclivecox]

You can add a securityContext to your container, e.g.:

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: seldon-model
spec:
  name: test-deployment
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - image: seldonio/mock_classifier_rest:1.3
          name: classifier
          securityContext:
            runAsUser: 1000
    graph:
      children: []
      endpoint:
        type: REST
      name: classifier
      type: MODEL
    name: example
    replicas: 1

[ukclivecox]

I looked at using runAsNonRoot but actually all our containers are defaulted to root but can run as non root. This means we would need to update all our images to have a specific non-root userId. For s2i this is not so easy when combined with the install asssembly scripts we run to install python packages for example.

I therefore plan to just update the docs with above example to show how you can set the defualt userid glocally or at the image level.

@adriangonz

[adriangonz]

Thanks for looking into this @cliveseldon. I agree that changing the user ID in s2i is probably a bit of a pain. Updating the docs sounds good!