seldon-container-engine keeps restarting because readinessProbe failed #3036

fahadh4ilyas · 2021-03-10T05:39:19Z

Describe the bug

I have a simple SeldonDeployment contain 2 models. After deploying to Minikube, all containers start dan run successfully except seldon-container-engine which keeps restarting because readinessProbe failed.

To reproduce

Here is Dockerfile of my models.

FROM python:3.6-slim
COPY . /app
WORKDIR /app
RUN pip install -r requirements.txt
EXPOSE 5000

# Define environment variable
ENV MODEL_NAME Transformer
ENV API_TYPE REST
ENV SERVICE_TYPE TRANSFORMER
ENV PERSISTENCE 0

CMD exec seldon-core-microservice $MODEL_NAME $API_TYPE --service-type $SERVICE_TYPE --persistence $PERSISTENCE

FROM python:3.6-slim
COPY . /app
WORKDIR /app
RUN pip install -r requirements.txt
EXPOSE 5000

# Define environment variable
ENV MODEL_NAME Transformer
ENV API_TYPE REST
ENV SERVICE_TYPE MODEL
ENV PERSISTENCE 0

CMD exec seldon-core-microservice $MODEL_NAME $API_TYPE --service-type $SERVICE_TYPE --persistence $PERSISTENCE

And here are the inside of Transformer file

class Transformer:

    def __init__(self):

        pass
        

    def transform_input(self, X, names = None, meta = None):

        print('Transformed successfully')
        print('X:', X)
        print('names:', names)
        print('meta:', meta)

        return X

class Transformer:

    def __init__(self):

        pass
        

    def predict(self, X, names = None, meta = None):

        print('Predicted successfully')
        print('X:', X)
        print('names:', names)
        print('meta:', meta)

        return X

Here is the yaml file of seldondeployment

---
apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
  labels:
    app: seldon
  name: "seldon-deployment"
  namespace: test
spec:
  annotations:
    project_name: Sample Pipeline
    deployment_version: v1
  name: "seldon-deployment"
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - image: sample_transformer
          imagePullPolicy: IfNotPresent
          name: textmaker
        - image: sample_predictor
          imagePullPolicy: IfNotPresent
          name: modelmaker
    graph:
      children:
      - name: modelmaker
        endpoint:
          type: REST
        type: MODEL
        children: []
      name: textmaker
      endpoint:
        type: REST
      type: TRANSFORMER
    name: single-model
    replicas: 1
    annotations:
      predictor_version: v1

Expected behaviour

seldon-container-engine is running succesfully without readinessProbe problem.

Environment

Cloud Provider: Minikube
Kubernetes Cluster Version: 1.20
Deployed Seldon System Images: docker.io/seldonio/seldon-core-operator:1.4.0

Model Details

Logs of seldon-container-engine

{"level":"info","ts":1615354266.5420434,"logger":"entrypoint","msg":"Hostname unset will use localhost"}
{"level":"info","ts":1615354266.6263168,"logger":"entrypoint","msg":"Starting","worker":1}
{"level":"info","ts":1615354266.6264114,"logger":"entrypoint","msg":"Starting","worker":2}
{"level":"info","ts":1615354266.6264274,"logger":"entrypoint","msg":"Starting","worker":3}
{"level":"info","ts":1615354266.6264443,"logger":"entrypoint","msg":"Starting","worker":4}
{"level":"info","ts":1615354266.626452,"logger":"entrypoint","msg":"Starting","worker":5}
{"level":"info","ts":1615354266.6276093,"logger":"entrypoint","msg":"Running http server ","port":8000}
{"level":"info","ts":1615354266.6276348,"logger":"entrypoint","msg":"Creating non-TLS listener","port":8000}
{"level":"info","ts":1615354266.628931,"logger":"SeldonRestApi","msg":"Listening","Address":"0.0.0.0:8000"}
{"level":"info","ts":1615354308.8727303,"logger":"entrypoint","msg":"shutting down"}

Logs of textmaker model

2021-03-10 05:20:02,108 - seldon_core.microservice:main:225 - INFO: Starting microservice.py:main
2021-03-10 05:20:02,109 - seldon_core.microservice:main:226 - INFO: Seldon Core version: 1.4.0
2021-03-10 05:20:02,110 - seldon_core.microservice:main:332 - INFO: Parse JAEGER_EXTRA_TAGS []
2021-03-10 05:20:02,111 - seldon_core.microservice:load_annotations:143 - INFO: Found annotation deployment_version:v1
2021-03-10 05:20:02,111 - seldon_core.microservice:load_annotations:143 - INFO: Found annotation kubernetes.io/config.seen:2021-03-10T12:19:34.962586217+07:00
2021-03-10 05:20:02,111 - seldon_core.microservice:load_annotations:143 - INFO: Found annotation kubernetes.io/config.source:api
2021-03-10 05:20:02,111 - seldon_core.microservice:load_annotations:143 - INFO: Found annotation predictor_version:v1
2021-03-10 05:20:02,111 - seldon_core.microservice:load_annotations:143 - INFO: Found annotation project_name:Sample Pipeline
2021-03-10 05:20:02,111 - seldon_core.microservice:load_annotations:143 - INFO: Found annotation prometheus.io/path:/prometheus
2021-03-10 05:20:02,111 - seldon_core.microservice:load_annotations:143 - INFO: Found annotation prometheus.io/scrape:true
2021-03-10 05:20:02,111 - seldon_core.microservice:load_annotations:143 - INFO: Found annotation sidecar.istio.io/status:{\"version\":\"2696c96840179bdcd8f86ed60a643396f0cb250f56458b46a7fc667e6c75ef7f\",\"initContainers\":[\"istio-init\"],\"containers\":[\"istio-proxy\"],\"volumes\":[\"istio-envoy\",\"sds-uds-path\",\"istio-token\"],\"imagePullSecrets\":null}
2021-03-10 05:20:02,111 - seldon_core.microservice:load_annotations:143 - INFO: Found annotation v1:
2021-03-10 05:20:02,111 - seldon_core.microservice:main:335 - INFO: Annotations: {'deployment_version': 'v1', 'kubernetes.io/config.seen': '2021-03-10T12:19:34.962586217+07:00', 'kubernetes.io/config.source': 'api', 'predictor_version': 'v1', 'project_name': 'Sample Pipeline', 'prometheus.io/path': '/prometheus', 'prometheus.io/scrape': 'true', 'sidecar.istio.io/status': '{\\"version\\":\\"2696c96840179bdcd8f86ed60a643396f0cb250f56458b46a7fc667e6c75ef7f\\",\\"initContainers\\":[\\"istio-init\\"],\\"containers\\":[\\"istio-proxy\\"],\\"volumes\\":[\\"istio-envoy\\",\\"sds-uds-path\\",\\"istio-token\\"],\\"imagePullSecrets\\":null}', 'v1': ''}
2021-03-10 05:20:02,111 - seldon_core.microservice:main:339 - INFO: Importing Transformer
2021-03-10 05:20:02,145 - seldon_core.microservice:main:413 - INFO: REST gunicorn microservice running on port 9000
2021-03-10 05:20:02,148 - seldon_core.microservice:main:474 - INFO: REST metrics microservice running on port 6000
2021-03-10 05:20:02,148 - seldon_core.microservice:main:484 - INFO: Starting servers
2021-03-10 05:20:02,156 - seldon_core.wrapper:_set_flask_app_configs:204 - INFO: App Config: <Config {'ENV': 'production', 'DEBUG': False, 'TESTING': False, 'PROPAGATE_EXCEPTIONS': None, 'PRESERVE_CONTEXT_ON_EXCEPTION': None, 'SECRET_KEY': None, 'PERMANENT_SESSION_LIFETIME': datetime.timedelta(31), 'USE_X_SENDFILE': False, 'SERVER_NAME': None, 'APPLICATION_ROOT': '/', 'SESSION_COOKIE_NAME': 'session', 'SESSION_COOKIE_DOMAIN': None, 'SESSION_COOKIE_PATH': None, 'SESSION_COOKIE_HTTPONLY': True, 'SESSION_COOKIE_SECURE': False, 'SESSION_COOKIE_SAMESITE': None, 'SESSION_REFRESH_EACH_REQUEST': True, 'MAX_CONTENT_LENGTH': None, 'SEND_FILE_MAX_AGE_DEFAULT': datetime.timedelta(0, 43200), 'TRAP_BAD_REQUEST_ERRORS': None, 'TRAP_HTTP_EXCEPTIONS': False, 'EXPLAIN_TEMPLATE_LOADING': False, 'PREFERRED_URL_SCHEME': 'http', 'JSON_AS_ASCII': True, 'JSON_SORT_KEYS': True, 'JSONIFY_PRETTYPRINT_REGULAR': False, 'JSONIFY_MIMETYPE': 'application/json', 'TEMPLATES_AUTO_RELOAD': None, 'MAX_COOKIE_SIZE': 4093}>
2021-03-10 05:20:02,161 - seldon_core.wrapper:_set_flask_app_configs:204 - INFO: App Config: <Config {'ENV': 'production', 'DEBUG': False, 'TESTING': False, 'PROPAGATE_EXCEPTIONS': None, 'PRESERVE_CONTEXT_ON_EXCEPTION': None, 'SECRET_KEY': None, 'PERMANENT_SESSION_LIFETIME': datetime.timedelta(31), 'USE_X_SENDFILE': False, 'SERVER_NAME': None, 'APPLICATION_ROOT': '/', 'SESSION_COOKIE_NAME': 'session', 'SESSION_COOKIE_DOMAIN': None, 'SESSION_COOKIE_PATH': None, 'SESSION_COOKIE_HTTPONLY': True, 'SESSION_COOKIE_SECURE': False, 'SESSION_COOKIE_SAMESITE': None, 'SESSION_REFRESH_EACH_REQUEST': True, 'MAX_CONTENT_LENGTH': None, 'SEND_FILE_MAX_AGE_DEFAULT': datetime.timedelta(0, 43200), 'TRAP_BAD_REQUEST_ERRORS': None, 'TRAP_HTTP_EXCEPTIONS': False, 'EXPLAIN_TEMPLATE_LOADING': False, 'PREFERRED_URL_SCHEME': 'http', 'JSON_AS_ASCII': True, 'JSON_SORT_KEYS': True, 'JSONIFY_PRETTYPRINT_REGULAR': False, 'JSONIFY_MIMETYPE': 'application/json', 'TEMPLATES_AUTO_RELOAD': None, 'MAX_COOKIE_SIZE': 4093}>
[2021-03-10 05:20:02 +0000] [40] [INFO] Starting gunicorn 20.0.4
[2021-03-10 05:20:02 +0000] [40] [INFO] Listening at: http://0.0.0.0:6000 (40)
[2021-03-10 05:20:02 +0000] [40] [INFO] Using worker: sync
[2021-03-10 05:20:02 +0000] [43] [INFO] Booting worker with pid: 43
[2021-03-10 05:20:02 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2021-03-10 05:20:02 +0000] [1] [INFO] Listening at: http://0.0.0.0:9000 (1)
[2021-03-10 05:20:02 +0000] [1] [INFO] Using worker: threads
[2021-03-10 05:20:02 +0000] [44] [INFO] Booting worker with pid: 44

Logs of modelmaker model

2021-03-10 05:20:07,669 - seldon_core.microservice:main:225 - INFO: Starting microservice.py:main
2021-03-10 05:20:07,669 - seldon_core.microservice:main:226 - INFO: Seldon Core version: 1.4.0
2021-03-10 05:20:07,672 - seldon_core.microservice:main:332 - INFO: Parse JAEGER_EXTRA_TAGS []
2021-03-10 05:20:07,672 - seldon_core.microservice:load_annotations:143 - INFO: Found annotation deployment_version:v1
2021-03-10 05:20:07,673 - seldon_core.microservice:load_annotations:143 - INFO: Found annotation kubernetes.io/config.seen:2021-03-10T12:19:34.962586217+07:00
2021-03-10 05:20:07,673 - seldon_core.microservice:load_annotations:143 - INFO: Found annotation kubernetes.io/config.source:api
2021-03-10 05:20:07,673 - seldon_core.microservice:load_annotations:143 - INFO: Found annotation predictor_version:v1
2021-03-10 05:20:07,673 - seldon_core.microservice:load_annotations:143 - INFO: Found annotation project_name:Sample Pipeline
2021-03-10 05:20:07,673 - seldon_core.microservice:load_annotations:143 - INFO: Found annotation prometheus.io/path:/prometheus
2021-03-10 05:20:07,673 - seldon_core.microservice:load_annotations:143 - INFO: Found annotation prometheus.io/scrape:true
2021-03-10 05:20:07,673 - seldon_core.microservice:load_annotations:143 - INFO: Found annotation sidecar.istio.io/status:{\"version\":\"2696c96840179bdcd8f86ed60a643396f0cb250f56458b46a7fc667e6c75ef7f\",\"initContainers\":[\"istio-init\"],\"containers\":[\"istio-proxy\"],\"volumes\":[\"istio-envoy\",\"sds-uds-path\",\"istio-token\"],\"imagePullSecrets\":null}
2021-03-10 05:20:07,673 - seldon_core.microservice:load_annotations:143 - INFO: Found annotation v1:
2021-03-10 05:20:07,673 - seldon_core.microservice:main:335 - INFO: Annotations: {'deployment_version': 'v1', 'kubernetes.io/config.seen': '2021-03-10T12:19:34.962586217+07:00', 'kubernetes.io/config.source': 'api', 'predictor_version': 'v1', 'project_name': 'Sample Pipeline', 'prometheus.io/path': '/prometheus', 'prometheus.io/scrape': 'true', 'sidecar.istio.io/status': '{\\"version\\":\\"2696c96840179bdcd8f86ed60a643396f0cb250f56458b46a7fc667e6c75ef7f\\",\\"initContainers\\":[\\"istio-init\\"],\\"containers\\":[\\"istio-proxy\\"],\\"volumes\\":[\\"istio-envoy\\",\\"sds-uds-path\\",\\"istio-token\\"],\\"imagePullSecrets\\":null}', 'v1': ''}
2021-03-10 05:20:07,673 - seldon_core.microservice:main:339 - INFO: Importing Transformer
2021-03-10 05:20:07,705 - seldon_core.microservice:main:413 - INFO: REST gunicorn microservice running on port 9001
2021-03-10 05:20:07,706 - seldon_core.microservice:main:474 - INFO: REST metrics microservice running on port 6001
2021-03-10 05:20:07,706 - seldon_core.microservice:main:484 - INFO: Starting servers
2021-03-10 05:20:07,713 - seldon_core.wrapper:_set_flask_app_configs:204 - INFO: App Config: <Config {'ENV': 'production', 'DEBUG': False, 'TESTING': False, 'PROPAGATE_EXCEPTIONS': None, 'PRESERVE_CONTEXT_ON_EXCEPTION': None, 'SECRET_KEY': None, 'PERMANENT_SESSION_LIFETIME': datetime.timedelta(31), 'USE_X_SENDFILE': False, 'SERVER_NAME': None, 'APPLICATION_ROOT': '/', 'SESSION_COOKIE_NAME': 'session', 'SESSION_COOKIE_DOMAIN': None, 'SESSION_COOKIE_PATH': None, 'SESSION_COOKIE_HTTPONLY': True, 'SESSION_COOKIE_SECURE': False, 'SESSION_COOKIE_SAMESITE': None, 'SESSION_REFRESH_EACH_REQUEST': True, 'MAX_CONTENT_LENGTH': None, 'SEND_FILE_MAX_AGE_DEFAULT': datetime.timedelta(0, 43200), 'TRAP_BAD_REQUEST_ERRORS': None, 'TRAP_HTTP_EXCEPTIONS': False, 'EXPLAIN_TEMPLATE_LOADING': False, 'PREFERRED_URL_SCHEME': 'http', 'JSON_AS_ASCII': True, 'JSON_SORT_KEYS': True, 'JSONIFY_PRETTYPRINT_REGULAR': False, 'JSONIFY_MIMETYPE': 'application/json', 'TEMPLATES_AUTO_RELOAD': None, 'MAX_COOKIE_SIZE': 4093}>
2021-03-10 05:20:07,716 - seldon_core.wrapper:_set_flask_app_configs:204 - INFO: App Config: <Config {'ENV': 'production', 'DEBUG': False, 'TESTING': False, 'PROPAGATE_EXCEPTIONS': None, 'PRESERVE_CONTEXT_ON_EXCEPTION': None, 'SECRET_KEY': None, 'PERMANENT_SESSION_LIFETIME': datetime.timedelta(31), 'USE_X_SENDFILE': False, 'SERVER_NAME': None, 'APPLICATION_ROOT': '/', 'SESSION_COOKIE_NAME': 'session', 'SESSION_COOKIE_DOMAIN': None, 'SESSION_COOKIE_PATH': None, 'SESSION_COOKIE_HTTPONLY': True, 'SESSION_COOKIE_SECURE': False, 'SESSION_COOKIE_SAMESITE': None, 'SESSION_REFRESH_EACH_REQUEST': True, 'MAX_CONTENT_LENGTH': None, 'SEND_FILE_MAX_AGE_DEFAULT': datetime.timedelta(0, 43200), 'TRAP_BAD_REQUEST_ERRORS': None, 'TRAP_HTTP_EXCEPTIONS': False, 'EXPLAIN_TEMPLATE_LOADING': False, 'PREFERRED_URL_SCHEME': 'http', 'JSON_AS_ASCII': True, 'JSON_SORT_KEYS': True, 'JSONIFY_PRETTYPRINT_REGULAR': False, 'JSONIFY_MIMETYPE': 'application/json', 'TEMPLATES_AUTO_RELOAD': None, 'MAX_COOKIE_SIZE': 4093}>
[2021-03-10 05:20:07 +0000] [40] [INFO] Starting gunicorn 20.0.4
[2021-03-10 05:20:07 +0000] [40] [INFO] Listening at: http://0.0.0.0:6001 (40)
[2021-03-10 05:20:07 +0000] [40] [INFO] Using worker: sync
[2021-03-10 05:20:07 +0000] [43] [INFO] Booting worker with pid: 43
[2021-03-10 05:20:07 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2021-03-10 05:20:07 +0000] [1] [INFO] Listening at: http://0.0.0.0:9001 (1)
[2021-03-10 05:20:07 +0000] [1] [INFO] Using worker: threads
[2021-03-10 05:20:07 +0000] [44] [INFO] Booting worker with pid: 44

The text was updated successfully, but these errors were encountered:

ukclivecox · 2021-03-11T08:21:05Z

Two comments:

It looks like you created your own Dockerfile. Which version of the seldon-core codebase did you build from?
I don't see any errors in the executor logs

fahadh4ilyas · 2021-03-11T10:57:41Z

Two comments:

It looks like you created your own Dockerfile. Which version of the seldon-core codebase did you build from?

I don't see any errors in the executor logs

I'm using seldon-core==1.4.0. Based on seldon-core-operator image.

After checking further, it seem the reasons of error is because the namespace has label istio-injection=enabled and makes the deployment also create istio-proxy container inside the pod. After I create another namespace without that label, the deployment run successfully.

The problem is, why is it not working when the namespace has that label? I 100% copied the way to create deployment based on this example.

axsaucedo · 2021-03-15T18:57:39Z

@fahadh4ilyas does it still not work when you disable istio-injection?

ukclivecox · 2021-03-25T07:34:24Z

Closing. Please reopen if still an issue.

fahadh4ilyas added bug triage Needs to be triaged and prioritised accordingly labels Mar 10, 2021

ukclivecox added awaiting-feedback and removed triage Needs to be triaged and prioritised accordingly labels Mar 11, 2021

ukclivecox closed this as completed Mar 25, 2021

ukclivecox removed awaiting-feedback bug labels Mar 25, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

seldon-container-engine keeps restarting because readinessProbe failed #3036

seldon-container-engine keeps restarting because readinessProbe failed #3036

fahadh4ilyas commented Mar 10, 2021

ukclivecox commented Mar 11, 2021

fahadh4ilyas commented Mar 11, 2021

axsaucedo commented Mar 15, 2021

ukclivecox commented Mar 25, 2021

seldon-container-engine keeps restarting because readinessProbe failed #3036

seldon-container-engine keeps restarting because readinessProbe failed #3036

Comments

fahadh4ilyas commented Mar 10, 2021

Describe the bug

To reproduce

Expected behaviour

Environment

Model Details

ukclivecox commented Mar 11, 2021

fahadh4ilyas commented Mar 11, 2021

axsaucedo commented Mar 15, 2021

ukclivecox commented Mar 25, 2021