Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

incomplete meta.requestPath in responses and redundant "tags" field #3477

Closed
yaliqin opened this issue Aug 9, 2021 — with Board Genius Sync · 3 comments · Fixed by #3604
Closed

incomplete meta.requestPath in responses and redundant "tags" field #3477

yaliqin opened this issue Aug 9, 2021 — with Board Genius Sync · 3 comments · Fixed by #3604
Assignees

Comments

Copy link

yaliqin commented Aug 9, 2021

Describe the bug

When we use inference graph including three models and one combiner to combine the output from three models, we got incomplete requestPath in the meta data and redundant "tags" field like below:
"meta": {"requestPath": {"api-call": "adultdetection:0.76", "combiner": "combiner:0.65"},"tags": {"tags": {"batch_id": "test", "batch_index": 1.0, "batch_instance_id": "6e95e4f6-f272-11eb-b28a-0a58c0ff0f0b”}}}

To reproduce

Expected behaviour

1, have the option to remove requestPath
or 2, have the complete image list of all models
3, remove the redundant level of "tags" field

Environment

Model Details

  • Images of your model: [Output of: kubectl get seldondeployment -n <yourmodelnamespace> <seldondepname> -o yaml | grep image: where <yourmodelnamespace>]
  • Logs of your model: [You can get the logs of your model by running kubectl logs -n <yourmodelnamespace> <seldonpodname> <container>]
@yaliqin yaliqin added bug triage Needs to be triaged and prioritised accordingly labels Aug 9, 2021
@RafalSkolasinski
Copy link
Member

Hi @yaliqin,

Please provide your deployment yaml definition and version of images that are being used / seldon core wrapper version so we can assist better.

I.r.t. double tags level - this look like if you are using batch processor. Redundant level of tags was recently removed by @majolo.

@RafalSkolasinski RafalSkolasinski self-assigned this Aug 10, 2021
@axsaucedo axsaucedo added ml-engineering and removed triage Needs to be triaged and prioritised accordingly labels Aug 10, 2021 — with Board Genius Sync
@yaliqin
Copy link
Author

yaliqin commented Aug 10, 2021

Hi @RafalSkolasinski , thank you for the information. In which release was the redundant "tags" fields removed?
The yaml file I used is listed below:

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  labels:
    app: seldon
  name: brm
  namespace: seldon
spec:
  name: brm
  annotations:
      seldon.io/engine-seldon-log-messages-externally: 'true'
  predictors:
    - componentSpecs:
        - hpaSpec:
            maxReplicas: 4
            metrics:
            - resource:
                name: cpu
                targetAverageUtilization: 30
              type: Resource
            minReplicas: 1
          spec:
            containers:
            - image: artifactory.paypalinc.com/aiplatform/adultdetection:0.76
              imagePullPolicy: IfNotPresent
              name: api-call
              resources:
                  requests:
                    cpu: "0.5"
                    memory: 1Gi
              env:
                - name: SELDON_LOG_LEVEL
                  value: DEBUG
                - name: APP_CONTEXT
                  valueFrom:
                    secretKeyRef:
                      name: ms-api-keymaker
                      key: appcontext
                - name: KEYMAKER_ENDPOINT
                  value: >-
                    https://keymakerapi.g.paypalinc.com:21358/kmsapi/v1/keyobject/all
              volumeMounts:
                - mountPath: /app/test-data
                  name: content-moderation-vol
            - image: artifactory.paypalinc.com/aiplatform/hate-symbol-detection:0.0.1_202107302041
              imagePullPolicy: IfNotPresent
              name: hate-detection
              resources:
                  requests:
                    cpu: "1"
                    memory: 2Gi
              env:
                - name: SELDON_LOG_LEVEL
                  value: DEBUG
              securityContext:
                runAsUser: 0000
              volumeMounts:
                - mountPath: /app/test-data
                  name: content-moderation-vol
            - image: artifactory.paypalinc.com/aiplatform/brm-inspection-cipds:0.2.2
              imagePullPolicy: IfNotPresent
              name: drug-detection
              resources:
                  requests:
                    cpu: "8"
                    memory: 4Gi
              env:
                - name: SELDON_LOG_LEVEL
                  value: DEBUG
              securityContext:
                runAsUser: 0000
              volumeMounts:
                - mountPath: /ContentModeration-Docker/test-data
                  name: content-moderation-vol
            - image: artifactory.paypalinc.com/aiplatform/combiner:0.65
              imagePullPolicy: IfNotPresent
              name: combiner
              resources:
                  requests:
                    cpu: "0.5"
                    memory: 1Gi
              env:
                - name: SELDON_LOG_LEVEL
                  value: DEBUG
            volumes:
              - name: content-moderation-vol
                persistentVolumeClaim:
                  claimName: content-moderation-pvc
      graph:
        children:
          - children: []
            name: api-call
            type: MODEL
            endpoint:
              type: REST
          - children: []
            name: hate-detection
            type: MODEL
            endpoint:
              type: REST
          - children: [ ]
            name: drug-detection
            type: MODEL
            endpoint:
              type: REST
        endpoint:
          type: REST
        name: combiner
        type: COMBINER
        logger:
          mode: all
      name: combiner-graph

The seldon core version is 1.7.0

@RafalSkolasinski
Copy link
Member

Thanks for the yaml definition, I will have a look to determine what is going wrong.

I.r.t to the double tags, I just checked and the fix was made after 1.9.1 release so currently it is only in master. Should be part of 1.10.0 release that will be happening in near future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants