Explore re-allowing multiple shadow deployments (for Istio only as Ambassador doesn't support)

[axsaucedo]

The ability to deploy multiple shadows was removed in 1.3.0 due to Ambassador not supporting multiple shadows, which means that now an error is thrown if more than 2 shadows are created. This issue encompasses exploring whether we can enable the ability to have more than 2 shadow models when istio is enabled but limit it when ambassador is enabled.

[RafalSkolasinski]

It seems that Istio also does not support more than one shadow (mirror): istio/istio#13330

Should we close this one?

[axsaucedo]

@RafalSkolasinski as per discussion closing

[axsaucedo]

Revisiting this and tested to validate whether canary and shadow work together. It seems the limitation of multiple shadows are only on the envoy mirror, but if provided with a canary and shadow, both the canary gets the correct traffic and the shadow gets 100% of the traffic.

Example here:

apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
  labels:
    app: seldon
  name: example
spec:
  name: canary-example
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - image: seldonio/mock_classifier:1.11.2
          imagePullPolicy: IfNotPresent
          name: classifier
        terminationGracePeriodSeconds: 1
    graph:
      children: []
      endpoint:
        type: REST
      name: classifier
      type: MODEL
    name: main
    replicas: 1
    traffic: 75
  - componentSpecs:
    - spec:
        containers:
        - image: seldonio/mock_classifier:1.11.2
          imagePullPolicy: IfNotPresent
          name: classifier
        terminationGracePeriodSeconds: 1
    graph:
      children: []
      endpoint:
        type: REST
      name: classifier
      type: MODEL
    name: canary
    replicas: 1
    traffic: 25
  - componentSpecs:
    - spec:
        containers:
        - image: seldonio/mock_classifier:1.11.2
          imagePullPolicy: IfNotPresent
          name: classifier
        terminationGracePeriodSeconds: 1
    graph:
      children: []
      endpoint:
        type: REST
      name: classifier
      type: MODEL
    name: shadow
    replicas: 1
    shadow: true