Naming services and service versions in Kubernetes #13
Comments
|
For a better canary deployment, than what deployment controller does; we still follow the same process. So istio service == typical k8s app label And we really don't care about the actual k8s service beyond its selector So with that view, subject is the "app" and versions / tracks are considered parts of the same app. We do need to model AB tests and Canary deployment as first class scenarios. Specifically how istio canary relates to canary using k8deployment controller. |
|
I think it's important to disambiguate that the version here refers to both the service backend version and the service config version. For the latter, it is important not to restart the service container to apply configuration stages. So it is up to the SDS component to assign versions to service endpoints. |
|
I agree with @mjog. The discussion so far has been going towards the
following:
There will be a k8s single service (as usual)
Sets of pods in that service differentiate different versions of the
service.
A collection of pods comprising a particular version of the service can be
identified by a unique set of labels. In other words, label selectors
indicate the versions.
Routing rules can be expressed in config as
"From pods with label selectors to pods with label selectors."
The target "service" for a request can be identified by the DNS name used
(?). The target version can be identified by the pod selector labels
provided in the routing rules, by path prefixes, http headers.
Ideally, the user should be using deployments or RCs for each version of
the service.
With regard to modeling A/B tests, I think that is going to be a very hard
one. A/B in strict sense is a hypothesis based testing concept. Measure
metrics and select the version that users like. And it's not necessarily
A/B. One can test with 20 different variations , tests can run for short or
long time.
Essentially, notion of canary, A/B, red/black, blue/green vary across
industry. They have been (mis)used in very confusing ways.
My suggestion would be to provide the core primitive that powers all these
use cases: version and content based routing as first class entities.
We can then write reference utilities that use the APIs to demonstrate one
form of canary testing (based on our definition), A/B testing, etc. these
can be improved upon by the audience for complicated use cases. WDYT!
On Wed, Dec 14, 2016 at 11:50 PM mandarjog ***@***.***> wrote:
For a better canary deployment, than what deployment controller does; we
still follow the same process. So istio service == typical k8s app label
And we really don't care about the actual k8s service beyond its selector
So with that view, subject is the "app" and versions / tracks are
considered parts of the same app.
Selectors can use labels to target policies to specific versions.
We do need to model AB tests and Canary deployment as first class
scenarios.
Specifically how istio canary relates to canary using k8deployment
controller.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#13 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AH0qd8KwH9mQyOz18XingnFGuxFjeBS5ks5rIMcCgaJpZM4LNmwH>
.
--
~shriram
|
|
Yes, I tend to agree that the version should be a core primitive in the rules. This will unblock us to move on and allow the various use cases for staging, rollouts, testing to be expressed in terms of these rules. |
|
So we won't need to add any istio specific labels at all to distinguish
versions. We can be prescriptive about it. The user would be
differentiating the pods already by deploying the new pods with labels that
contain (service selector labels and additional labels).
If the user wants to have multiple versions running concurrently, they
would have to differentiate the pods in each version through some choice of
label selectors that would divide the set of pods under the service into
disjoint sets (equivalence classes). The disjoint set is not a strict
requirement though (it's just a sane recommendation).
From istio routing perspective, we need to be able to create collections of
pods representing a service version that can then be passed to the proxies
for as upstream clusters.
In terms of logistics, users could deploy the newer versions as just one
pod or a RC explicitly.
On Thu, Dec 15, 2016 at 3:38 PM Kuat ***@***.***> wrote:
Yes, I tend to agree that the version should be a core primitive in the
rules. This will unblock us to move on and allow the various use cases for
staging, rollouts, testing to be expressed in terms of these rules.
More concretely, we all agree that Istio service is a Kubernetes service.
Versions within the service can be specified with extra labels. I am
hesitant to add explicit pod labels to signify versions since Deployments
will remove them (we should wait until we have a hook into pod constructor).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#13 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AH0qd81SIBrK8eU4XN7s8qpNIa0ogZPFks5rIaVIgaJpZM4LNmwH>
.
--
~shriram
|
|
For any attributes that let us classify targets; such as pod.labels I think we are ok. A ---> B How do we gradually change. In this case we do not have any way to sub-classify targets within (B). All pods serving B are fungible.
|
|
@mandarjog if we employ SDS for this task, we can categorize B pods by changing the mapping:
|
|
Where is this auth enforced ? At mixer or proxy ? Or at B ?
If latter, then the approach I described (version based routing) / Kuats
example works. If former, then we have to roll out new config to mixer
which applies the auth checks for 5% of Proxy requests. (Or pushes it to
proxy to let it do the check 5% of time) or creates a deployment of B with
additional annotations and routes 5% of traffic to it.
On Thu, Dec 15, 2016 at 4:19 PM Kuat ***@***.***> wrote:
@mandarjog <https://github.com/mandarjog> if we employ SDS for this task,
we can categorize B pods by changing the mapping:
(service name, version) -> (IP endpoint)
We'd have three policies:
<route A to B with 95% to B0 and 5% to B1>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#13 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AH0qd6NyNF70EvEVrtMBQ1lCKxnA-9B0ks5rIa7qgaJpZM4LNmwH>
.
--
~shriram
|
|
Let's say that B service container is unchanged. Then it's enforced either at the Proxy (next to B) or at the Mixer. My example was referencing the Proxy way - Manager pushes two distinct configs to B1 and B2 and programs all proxies to respect rule (3). For the Mixer, rule (3) can be enforced at Mixer as long as the pods send their version (B1 or B2), meaning proxy has to be aware of which version it is assigned to. Either way, the pod-runs-which-service-version is known at the proxy. |
|
I think @mjog's point is what happens if B does not change? And I start
doing auth check for 5% of requests from A to B? In this case, it's done
only at the mixer (or possibly the proxy if this can be cached).
I think we are agreeing on the same idea.
The concept of non deterministic / weighed application of routing rule or
policy on a per request basis forms the foundation for building various
controllers that orchestrate canary, etc. user needs to be able to specify
this via the config.
The support for this feature needs to be present at both proxy and mixer.
On Thu, Dec 15, 2016 at 4:50 PM Kuat <notifications@github.com> wrote:
Let's say that B service container is unchanged. Then it's enforced either
at the Proxy (next to B) or at the Mixer. My example was referencing the
Proxy way - Manager pushes two distinct configs to B1 and B2 and programs
all proxies to respect rule (3). For the Mixer, rule (3) can be enforced at
Mixer as long as the pods send their version (B1 or B2), meaning proxy has
to be aware of which version it is assigned to. Either way, the
pod-runs-which-service-version is known at the proxy.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#13 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AH0qd_M0wCV0Zy53dxzMbLmFebPS71r0ks5rIbYHgaJpZM4LNmwH>
.
…--
~shriram
|
|
@mandarjog @rshriram would it be possible to enforce the routing rule policy without the Mixer? |
|
Routing rules and policies are two different things. Routing rules will
always flow through the config. The cluster membership (pods and their IPs)
will flow through SDS/CDS. The Mixer will have no hand in it.
The policies (auth, rate limit, etc) are in mixers hand. It's up to the
mixer to cache decisions at the proxy. Applying policies to a certain
percentage of requests can be done at mixer or proxy depending on policy
and caching. @mjog example falls in this category. Envoy's rate limit (with
percentage based application) falls in this category as well.
WDYT?
On Thu, Dec 15, 2016 at 8:57 PM Kuat <notifications@github.com> wrote:
@mandarjog <https://github.com/mandarjog> @rshriram
<https://github.com/rshriram> would it be possible to enforce the routing
rule policy without the Mixer?
I understand there needs to be some runtime component to publish this
information to the proxy mesh. We certainly need SDS, maybe CDS. Would that
be enough? Can we decouple most routing rules from the Mixer at least?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#13 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AH0qdwUx7XNYh8a-6D_DAiJ4esgbdvBWks5rIe_lgaJpZM4LNmwH>
.
…--
~shriram
|
|
OK, I just want to make sure we can run percentage-base rule (see (3) above) without the Mixer. |
|
After some more thought, I think that any non deterministic (percentage based) behaviour has to be confined to the proxy. An important reason is that caching results of a non-deterministic function can have unexpected results. Since proxy is going to cache responses from mixer, they must be deterministic. In the A --> B auth example we have been talking about. It is the job of the proxy to inject an attribute / header on the request so that downstream mixer can use this attribute in the selector, and deterministically apply auth or no auth. |
|
Also been resolved in design decision.. First cut in the ProxyConfig proto. |
The routing rules (see #12) will refer to multiple versions of the same logical service.
If the logical service A has versions "v1" and "v2", we will need to represent both versions as Kubernetes service.
There are approaches:
(a) The natural approach is to identify Kubernetes service A with Istio service A. Now each pod in service A is identical and runs the same proxy container. The only way to differentiate between them is to register them in the Manager and assign pod IPs to versions "v1" and "v2" in the Manager. This approach does not work if the service container is different between "v1" and "v2".
CORRECTION: we can deal with this by binding version to the pod template in the deployment.
(b) The other approach is to instantiate two Kubernetes services "A-v1" and "A-v2". Then the name encodes both the logical service name and version. Due to restrictions in the naming scheme, "A-v1" is likely to be the choice here.
@rshriram any insights on this question?
@mjog how do we map "subjects" in the rules to the concrete service and pod name?
The text was updated successfully, but these errors were encountered: