From 9bc353e59df131c37b82f42b81237e8a6e2f4e00 Mon Sep 17 00:00:00 2001 From: Reto Lehmann Date: Thu, 20 Jul 2023 10:42:59 +0200 Subject: [PATCH 01/20] update service-mesh integration setup documentation (SRVCOM-2368) --- modules/ROOT/nav.adoc | 10 +- .../common-service-mesh-setup.adoc} | 97 ++++++++++++------- ...eventing-service-mesh-containersource.adoc | 0 .../eventing-service-mesh-sinkbinding.adoc | 0 4 files changed, 68 insertions(+), 39 deletions(-) rename modules/{serverless-eventing/pages/service-mesh/eventing-service-mesh-setup.adoc => serverless-common/pages/service-mesh/common-service-mesh-setup.adoc} (75%) rename modules/{serverless-eventing => serverless-common}/pages/service-mesh/eventing-service-mesh-containersource.adoc (100%) rename modules/{serverless-eventing => serverless-common}/pages/service-mesh/eventing-service-mesh-sinkbinding.adoc (100%) diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index e90ae53c..a3d32d20 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -1,9 +1,9 @@ * xref:index.adoc[Overview] -* Serverless Eventing -** Using Eventing with OpenShift Service Mesh -*** xref:serverless-eventing:service-mesh/eventing-service-mesh-setup.adoc[Setup Eventing with OpenShift Service Mesh] -*** xref:serverless-eventing:service-mesh/eventing-service-mesh-containersource.adoc[Using ContainerSource with OpenShift Service Mesh] -*** xref:serverless-eventing:service-mesh/eventing-service-mesh-sinkbinding.adoc[Using SinkBinding with OpenShift Service Mesh] +* Serverless Common +** Using OpenShift Serverless with OpenShift Service Mesh +*** xref:serverless-common:service-mesh/common-service-mesh-setup.adoc[Setup Serverless with OpenShift Service Mesh] +*** xref:serverless-common:service-mesh/eventing-service-mesh-containersource.adoc[Eventing: Using ContainerSource with OpenShift Service Mesh] +*** xref:serverless-common:service-mesh/eventing-service-mesh-sinkbinding.adoc[Eventing: Using SinkBinding with OpenShift Service Mesh] * Serverless Logic ** xref:serverless-logic:about.adoc[About OpenShift Serverless Logic] ** User Guides diff --git a/modules/serverless-eventing/pages/service-mesh/eventing-service-mesh-setup.adoc b/modules/serverless-common/pages/service-mesh/common-service-mesh-setup.adoc similarity index 75% rename from modules/serverless-eventing/pages/service-mesh/eventing-service-mesh-setup.adoc rename to modules/serverless-common/pages/service-mesh/common-service-mesh-setup.adoc index 08810339..1eac3847 100644 --- a/modules/serverless-eventing/pages/service-mesh/eventing-service-mesh-setup.adoc +++ b/modules/serverless-common/pages/service-mesh/common-service-mesh-setup.adoc @@ -1,28 +1,69 @@ -= Setup Eventing with {SMProductName} += Setup {serverlessproductname} with {SMProductName} :compat-mode!: // Metadata: -:description: Setup Eventing with {SMProductName} +:description: Setup {serverlessproductname} with {SMProductName} // TODO +[IMPORTANT] +==== +This page describes the integration of {serverlessproductname} with {smproductname}. In this setup, it is assumed that, all Knative internal components, as well as Knative Services, are part of the Service Mesh and have istio-sidecars injected. This means, that strict mTLS is enforced within the whole mesh. All requests to Knative Services require an mTLS connection (the client must send its certificate), except calls coming from OpenShift Routing. +==== + .Prerequisites * You have access to an {product-title} account with cluster administrator access. * You have created a project or have access to a project with the appropriate roles and permissions to create applications and other workloads in {product-title}. -* Install the {SMProductName} Operator and create a `ServiceMeshControlPlane` resource in the `istio-system` namespace. If you want to use mTLS functionality, you must also set the `spec.security.dataPlane.mtls` field for the `ServiceMeshControlPlane` resource to `true`. +* Install the OpenShift CLI (`oc`). + +* Install the {SMProductName} Operator. + [IMPORTANT] ==== Using {ServerlessProductName} with {SMProductShortName} is only supported with {SMProductName} version 2.0.5 or later. ==== -* Install the {ServerlessOperatorName}. +.Installing and configuring {smproductname} -* Install the OpenShift CLI (`oc`). +. Create a `ServiceMeshControlPlane` resource in the `istio-system` namespace. Make sure to use the following settings, to make Service Mesh work with Serverless: ++ +[IMPORTANT] +==== +If you have an existing `ServiceMeshControlPlane`, make sure that you have the same configuration applied. +==== ++ +[source,yaml] +---- +apiVersion: maistra.io/v2 +kind: ServiceMeshControlPlane +metadata: + name: basic + namespace: istio-system +spec: + profiles: + - default + security: + dataPlane: + mtls: true <1> + techPreview: + meshConfig: + defaultConfig: + terminationDrainDuration: 35s <2> + proxy: + networking: + trafficControl: + inbound: + excludedPorts: <3> + - 8444 # metrics + - 8022 # serving: wait-for-drain k8s pre-stop hook +---- +<1> This enforces strict mTLS in the mesh. Only calls using a valid client-certificate are allowed. +<2> OpenShift Serverless has a graceful termination for Knative Services of 30 seconds. The istio-proxy needs to have a longer termination duration to make sure no requests are dropped. +<3> Those ports are called by Kubernetes and cluster monitoring which are not part of the mesh and cannot be called using mTLS, thus those ports are excluded from the Service Mesh. ++ -.Procedure . Add the namespaces that you would like to integrate with {SMProductShortName} to the `ServiceMeshMemberRoll` object as members: + @@ -55,7 +96,7 @@ $ oc apply -f . Create the necessary gateways so that {SMProductShortName} can accept traffic: + -.Example `knative-local-gateway` object using HTTP +.Example `knative-local-gateway` object using `ISTIO_MUTUAL` (mTLS) [source,yaml] ---- apiVersion: networking.istio.io/v1alpha3 @@ -88,8 +129,10 @@ spec: servers: - port: number: 8081 - name: http - protocol: HTTP <2> + name: https + protocol: HTTPS <2> + tls: + mode: ISTIO_MUTUAL <2> hosts: - "*" --- @@ -110,30 +153,7 @@ spec: targetPort: 8081 ---- <1> Add the name of the secret that contains the wildcard certificate. -<2> The `knative-local-gateway` serves HTTP traffic. Using HTTP means that traffic coming from outside of {SMProductShortName}, but using an internal hostname, such as `example.default.svc.cluster.local`, is not encrypted. You can set up encryption for this path by creating another wildcard certificate and an additional gateway that uses a different `protocol` spec. -+ -.Example `knative-local-gateway` object using HTTPS -[source,yaml] ----- -apiVersion: networking.istio.io/v1alpha3 -kind: Gateway -metadata: - name: knative-local-gateway - namespace: knative-serving -spec: - selector: - istio: ingressgateway - servers: - - port: - number: 443 - name: https - protocol: HTTPS - hosts: - - "*" - tls: - mode: SIMPLE - credentialName: ----- +<2> The `knative-local-gateway` serves HTTPS traffic and expects all clients to send requests using mTLS. This means, that only traffic coming from withing {SMProductShortName} is possible. Workloads from outside the {smproductshortname} must use the external domain using the `istio-ingressgateway`. . Apply the `Gateway` resources: + @@ -142,7 +162,11 @@ spec: $ oc apply -f ---- -. Install Knative Serving by creating the following `KnativeServing` custom resource, which also enables the Istio integration: +.Installing and configuring {serverless} + +. First, install the {serverless} Operator. + +. Then, install Knative Serving by creating the following `KnativeServing` custom resource, which also enables the Istio integration: + [source,yaml] ---- @@ -354,6 +378,11 @@ spec: <1> A namespace that is part of the Service Mesh member roll. <2> Instructs Knative Serving to generate an {product-title} pass-through enabled route, so that the certificates you have generated are served through the ingress gateway directly. <3> Injects {SMProductShortName} sidecars into the Knative service pods. ++ +[IMPORTANT] +==== +Please note, that you have to always add the annotation from the example above to all your Knative `Service` to make them work with {SMPRODUCTSHORTNAME}. +==== . Apply the `Service` resource: + diff --git a/modules/serverless-eventing/pages/service-mesh/eventing-service-mesh-containersource.adoc b/modules/serverless-common/pages/service-mesh/eventing-service-mesh-containersource.adoc similarity index 100% rename from modules/serverless-eventing/pages/service-mesh/eventing-service-mesh-containersource.adoc rename to modules/serverless-common/pages/service-mesh/eventing-service-mesh-containersource.adoc diff --git a/modules/serverless-eventing/pages/service-mesh/eventing-service-mesh-sinkbinding.adoc b/modules/serverless-common/pages/service-mesh/eventing-service-mesh-sinkbinding.adoc similarity index 100% rename from modules/serverless-eventing/pages/service-mesh/eventing-service-mesh-sinkbinding.adoc rename to modules/serverless-common/pages/service-mesh/eventing-service-mesh-sinkbinding.adoc From 974179d48616a97812e644a5569e8aed855fec7e Mon Sep 17 00:00:00 2001 From: Reto Lehmann Date: Thu, 20 Jul 2023 11:17:06 +0200 Subject: [PATCH 02/20] add page for network-isolation based on Service Mesh (SRVCOM-2368) --- modules/ROOT/nav.adoc | 1 + .../multi-tenancy-service-mesh.drawio.svg | 4 + ...common-service-mesh-network-isolation.adoc | 106 ++++++++++++++++++ 3 files changed, 111 insertions(+) create mode 100644 modules/serverless-common/assets/images/service-mesh/multi-tenancy-service-mesh.drawio.svg create mode 100644 modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index a3d32d20..e2f87b45 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -2,6 +2,7 @@ * Serverless Common ** Using OpenShift Serverless with OpenShift Service Mesh *** xref:serverless-common:service-mesh/common-service-mesh-setup.adoc[Setup Serverless with OpenShift Service Mesh] +*** xref:serverless-common:service-mesh/common-service-mesh-network-isolation.adoc[Use Service Mesh to isolate network-traffic] *** xref:serverless-common:service-mesh/eventing-service-mesh-containersource.adoc[Eventing: Using ContainerSource with OpenShift Service Mesh] *** xref:serverless-common:service-mesh/eventing-service-mesh-sinkbinding.adoc[Eventing: Using SinkBinding with OpenShift Service Mesh] * Serverless Logic diff --git a/modules/serverless-common/assets/images/service-mesh/multi-tenancy-service-mesh.drawio.svg b/modules/serverless-common/assets/images/service-mesh/multi-tenancy-service-mesh.drawio.svg new file mode 100644 index 00000000..181aa5c0 --- /dev/null +++ b/modules/serverless-common/assets/images/service-mesh/multi-tenancy-service-mesh.drawio.svg @@ -0,0 +1,4 @@ + + + +







tenant-1
*.tenant-1.svc.cluster.local
tenant-1...







tenant-2
 *.tenant-2.svc.cluster.local
tenant-2...






    knative-serving
knative-serving...






istio-system
istio-system...
Istio Ingress
Gateway
Istio Ingress...
Activator
+ IstioProxy
Activator...
KSVC IstioProxy + QueueProxy
KSVC IstioPro...
KSVC
IstioProxy + QueueProxy
KSVC...
2)
2)
1)
1)
1)
1)
4)
4)
1)
1)
AuthorizationPolicy
from:
    namespaces: ["tenant-1"]
to:
  hosts: ["*.tenant-1.svc.cluster.local"]
AuthorizationPolicy...
AuthorizationPolicy
from:
    namespaces: ["tenant-2"]
to:
  hosts: ["*.tenant-2.svc.cluster.local"]
AuthorizationPolicy...
Activator is the last place we know the "real" source
Activator is the last place...
3)
3)
Legend
Legend
tenant-1: all shown paths are allowed
tenant-1: all shown paths are allow...
tenant-2: 
1) tenant-2 can call ingress-gateway and activator directly.
2) tenant-2 can not call tenant-1 directly. Rejected in tenant-1 istio-proxy.
3) tenant-2 can not call tenant-1 via ingress-gateway. Rejected in tenant-1 istio-proxy.
4) tenant-2 can not call tenant-1 via activator. Rejected in activator istio-proxy, as this is the last known place of the "real" source.
tenant-2:...Text is not SVG - cannot display \ No newline at end of file diff --git a/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc b/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc new file mode 100644 index 00000000..3cd6ccdb --- /dev/null +++ b/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc @@ -0,0 +1,106 @@ += Use {smproductname} to isolate network traffic with {serverlessproductname} +:compat-mode!: +// Metadata: +:description: Use {smproductname} to isolate network traffic with {serverlessproductname} + +// TODO + +{smproductshortname} can be used to isolate network-traffic between "tenants" using {smproductshortname} `AuthorizationPolicy` resources. {serverlessproductname} can also leverage this, using several {smproductshortname} resources, as routing in {serverlessproductname} has multiple ways to reach the user workload (for example when scaled to zero). + + +.High-level architecture +The high-level architecture consists of `AuthorizationPolicies` in the `knative-serving` and the tenants namespaces, while all components are part of the {smproductshortname}. The respective `istio-proxy` sidecars take care of enforcing those rules to isolate traffic between tenants. + +image::service-mesh/multi-tenancy-service-mesh.drawio.svg[] + + +.Prerequisites + +* You have access to an {product-title} account with cluster administrator access. + +* You have created a project or have access to a project with the appropriate roles and permissions to create applications and other workloads in {product-title}. + +* You have set up {serverlessproductname} and {smproductname} as documented in xref:./common-service-mesh-setup.adoc[] + + +.Securing the {smproductshortname} + +. As in the set-up documentation, make sure that all your tenants are part of the same `ServiceMeshMemberRoll` object as members: ++ +[source,yaml] +---- +apiVersion: maistra.io/v1 +kind: ServiceMeshMemberRoll +metadata: + name: default + namespace: istio-system +spec: + members: + - knative-serving + - knative-eventing + - tenant-1 + - tenant-2 +---- ++ +All the namespaces that are part of the Mesh must enforce mTLS in strict mode. This forces Istio to only accept connections with a client-certificate present and allows the `istio-proxy` to validate the origin using `AuthorizationPolicy`. ++ +. For {serverlessproductname} internal components to work, the following `AuthorizationPolicies` are necessary in the `knative-serving` namespace: ++ +[source,yaml] +---- +--- +apiVersion: security.istio.io/v1beta1 +kind: AuthorizationPolicy +metadata: + name: allow-traffic-to-activator <1> + namespace: knative-serving +spec: + selector: + matchLabels: + app: activator + action: ALLOW + rules: + - from: + - source: + namespaces: [ "knative-serving", "istio-system" ] +--- +apiVersion: security.istio.io/v1beta1 +kind: AuthorizationPolicy +metadata: + name: allow-traffic-to-autoscaler <2> + namespace: knative-serving +spec: + selector: + matchLabels: + app: autoscaler + action: ALLOW + rules: + - from: + - source: + namespaces: [ "knative-serving" ] +--- +---- +<1> This rule allows traffic from `istio-system` and `knative-serving` to activator +<2> This rule allows traffic from `knative-serving` to autoscaler ++ +[NOTE] +==== +As soon as one `AuthorizationPolicy` is installed in a namespace, all other traffic is automatically denied. +==== + +. With this set up in place, cluster administrators can use their own `AuthorizationPolicies` to define which namespaces (tenants) can communicate with each other. Each namespace needs: +* One AuthorizationPolicy limiting directly incoming traffic in the tenants namespace +* One AuthorizationPolicy limiting incoming traffic via activator in the `knative-serving` namespace +* One AuthorizationPolicy allowing Kubernetes to call PreStopHooks on Knative Services +* One AuthorizationPolicy allowing cluster monitoring to… // TODO + ++ +As it is a cumbersome task to create all those policies by hand, you can use our helm generator to create the necessary resources for each tenant: + + TODO + + +.Verifying the {smproductshortname} + + TODO + From e18b41eb985b8a858cff5a1ba8c47bc98c1a6629 Mon Sep 17 00:00:00 2001 From: Reto Lehmann Date: Thu, 20 Jul 2023 12:58:01 +0200 Subject: [PATCH 03/20] add helm generator to the network-isolation docs --- ...common-service-mesh-network-isolation.adoc | 33 +++++++++++++++---- 1 file changed, 26 insertions(+), 7 deletions(-) diff --git a/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc b/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc index 3cd6ccdb..87d21c61 100644 --- a/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc +++ b/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc @@ -89,15 +89,34 @@ As soon as one `AuthorizationPolicy` is installed in a namespace, all other traf ==== . With this set up in place, cluster administrators can use their own `AuthorizationPolicies` to define which namespaces (tenants) can communicate with each other. Each namespace needs: -* One AuthorizationPolicy limiting directly incoming traffic in the tenants namespace -* One AuthorizationPolicy limiting incoming traffic via activator in the `knative-serving` namespace -* One AuthorizationPolicy allowing Kubernetes to call PreStopHooks on Knative Services -* One AuthorizationPolicy allowing cluster monitoring to… // TODO - +- One AuthorizationPolicy limiting directly incoming traffic in the tenants namespace +- One AuthorizationPolicy limiting incoming traffic via activator in the `knative-serving` namespace +- One AuthorizationPolicy allowing Kubernetes to call PreStopHooks on Knative Services +- One AuthorizationPolicy allowing cluster monitoring to… // TODO ++ +As it is a cumbersome task to create all those policies by hand, you can use our link:https://github.com/openshift-knative/knative-istio-authz-chart[helm based generator] to create the necessary resources for each tenant: + -As it is a cumbersome task to create all those policies by hand, you can use our helm generator to create the necessary resources for each tenant: +[source,terminal] +---- +# For Serving +helm template oci://quay.io/pierdipi/knative-istio-authz-onboarding --version 0.1.0 --set "namespaces={tenant-1}" --set "name=tenant-1" --set=eventing.enabled=false +helm template oci://quay.io/pierdipi/knative-istio-authz-onboarding --version 0.1.0 --set "namespaces={tenant-2}" --set "name=tenant-2" --set=eventing.enabled=false - TODO +# For Eventing +helm template oci://quay.io/pierdipi/knative-istio-authz-onboarding --version 0.1.0 --set "name=order-service" --set "namespaces={ns1, ns2}" --set=serving.enabled=false +---- ++ +And apply the generated resources to your cluster: ++ +[source,terminal] +---- +$ oc apply -f +---- ++ +[NOTE] +==== +The helm chart has several options that can be passed to configure the generated resources. Please refer to the link:https://github.com/openshift-knative/knative-istio-authz-chart/blob/main/values.yaml[values.yaml] for a full reference. +==== .Verifying the {smproductshortname} From 3dea20d3f6d0849b4dab65121e1418dc3b6093ce Mon Sep 17 00:00:00 2001 From: Reto Lehmann Date: Thu, 20 Jul 2023 13:42:22 +0200 Subject: [PATCH 04/20] refer to external yaml files on GitHub --- ...common-service-mesh-network-isolation.adoc | 60 +++++-------------- 1 file changed, 14 insertions(+), 46 deletions(-) diff --git a/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc b/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc index 87d21c61..57f3e656 100644 --- a/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc +++ b/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc @@ -44,49 +44,20 @@ spec: + All the namespaces that are part of the Mesh must enforce mTLS in strict mode. This forces Istio to only accept connections with a client-certificate present and allows the `istio-proxy` to validate the origin using `AuthorizationPolicy`. + -. For {serverlessproductname} internal components to work, the following `AuthorizationPolicies` are necessary in the `knative-serving` namespace: +. For {serverlessproductname} internal components to work, the several `AuthorizationPolicies` are necessary in the `knative-serving` and the `knative-eventing` namespace. The necessary resources are available in link:https://github.com/openshift-knative/knative-istio-authz-chart/tree/main/setup[here]: + +- `deny-all-by-default.yaml` denies all traffic that is not explicitly allowed in `knative-serving` and `knative-eventing` +- `allow-traffic-to-activator.yaml` allows traffic from `istio-system` and `knative-serving` to activator +- `allow-traffic-to-autoscaler.yaml` allows traffic from `knative-serving` to autoscaler +- `allow-probe-kafka-controller.yaml` allows health probes for kafka components in `knative-eventing` +- `allow-mt-channel-based-broker-to-channels.yaml` allows internal traffic for channel based brokers in `knative-eventing` + -[source,yaml] +Make sure to apply all those rules to your cluster with ++ +[source,terminal] ---- ---- -apiVersion: security.istio.io/v1beta1 -kind: AuthorizationPolicy -metadata: - name: allow-traffic-to-activator <1> - namespace: knative-serving -spec: - selector: - matchLabels: - app: activator - action: ALLOW - rules: - - from: - - source: - namespaces: [ "knative-serving", "istio-system" ] ---- -apiVersion: security.istio.io/v1beta1 -kind: AuthorizationPolicy -metadata: - name: allow-traffic-to-autoscaler <2> - namespace: knative-serving -spec: - selector: - matchLabels: - app: autoscaler - action: ALLOW - rules: - - from: - - source: - namespaces: [ "knative-serving" ] ---- +$ oc apply -f ---- -<1> This rule allows traffic from `istio-system` and `knative-serving` to activator -<2> This rule allows traffic from `knative-serving` to autoscaler -+ -[NOTE] -==== -As soon as one `AuthorizationPolicy` is installed in a namespace, all other traffic is automatically denied. -==== . With this set up in place, cluster administrators can use their own `AuthorizationPolicies` to define which namespaces (tenants) can communicate with each other. Each namespace needs: - One AuthorizationPolicy limiting directly incoming traffic in the tenants namespace @@ -97,13 +68,10 @@ As soon as one `AuthorizationPolicy` is installed in a namespace, all other traf As it is a cumbersome task to create all those policies by hand, you can use our link:https://github.com/openshift-knative/knative-istio-authz-chart[helm based generator] to create the necessary resources for each tenant: + [source,terminal] +.Create resources per tenant with helm ---- -# For Serving -helm template oci://quay.io/pierdipi/knative-istio-authz-onboarding --version 0.1.0 --set "namespaces={tenant-1}" --set "name=tenant-1" --set=eventing.enabled=false -helm template oci://quay.io/pierdipi/knative-istio-authz-onboarding --version 0.1.0 --set "namespaces={tenant-2}" --set "name=tenant-2" --set=eventing.enabled=false - -# For Eventing -helm template oci://quay.io/pierdipi/knative-istio-authz-onboarding --version 0.1.0 --set "name=order-service" --set "namespaces={ns1, ns2}" --set=serving.enabled=false +helm template oci://quay.io/pierdipi/knative-istio-authz-onboarding --version 0.1.0 --set "name=tenant-1" --set "namespaces={ns1, ns2}" +helm template oci://quay.io/pierdipi/knative-istio-authz-onboarding --version 0.1.0 --set "name=tenant-2" --set "namespaces={ns3, ns4}" ---- + And apply the generated resources to your cluster: From 7fe1604ba51756d9487175442cf912f302737464 Mon Sep 17 00:00:00 2001 From: Reto Lehmann Date: Thu, 20 Jul 2023 14:02:36 +0200 Subject: [PATCH 05/20] add improvements for general, more isolated istio installation (SRVCOM-2356) --- .../common-service-mesh-setup.adoc | 36 ++++++++++++++----- 1 file changed, 27 insertions(+), 9 deletions(-) diff --git a/modules/serverless-common/pages/service-mesh/common-service-mesh-setup.adoc b/modules/serverless-common/pages/service-mesh/common-service-mesh-setup.adoc index 1eac3847..2830f70e 100644 --- a/modules/serverless-common/pages/service-mesh/common-service-mesh-setup.adoc +++ b/modules/serverless-common/pages/service-mesh/common-service-mesh-setup.adoc @@ -5,10 +5,17 @@ // TODO -[IMPORTANT] -==== -This page describes the integration of {serverlessproductname} with {smproductname}. In this setup, it is assumed that, all Knative internal components, as well as Knative Services, are part of the Service Mesh and have istio-sidecars injected. This means, that strict mTLS is enforced within the whole mesh. All requests to Knative Services require an mTLS connection (the client must send its certificate), except calls coming from OpenShift Routing. -==== +.Assumptions and limitations + +This page describes the integration of {serverlessproductname} with {smproductname}. In this setup, it is assumed that: + +* All Knative internal components, as well as Knative Services, are part of the {smproductshortname} and have istio-sidecars injected. This means, that strict mTLS is enforced within the whole mesh. All requests to Knative Services require an mTLS connection (the client must send its certificate), except calls coming from OpenShift Routing. +* {serverlessproductname} with Kourier disabled can only target *one* istio service mesh. Multiple meshes can be present in the cluster, but {serverlessproductname} will only be available on one of them. +* The mesh that {serverlessproductname} is part of has to be distinct and ideally only for {serverlessproductname} workloads, as additional configuration like `Gateways` might interfere with the automated configuration by {serverlessproductname}. Other namespaces can be part of this Mesh, but must not configure the istio custom resource `Gateway` that could overlap with {serverlessproductname} `knative-local-gateway` and `knative-ingress-gateway`. +* {smproductname} only allows one `Gateway` to claim a wildcard host binding (`hosts: *`) on the same port (`port: 443`). So the `knative-ingress-gateway` and `knative-local-gateway` have to be unique in the Mesh. If another Mesh is already binding this configuration, a separate Mesh has to be created for {serverlessproductname} workloads. +* Cluster external Knative Services are expected to be called via OpenShift Ingress using OpenShift Routes. It is not supported to access istio directly, e.g. by exposing the `istio-ingressgateway` using a Service with type `NodePort` or `LoadBalancer`. +* Changing the target `ServiceMeshMemberRole`, that {serverlessproductname} is part of (meaning moving {serverlessproductname} to another mesh) is not supported. The only way to change the targeted Service Mesh is to uninstall and reinstall {serverlessproductname}. + .Prerequisites @@ -51,18 +58,24 @@ spec: meshConfig: defaultConfig: terminationDrainDuration: 35s <2> + gateways: + ingress: + service: + metadata: + labels: + knative: ingressgateway <3> proxy: networking: trafficControl: inbound: - excludedPorts: <3> + excludedPorts: <4> - 8444 # metrics - 8022 # serving: wait-for-drain k8s pre-stop hook ---- <1> This enforces strict mTLS in the mesh. Only calls using a valid client-certificate are allowed. <2> OpenShift Serverless has a graceful termination for Knative Services of 30 seconds. The istio-proxy needs to have a longer termination duration to make sure no requests are dropped. -<3> Those ports are called by Kubernetes and cluster monitoring which are not part of the mesh and cannot be called using mTLS, thus those ports are excluded from the Service Mesh. -+ +<3> Defining a specific selector for the `ingress-gateway` to target only the Knative gateway. +<4> Those ports are called by Kubernetes and cluster monitoring which are not part of the mesh and cannot be called using mTLS, thus those ports are excluded from the Service Mesh. . Add the namespaces that you would like to integrate with {SMProductShortName} to the `ServiceMeshMemberRoll` object as members: @@ -106,7 +119,7 @@ metadata: namespace: knative-serving spec: selector: - istio: ingressgateway + knative: ingressgateway servers: - port: number: 443 @@ -125,7 +138,7 @@ metadata: namespace: knative-serving spec: selector: - istio: ingressgateway + knative: ingressgateway servers: - port: number: 8081 @@ -188,9 +201,14 @@ spec: annotations: "sidecar.istio.io/inject": "true" "sidecar.istio.io/rewriteAppHTTPProbers": "true" + config: + istio: <3> + gateway.knative-serving.knative-ingress-gateway: istio-ingressgateway..svc.cluster.local + local-gateway.knative-serving.knative-local-gateway: knative-local-gateway..svc.cluster.local ---- <1> Enables Istio integration. <2> Enables sidecar injection for Knative Serving data plane pods. +<3> Optional: if your istio is *NOT* running in `istio-system`, set those two flags with the correct namespace. . Apply the `KnativeServing` resource: + From 8cdf267adac84d3601a8ad945d94557d4a8b984d Mon Sep 17 00:00:00 2001 From: Reto Lehmann Date: Thu, 20 Jul 2023 14:10:28 +0200 Subject: [PATCH 06/20] minor fixes --- .../common-service-mesh-network-isolation.adoc | 10 ++++++++-- .../pages/service-mesh/common-service-mesh-setup.adoc | 2 +- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc b/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc index 57f3e656..8fd839bc 100644 --- a/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc +++ b/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc @@ -5,14 +5,20 @@ // TODO -{smproductshortname} can be used to isolate network-traffic between "tenants" using {smproductshortname} `AuthorizationPolicy` resources. {serverlessproductname} can also leverage this, using several {smproductshortname} resources, as routing in {serverlessproductname} has multiple ways to reach the user workload (for example when scaled to zero). +{smproductshortname} can be used to isolate network-traffic between "tenants" using {smproductshortname} `AuthorizationPolicy` resources. {serverlessproductname} can also leverage this, using several {smproductshortname} resources. .High-level architecture -The high-level architecture consists of `AuthorizationPolicies` in the `knative-serving` and the tenants namespaces, while all components are part of the {smproductshortname}. The respective `istio-proxy` sidecars take care of enforcing those rules to isolate traffic between tenants. +The high-level architecture consists of `AuthorizationPolicies` in the `knative-serving`, `knative-eventing` and the tenants namespaces, while all components are part of the {smproductshortname}. The respective `istio-proxy` sidecars take care of enforcing those rules to isolate traffic between tenants. + +*Knative Serving* image::service-mesh/multi-tenancy-service-mesh.drawio.svg[] +*Knative Eventing* + +TODO pierdipi, please add your diagram here! + .Prerequisites diff --git a/modules/serverless-common/pages/service-mesh/common-service-mesh-setup.adoc b/modules/serverless-common/pages/service-mesh/common-service-mesh-setup.adoc index 2830f70e..7492f15b 100644 --- a/modules/serverless-common/pages/service-mesh/common-service-mesh-setup.adoc +++ b/modules/serverless-common/pages/service-mesh/common-service-mesh-setup.adoc @@ -166,7 +166,7 @@ spec: targetPort: 8081 ---- <1> Add the name of the secret that contains the wildcard certificate. -<2> The `knative-local-gateway` serves HTTPS traffic and expects all clients to send requests using mTLS. This means, that only traffic coming from withing {SMProductShortName} is possible. Workloads from outside the {smproductshortname} must use the external domain using the `istio-ingressgateway`. +<2> The `knative-local-gateway` serves HTTPS traffic and expects all clients to send requests using mTLS. This means, that only traffic coming from withing {SMProductShortName} is possible. Workloads from outside the {smproductshortname} must use the external domain via OpenShift Routing. . Apply the `Gateway` resources: + From 4743483dc36e380a781d4b70c8342b55dffe867d Mon Sep 17 00:00:00 2001 From: Reto Lehmann Date: Thu, 20 Jul 2023 14:55:57 +0200 Subject: [PATCH 07/20] remove todo for monitoring resources --- .../service-mesh/common-service-mesh-network-isolation.adoc | 1 - 1 file changed, 1 deletion(-) diff --git a/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc b/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc index 8fd839bc..3d42e777 100644 --- a/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc +++ b/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc @@ -69,7 +69,6 @@ $ oc apply -f - One AuthorizationPolicy limiting directly incoming traffic in the tenants namespace - One AuthorizationPolicy limiting incoming traffic via activator in the `knative-serving` namespace - One AuthorizationPolicy allowing Kubernetes to call PreStopHooks on Knative Services -- One AuthorizationPolicy allowing cluster monitoring to… // TODO + As it is a cumbersome task to create all those policies by hand, you can use our link:https://github.com/openshift-knative/knative-istio-authz-chart[helm based generator] to create the necessary resources for each tenant: + From d6e8df75c9f8cf1243b633dfd684ff7519c0869f Mon Sep 17 00:00:00 2001 From: Reto Lehmann Date: Mon, 24 Jul 2023 14:23:48 +0200 Subject: [PATCH 08/20] add resources for setup --- ...common-service-mesh-network-isolation.adoc | 200 +++++++++++++++++- 1 file changed, 191 insertions(+), 9 deletions(-) diff --git a/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc b/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc index 3d42e777..8538ebef 100644 --- a/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc +++ b/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc @@ -50,13 +50,195 @@ spec: + All the namespaces that are part of the Mesh must enforce mTLS in strict mode. This forces Istio to only accept connections with a client-certificate present and allows the `istio-proxy` to validate the origin using `AuthorizationPolicy`. + -. For {serverlessproductname} internal components to work, the several `AuthorizationPolicies` are necessary in the `knative-serving` and the `knative-eventing` namespace. The necessary resources are available in link:https://github.com/openshift-knative/knative-istio-authz-chart/tree/main/setup[here]: - -- `deny-all-by-default.yaml` denies all traffic that is not explicitly allowed in `knative-serving` and `knative-eventing` -- `allow-traffic-to-activator.yaml` allows traffic from `istio-system` and `knative-serving` to activator -- `allow-traffic-to-autoscaler.yaml` allows traffic from `knative-serving` to autoscaler -- `allow-probe-kafka-controller.yaml` allows health probes for kafka components in `knative-eventing` -- `allow-mt-channel-based-broker-to-channels.yaml` allows internal traffic for channel based brokers in `knative-eventing` +. For {serverlessproductname} internal components to work, the several `AuthorizationPolicies` are necessary in the `knative-serving` and the `knative-eventing` namespace: ++ +[source,yaml] +---- +--- +apiVersion: security.istio.io/v1beta1 +kind: AuthorizationPolicy +metadata: + name: deny-all-by-default + namespace: knative-eventing +spec: { } +--- +apiVersion: security.istio.io/v1beta1 +kind: AuthorizationPolicy +metadata: + name: deny-all-by-default + namespace: knative-serving +spec: { } +--- +apiVersion: security.istio.io/v1beta1 +kind: AuthorizationPolicy +metadata: + name: allow-mt-channel-based-broker-ingress-to-imc-dispatcher + namespace: knative-eventing +spec: + action: ALLOW + selector: + matchLabels: + app.kubernetes.io/component: "imc-dispatcher" + rules: + - from: + - source: + namespaces: [ "knative-eventing" ] + principals: [ "cluster.local/ns/knative-eventing/sa/mt-broker-ingress" ] + to: + - operation: + methods: [ "POST" ] +--- +apiVersion: security.istio.io/v1beta1 +kind: AuthorizationPolicy +metadata: + name: allow-mt-channel-based-broker-ingress-to-kafka-channel + namespace: knative-eventing +spec: + action: ALLOW + selector: + matchLabels: + app.kubernetes.io/component: "kafka-channel-receiver" + rules: + - from: + - source: + namespaces: [ "knative-eventing" ] + principals: [ "cluster.local/ns/knative-eventing/sa/mt-broker-ingress" ] + to: + - operation: + methods: [ "POST" ] +--- +apiVersion: security.istio.io/v1beta1 +kind: AuthorizationPolicy +metadata: + name: allow-kafka-channel-to-mt-channel-based-broker-filter + namespace: knative-eventing +spec: + action: ALLOW + selector: + matchLabels: + app.kubernetes.io/component: "broker-filter" + rules: + - from: + - source: + namespaces: [ "knative-eventing" ] + principals: [ "cluster.local/ns/knative-eventing/sa/knative-kafka-channel-data-plane" ] + to: + - operation: + methods: [ "POST" ] +--- +apiVersion: security.istio.io/v1beta1 +kind: AuthorizationPolicy +metadata: + name: allow-imc-to-mt-channel-based-broker-filter + namespace: knative-eventing +spec: + action: ALLOW + selector: + matchLabels: + app.kubernetes.io/component: "broker-filter" + rules: + - from: + - source: + namespaces: [ "knative-eventing" ] + principals: [ "cluster.local/ns/knative-eventing/sa/imc-dispatcher" ] + to: + - operation: + methods: [ "POST" ] +--- +apiVersion: security.istio.io/v1beta1 +kind: AuthorizationPolicy +metadata: + name: allow-probe-kafka-broker-receiver + namespace: knative-eventing +spec: + action: ALLOW + selector: + matchLabels: + app.kubernetes.io/component: "kafka-broker-receiver" + rules: + - from: + - source: + namespaces: [ "knative-eventing" ] + principals: [ "cluster.local/ns/knative-eventing/sa/kafka-controller" ] + to: + - operation: + methods: [ "GET" ] +--- +apiVersion: security.istio.io/v1beta1 +kind: AuthorizationPolicy +metadata: + name: allow-probe-kafka-sink-receiver + namespace: knative-eventing +spec: + action: ALLOW + selector: + matchLabels: + app.kubernetes.io/component: "kafka-sink-receiver" + rules: + - from: + - source: + namespaces: [ "knative-eventing" ] + principals: [ "cluster.local/ns/knative-eventing/sa/kafka-controller" ] + to: + - operation: + methods: [ "GET" ] +--- +apiVersion: security.istio.io/v1beta1 +kind: AuthorizationPolicy +metadata: + name: allow-probe-kafka-channel-receiver + namespace: knative-eventing +spec: + action: ALLOW + selector: + matchLabels: + app.kubernetes.io/component: "kafka-channel-receiver" + rules: + - from: + - source: + namespaces: [ "knative-eventing" ] + principals: [ "cluster.local/ns/knative-eventing/sa/kafka-controller" ] + to: + - operation: + methods: [ "GET" ] +--- +apiVersion: security.istio.io/v1beta1 +kind: AuthorizationPolicy +metadata: + name: allow-traffic-to-activator + namespace: knative-serving +spec: + selector: + matchLabels: + app: activator + action: ALLOW + rules: + - from: + - source: + namespaces: [ "knative-serving", "istio-system" ] +--- +apiVersion: security.istio.io/v1beta1 +kind: AuthorizationPolicy +metadata: + name: allow-traffic-to-autoscaler + namespace: knative-serving +spec: + selector: + matchLabels: + app: autoscaler + action: ALLOW + rules: + - from: + - source: + namespaces: [ "knative-serving" ] +--- +---- +These policies: +- Deny all traffic that is not explicitly allowed in `knative-serving` and `knative-eventing` +- Allow traffic from `istio-system` and `knative-serving` to activator +- Allow traffic from `knative-serving` to autoscaler +- Allow health probes for kafka components in `knative-eventing` +- Allow internal traffic for channel based brokers in `knative-eventing` + Make sure to apply all those rules to your cluster with + @@ -75,8 +257,8 @@ As it is a cumbersome task to create all those policies by hand, you can use our [source,terminal] .Create resources per tenant with helm ---- -helm template oci://quay.io/pierdipi/knative-istio-authz-onboarding --version 0.1.0 --set "name=tenant-1" --set "namespaces={ns1, ns2}" -helm template oci://quay.io/pierdipi/knative-istio-authz-onboarding --version 0.1.0 --set "name=tenant-2" --set "namespaces={ns3, ns4}" +helm template oci://quay.io/openshift-knative/knative-istio-authz-onboarding --version 1.31.0 --set "name=tenant-1" --set "namespaces={ns1, ns2}" +helm template oci://quay.io/openshift-knative/knative-istio-authz-onboarding --version 1.31.0 --set "name=tenant-2" --set "namespaces={ns3, ns4}" ---- + And apply the generated resources to your cluster: From 6cfb94fa2cda28bc9b15477043cefdf00c77808d Mon Sep 17 00:00:00 2001 From: Reto Lehmann Date: Wed, 26 Jul 2023 15:03:43 +0200 Subject: [PATCH 09/20] add block to verify setup --- ...common-service-mesh-network-isolation.adoc | 163 +++++++++++++++++- 1 file changed, 161 insertions(+), 2 deletions(-) diff --git a/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc b/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc index 8538ebef..5ca86299 100644 --- a/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc +++ b/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc @@ -274,7 +274,166 @@ The helm chart has several options that can be passed to configure the generated ==== -.Verifying the {smproductshortname} +.Verifying the configuration + +This verification is assuming that we have two tenants with one namespace each, all part of the `ServiceMeshMemberRoll`, configured with resources listed above. +We can then use curl to verify the connectivity: + +. Deploy Knative Services in both tenants namespaces and a `curl` pod to run test commands: ++ +[source,terminal] +---- +# Tenant 1 +cat <<-EOF | oc apply -f - +apiVersion: serving.knative.dev/v1 +kind: Service +metadata: + name: test-webapp + namespace: tenant-1 + annotations: + serving.knative.openshift.io/enablePassthrough: "true" +spec: + template: + metadata: + annotations: + sidecar.istio.io/inject: 'true' + autoscaling.knative.dev/target-burst-capacity: "-1" + spec: + containers: + - image: docker.io/openshift/hello-openshift + env: + - name: RESPONSE + value: "Hello Serverless!" +EOF + +cat <<-EOF | oc apply -f - +apiVersion: apps/v1 +kind: Deployment +metadata: + name: curl + namespace: tenant-1 + labels: + app: curl +spec: + replicas: 1 + selector: + matchLabels: + app: curl + template: + metadata: + labels: + app: curl + annotations: + sidecar.istio.io/inject: 'true' + spec: + containers: + - name: curl + image: curlimages/curl + command: + - sleep + - "3600" +EOF + +# Tenant 2 +cat <<-EOF | oc apply -f - +apiVersion: serving.knative.dev/v1 +kind: Service +metadata: + name: test-webapp + namespace: tenant-2 + annotations: + serving.knative.openshift.io/enablePassthrough: "true" +spec: + template: + metadata: + annotations: + sidecar.istio.io/inject: 'true' + autoscaling.knative.dev/target-burst-capacity: "-1" + spec: + containers: + - image: docker.io/openshift/hello-openshift + env: + - name: RESPONSE + value: "Hello Serverless!" +EOF +---- + +. Verification ++ +[source,terminal] +---- +# Test tenant-1 -> tenant-1 via cluster local domain (allowed) +oc exec deployment/curl -n tenant-1 -it -- curl -v http://test-webapp.tenant-1:80 + +HTTP/1.1 200 OK +content-length: 18 +content-type: text/plain; charset=utf-8 +date: Wed, 26 Jul 2023 12:49:59 GMT +server: envoy +x-envoy-upstream-service-time: 9 + +Hello Serverless! + + +# Test tenant-1 -> tenant-1 via external domain (allowed) +EXTERNAL_URL=$(oc get ksvc -n tenant-1 test-webapp -o custom-columns=:.status.url --no-headers) +oc exec deployment/curl -n tenant-1 -it -- curl -ik $EXTERNAL_URL + +HTTP/2 200 +content-length: 18 +content-type: text/plain; charset=utf-8 +date: Wed, 26 Jul 2023 12:55:30 GMT +server: istio-envoy +x-envoy-upstream-service-time: 3629 + +Hello Serverless! + + +# Test tenant-1 -> tenant-2 via cluster local domain (not allowed) +oc exec deployment/curl -n tenant-1 -it -- curl -v http://test-webapp.tenant-2:80 + +* processing: http://test-webapp.tenant-2:80 +* Trying 172.30.73.216:80... +* Connected to test-webapp.tenant-2 (172.30.73.216) port 80 +> GET / HTTP/1.1 +> Host: test-webapp.tenant-2 +> User-Agent: curl/8.2.0 +> Accept: */* +> +< HTTP/1.1 403 Forbidden +< content-length: 19 +< content-type: text/plain +< date: Wed, 26 Jul 2023 12:55:49 GMT +< server: envoy +< x-envoy-upstream-service-time: 6 +< +* Connection #0 to host test-webapp.tenant-2 left intact +RBAC: access denied + + +# Test tenant-1 -> tenant-2 via external domain (allowed) +EXTERNAL_URL=$(oc get ksvc -n tenant-2 test-webapp -o custom-columns=:.status.url --no-headers) +oc exec deployment/curl -n tenant-1 -it -- curl -ik $EXTERNAL_URL + +HTTP/2 200 +content-length: 18 +content-type: text/plain; charset=utf-8 +date: Wed, 26 Jul 2023 12:56:22 GMT +server: istio-envoy +x-envoy-upstream-service-time: 2856 + +Hello Serverless! +---- + +. Cleanup ++ +Delete the resources that were created for verification: ++ +[source,terminal] +---- +oc delete deployment/curl -n tenant-1 +oc delete ksvc/test-webapp -n tenant-1 +oc delete ksvc/test-webapp -n tenant-2 +---- - TODO From 8711a956dc97c8ea15947660b149e2f5e69b6ffa Mon Sep 17 00:00:00 2001 From: Reto Lehmann Date: Wed, 2 Aug 2023 09:00:14 +0200 Subject: [PATCH 10/20] use updated diagram of tenancy diagram --- .../images/service-mesh/multi-tenancy-service-mesh.drawio.svg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/serverless-common/assets/images/service-mesh/multi-tenancy-service-mesh.drawio.svg b/modules/serverless-common/assets/images/service-mesh/multi-tenancy-service-mesh.drawio.svg index 181aa5c0..63f07145 100644 --- a/modules/serverless-common/assets/images/service-mesh/multi-tenancy-service-mesh.drawio.svg +++ b/modules/serverless-common/assets/images/service-mesh/multi-tenancy-service-mesh.drawio.svg @@ -1,4 +1,4 @@ -







tenant-1
*.tenant-1.svc.cluster.local
tenant-1...







tenant-2
 *.tenant-2.svc.cluster.local
tenant-2...






    knative-serving
knative-serving...






istio-system
istio-system...
Istio Ingress
Gateway
Istio Ingress...
Activator
+ IstioProxy
Activator...
KSVC IstioProxy + QueueProxy
KSVC IstioPro...
KSVC
IstioProxy + QueueProxy
KSVC...
2)
2)
1)
1)
1)
1)
4)
4)
1)
1)
AuthorizationPolicy
from:
    namespaces: ["tenant-1"]
to:
  hosts: ["*.tenant-1.svc.cluster.local"]
AuthorizationPolicy...
AuthorizationPolicy
from:
    namespaces: ["tenant-2"]
to:
  hosts: ["*.tenant-2.svc.cluster.local"]
AuthorizationPolicy...
Activator is the last place we know the "real" source
Activator is the last place...
3)
3)
Legend
Legend
tenant-1: all shown paths are allowed
tenant-1: all shown paths are allow...
tenant-2: 
1) tenant-2 can call ingress-gateway and activator directly.
2) tenant-2 can not call tenant-1 directly. Rejected in tenant-1 istio-proxy.
3) tenant-2 can not call tenant-1 via ingress-gateway. Rejected in tenant-1 istio-proxy.
4) tenant-2 can not call tenant-1 via activator. Rejected in activator istio-proxy, as this is the last known place of the "real" source.
tenant-2:...Text is not SVG - cannot display \ No newline at end of file +







tenant-1
*.tenant-1.svc.cluster.local
tenant-1...







tenant-2
 *.tenant-2.svc.cluster.local
tenant-2...






    knative-serving
knative-serving...






istio-system
istio-system...
Istio Ingress
Gateway
Istio Ingress...
Activator
+ IstioProxy
Activator...
KSVC IstioProxy + QueueProxy
KSVC IstioPro...
KSVC
IstioProxy + QueueProxy
KSVC...
AuthorizationPolicy
from:
    namespaces: ["tenant-1"]
to:
  hosts: ["*.tenant-1.svc.cluster.local"]
AuthorizationPolicy...
AuthorizationPolicy
from:
    namespaces: ["tenant-2"]
to:
  hosts: ["*.tenant-2.svc.cluster.local"]
AuthorizationPolicy...
Legend:
all calls coming from tenant-2 pod
Legend:...
tenant-2 can not call tenant-1 directly. Rejected in tenant-1 istio-proxy.

tenant-2 can not call tenant-1 via ingress-gateway. Rejected in tenant-1 istio-proxy.

tenant-2 can not call tenant-1 via activator. Rejected in activator istio-proxy, as this is the last known place of the "real" source.
Note: as ingress-gateway can always call activator, this includes calls from tenant-2 via ingress-gateway to activator.
tenant-2 can not call tenant-1 directly. Rejected in tenant-1 istio-proxy....
Activator is the last place we know the "real" source
Activator is...Text is not SVG - cannot display \ No newline at end of file From e043ee135671fe0f2ee1ffd4f5b44cd2c4ef0b1e Mon Sep 17 00:00:00 2001 From: Pierangelo Di Pilato Date: Wed, 23 Aug 2023 09:36:06 +0200 Subject: [PATCH 11/20] Format document Signed-off-by: Pierangelo Di Pilato --- ...common-service-mesh-network-isolation.adoc | 23 ++++++++----------- 1 file changed, 10 insertions(+), 13 deletions(-) diff --git a/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc b/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc index 5ca86299..532ff283 100644 --- a/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc +++ b/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc @@ -7,9 +7,9 @@ {smproductshortname} can be used to isolate network-traffic between "tenants" using {smproductshortname} `AuthorizationPolicy` resources. {serverlessproductname} can also leverage this, using several {smproductshortname} resources. - .High-level architecture -The high-level architecture consists of `AuthorizationPolicies` in the `knative-serving`, `knative-eventing` and the tenants namespaces, while all components are part of the {smproductshortname}. The respective `istio-proxy` sidecars take care of enforcing those rules to isolate traffic between tenants. +The high-level architecture consists of `AuthorizationPolicies` in the `knative-serving`, `knative-eventing` and the tenants namespaces, while all components are part of the {smproductshortname}. +The respective `istio-proxy` sidecars take care of enforcing those rules to isolate traffic between tenants. *Knative Serving* @@ -19,9 +19,7 @@ image::service-mesh/multi-tenancy-service-mesh.drawio.svg[] TODO pierdipi, please add your diagram here! - .Prerequisites - * You have access to an {product-title} account with cluster administrator access. * You have created a project or have access to a project with the appropriate roles and permissions to create applications and other workloads in {product-title}. @@ -30,7 +28,6 @@ TODO pierdipi, please add your diagram here! .Securing the {smproductshortname} - . As in the set-up documentation, make sure that all your tenants are part of the same `ServiceMeshMemberRoll` object as members: + [source,yaml] @@ -48,7 +45,8 @@ spec: - tenant-2 ---- + -All the namespaces that are part of the Mesh must enforce mTLS in strict mode. This forces Istio to only accept connections with a client-certificate present and allows the `istio-proxy` to validate the origin using `AuthorizationPolicy`. +All the namespaces that are part of the Mesh must enforce mTLS in strict mode. +This forces Istio to only accept connections with a client-certificate present and allows the `istio-proxy` to validate the origin using `AuthorizationPolicy`. + . For {serverlessproductname} internal components to work, the several `AuthorizationPolicies` are necessary in the `knative-serving` and the `knative-eventing` namespace: + @@ -233,11 +231,10 @@ spec: namespaces: [ "knative-serving" ] --- ---- + These policies: - Deny all traffic that is not explicitly allowed in `knative-serving` and `knative-eventing` -- Allow traffic from `istio-system` and `knative-serving` to activator -- Allow traffic from `knative-serving` to autoscaler -- Allow health probes for kafka components in `knative-eventing` +- Allow traffic from `istio-system` and `knative-serving` to activator - Allow traffic from `knative-serving` to autoscaler - Allow health probes for kafka components in `knative-eventing` - Allow internal traffic for channel based brokers in `knative-eventing` + Make sure to apply all those rules to your cluster with @@ -247,7 +244,8 @@ Make sure to apply all those rules to your cluster with $ oc apply -f ---- -. With this set up in place, cluster administrators can use their own `AuthorizationPolicies` to define which namespaces (tenants) can communicate with each other. Each namespace needs: +. With this set up in place, cluster administrators can use their own `AuthorizationPolicies` to define which namespaces (tenants) can communicate with each other. +Each namespace needs: - One AuthorizationPolicy limiting directly incoming traffic in the tenants namespace - One AuthorizationPolicy limiting incoming traffic via activator in the `knative-serving` namespace - One AuthorizationPolicy allowing Kubernetes to call PreStopHooks on Knative Services @@ -270,12 +268,11 @@ $ oc apply -f + [NOTE] ==== -The helm chart has several options that can be passed to configure the generated resources. Please refer to the link:https://github.com/openshift-knative/knative-istio-authz-chart/blob/main/values.yaml[values.yaml] for a full reference. +The helm chart has several options that can be passed to configure the generated resources. +Please refer to the link:https://github.com/openshift-knative/knative-istio-authz-chart/blob/main/values.yaml[values.yaml] for a full reference. ==== - .Verifying the configuration - This verification is assuming that we have two tenants with one namespace each, all part of the `ServiceMeshMemberRoll`, configured with resources listed above. We can then use curl to verify the connectivity: From b5101c87da440758705684c78edc648cc041fa4a Mon Sep 17 00:00:00 2001 From: Pierangelo Di Pilato Date: Wed, 23 Aug 2023 09:44:34 +0200 Subject: [PATCH 12/20] Add Eventing diagram for Istio authorization policies Signed-off-by: Pierangelo Di Pilato --- .../multi-tenancy-service-mesh-Eventing.drawio.svg | 4 ++++ ...io.svg => multi-tenancy-service-mesh-Serving.drawio.svg} | 0 .../service-mesh/common-service-mesh-network-isolation.adoc | 6 +++--- 3 files changed, 7 insertions(+), 3 deletions(-) create mode 100644 modules/serverless-common/assets/images/service-mesh/multi-tenancy-service-mesh-Eventing.drawio.svg rename modules/serverless-common/assets/images/service-mesh/{multi-tenancy-service-mesh.drawio.svg => multi-tenancy-service-mesh-Serving.drawio.svg} (100%) diff --git a/modules/serverless-common/assets/images/service-mesh/multi-tenancy-service-mesh-Eventing.drawio.svg b/modules/serverless-common/assets/images/service-mesh/multi-tenancy-service-mesh-Eventing.drawio.svg new file mode 100644 index 00000000..430ed33e --- /dev/null +++ b/modules/serverless-common/assets/images/service-mesh/multi-tenancy-service-mesh-Eventing.drawio.svg @@ -0,0 +1,4 @@ + + + +







tenant-1
*.tenant-1.svc.cluster.local
tenant-1...







tenant-2
 *.tenant-2.svc.cluster.local
tenant-2...






    knative-eventing
knative-eventing...
Receiver
Receiver
Sender
Sender
1)
1)
AuthorizationPolicy
from:
    namespaces: ["tenant-1"]
to:
  hosts: ["*.tenant-1.svc.cluster.local"]
  paths: ["/tenant-1/*"]
AuthorizationPolicy...
AuthorizationPolicy
from:
    namespaces: ["tenant-2"]
to:
  hosts: ["*.tenant-2.svc.cluster.local"]
  paths: ["/tenant-2/*"]
AuthorizationPolicy...
Receiver is the last place we know the "real" source.

Sender sends "Kn-Namespace" header with the resource's namespace as value
Receiver is the last place w...
Legend
Legend
tenant-1: all shown paths are allowed
tenant-1: all shown paths are allow...
tenant-2: 
1) tenant-2 can call "receiver" directly and send events to resources in tenant-1 namespace.
2) tenant-2 can not call tenant-1 directly. Rejected in tenant-1 istio-proxy.
3) sources, triggers, and subscribers in tenant-2 namespaces can not call tenant-1 workloads. Rejected in tenant-1 istio-proxy.
tenant-2:...
3)
3)
AuthorizationPolicy
from:
    namespaces: ["knative-eventing"]
when:
    key: request.headers[Kn-Namespace]
    values: ["tenant-1"]
AuthorizationPolicy...
AuthorizationPolicy
from:
    namespaces: ["knative-eventing"]
when:
    key: request.headers[Kn-Namespace]
    values: ["tenant-2"]
AuthorizationPolicy...
ksvc
ksvc
svc
svc
ksvc
ksvc
svc
svcText is not SVG - cannot display \ No newline at end of file diff --git a/modules/serverless-common/assets/images/service-mesh/multi-tenancy-service-mesh.drawio.svg b/modules/serverless-common/assets/images/service-mesh/multi-tenancy-service-mesh-Serving.drawio.svg similarity index 100% rename from modules/serverless-common/assets/images/service-mesh/multi-tenancy-service-mesh.drawio.svg rename to modules/serverless-common/assets/images/service-mesh/multi-tenancy-service-mesh-Serving.drawio.svg diff --git a/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc b/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc index 532ff283..2a8f40e3 100644 --- a/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc +++ b/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc @@ -13,11 +13,11 @@ The respective `istio-proxy` sidecars take care of enforcing those rules to isol *Knative Serving* -image::service-mesh/multi-tenancy-service-mesh.drawio.svg[] +image::service-mesh/multi-tenancy-service-mesh-Serving.drawio.svg[] *Knative Eventing* -TODO pierdipi, please add your diagram here! +image::service-mesh/multi-tenancy-service-mesh-Eventing.drawio.svg[] .Prerequisites * You have access to an {product-title} account with cluster administrator access. @@ -237,7 +237,7 @@ These policies: - Allow traffic from `istio-system` and `knative-serving` to activator - Allow traffic from `knative-serving` to autoscaler - Allow health probes for kafka components in `knative-eventing` - Allow internal traffic for channel based brokers in `knative-eventing` + -Make sure to apply all those rules to your cluster with +Make sure to apply all those rules to your cluster with: + [source,terminal] ---- From 176969c9cd0a075e13ed5daab313aa6e2ea87bb4 Mon Sep 17 00:00:00 2001 From: Pierangelo Di Pilato Date: Wed, 23 Aug 2023 10:41:13 +0200 Subject: [PATCH 13/20] Fix format Signed-off-by: Pierangelo Di Pilato --- ...common-service-mesh-network-isolation.adoc | 22 ++++++++++--------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc b/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc index 2a8f40e3..1fd9975a 100644 --- a/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc +++ b/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc @@ -7,9 +7,9 @@ {smproductshortname} can be used to isolate network-traffic between "tenants" using {smproductshortname} `AuthorizationPolicy` resources. {serverlessproductname} can also leverage this, using several {smproductshortname} resources. + .High-level architecture -The high-level architecture consists of `AuthorizationPolicies` in the `knative-serving`, `knative-eventing` and the tenants namespaces, while all components are part of the {smproductshortname}. -The respective `istio-proxy` sidecars take care of enforcing those rules to isolate traffic between tenants. +The high-level architecture consists of `AuthorizationPolicies` in the `knative-serving`, `knative-eventing` and the tenants namespaces, while all components are part of the {smproductshortname}. The respective `istio-proxy` sidecars take care of enforcing those rules to isolate traffic between tenants. *Knative Serving* @@ -20,6 +20,7 @@ image::service-mesh/multi-tenancy-service-mesh-Serving.drawio.svg[] image::service-mesh/multi-tenancy-service-mesh-Eventing.drawio.svg[] .Prerequisites + * You have access to an {product-title} account with cluster administrator access. * You have created a project or have access to a project with the appropriate roles and permissions to create applications and other workloads in {product-title}. @@ -28,6 +29,7 @@ image::service-mesh/multi-tenancy-service-mesh-Eventing.drawio.svg[] .Securing the {smproductshortname} + . As in the set-up documentation, make sure that all your tenants are part of the same `ServiceMeshMemberRoll` object as members: + [source,yaml] @@ -45,8 +47,7 @@ spec: - tenant-2 ---- + -All the namespaces that are part of the Mesh must enforce mTLS in strict mode. -This forces Istio to only accept connections with a client-certificate present and allows the `istio-proxy` to validate the origin using `AuthorizationPolicy`. +All the namespaces that are part of the Mesh must enforce mTLS in strict mode. This forces Istio to only accept connections with a client-certificate present and allows the `istio-proxy` to validate the origin using `AuthorizationPolicy`. + . For {serverlessproductname} internal components to work, the several `AuthorizationPolicies` are necessary in the `knative-serving` and the `knative-eventing` namespace: + @@ -231,10 +232,11 @@ spec: namespaces: [ "knative-serving" ] --- ---- - These policies: - Deny all traffic that is not explicitly allowed in `knative-serving` and `knative-eventing` -- Allow traffic from `istio-system` and `knative-serving` to activator - Allow traffic from `knative-serving` to autoscaler - Allow health probes for kafka components in `knative-eventing` +- Allow traffic from `istio-system` and `knative-serving` to activator +- Allow traffic from `knative-serving` to autoscaler +- Allow health probes for kafka components in `knative-eventing` - Allow internal traffic for channel based brokers in `knative-eventing` + Make sure to apply all those rules to your cluster with: @@ -244,8 +246,7 @@ Make sure to apply all those rules to your cluster with: $ oc apply -f ---- -. With this set up in place, cluster administrators can use their own `AuthorizationPolicies` to define which namespaces (tenants) can communicate with each other. -Each namespace needs: +. With this set up in place, cluster administrators can use their own `AuthorizationPolicies` to define which namespaces (tenants) can communicate with each other. Each namespace needs: - One AuthorizationPolicy limiting directly incoming traffic in the tenants namespace - One AuthorizationPolicy limiting incoming traffic via activator in the `knative-serving` namespace - One AuthorizationPolicy allowing Kubernetes to call PreStopHooks on Knative Services @@ -268,11 +269,12 @@ $ oc apply -f + [NOTE] ==== -The helm chart has several options that can be passed to configure the generated resources. -Please refer to the link:https://github.com/openshift-knative/knative-istio-authz-chart/blob/main/values.yaml[values.yaml] for a full reference. +The helm chart has several options that can be passed to configure the generated resources. Please refer to the link:https://github.com/openshift-knative/knative-istio-authz-chart/blob/main/values.yaml[values.yaml] for a full reference. ==== + .Verifying the configuration + This verification is assuming that we have two tenants with one namespace each, all part of the `ServiceMeshMemberRoll`, configured with resources listed above. We can then use curl to verify the connectivity: From 8e08f934deb616dc015b2eafc6bebd5a3d41c875 Mon Sep 17 00:00:00 2001 From: Reto Lehmann Date: Mon, 28 Aug 2023 08:36:28 +0200 Subject: [PATCH 14/20] add review improvements --- modules/ROOT/nav.adoc | 2 +- ...common-service-mesh-network-isolation.adoc | 97 +++++++++---------- .../common-service-mesh-setup.adoc | 26 +++-- 3 files changed, 60 insertions(+), 65 deletions(-) diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index e2f87b45..d83e2ca4 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -1,5 +1,5 @@ * xref:index.adoc[Overview] -* Serverless Common +* Serverless ** Using OpenShift Serverless with OpenShift Service Mesh *** xref:serverless-common:service-mesh/common-service-mesh-setup.adoc[Setup Serverless with OpenShift Service Mesh] *** xref:serverless-common:service-mesh/common-service-mesh-network-isolation.adoc[Use Service Mesh to isolate network-traffic] diff --git a/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc b/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc index 1fd9975a..fb62bc21 100644 --- a/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc +++ b/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc @@ -3,13 +3,14 @@ // Metadata: :description: Use {smproductname} to isolate network traffic with {serverlessproductname} -// TODO - -{smproductshortname} can be used to isolate network-traffic between "tenants" using {smproductshortname} `AuthorizationPolicy` resources. {serverlessproductname} can also leverage this, using several {smproductshortname} resources. +{smproductshortname} can be used to isolate network-traffic between tenants on a shared OpenShift cluster using {smproductshortname} `AuthorizationPolicy` resources. +{serverlessproductname} can also leverage this, using several {smproductshortname} resources. +.Tenants +A tenant is a group of one or multiple OpenShift projects that can access each other over the network on a shared OpenShift cluster. .High-level architecture -The high-level architecture consists of `AuthorizationPolicies` in the `knative-serving`, `knative-eventing` and the tenants namespaces, while all components are part of the {smproductshortname}. The respective `istio-proxy` sidecars take care of enforcing those rules to isolate traffic between tenants. +The high-level architecture consists of `AuthorizationPolicies` in the `knative-serving`, `knative-eventing` and the tenants namespaces, while all components are part of the {smproductshortname}. The injected {smproductshortname} sidecars take care of enforcing those rules to isolate network traffic between tenants. *Knative Serving* @@ -23,14 +24,14 @@ image::service-mesh/multi-tenancy-service-mesh-Eventing.drawio.svg[] * You have access to an {product-title} account with cluster administrator access. -* You have created a project or have access to a project with the appropriate roles and permissions to create applications and other workloads in {product-title}. +* You have set up {serverlessproductname} and {smproductname} as documented in xref:./common-service-mesh-setup.adoc[]. -* You have set up {serverlessproductname} and {smproductname} as documented in xref:./common-service-mesh-setup.adoc[] +* You have created one or multiple OpenShift projects for each tenant. .Securing the {smproductshortname} -. As in the set-up documentation, make sure that all your tenants are part of the same `ServiceMeshMemberRoll` object as members: +. As in the set-up documentation, make sure that all your tenants OpenShift projects are part of the same `ServiceMeshMemberRoll` object as members: + [source,yaml] ---- @@ -41,19 +42,20 @@ metadata: namespace: istio-system spec: members: - - knative-serving - - knative-eventing - - tenant-1 - - tenant-2 + - knative-serving # static value, needs to be here, see setup page + - knative-eventing # static value, needs to be here, see setup page + - team-alpha-1 # example OpenShift project that belongs to the team-alpha tenant + - team-alpha-2 # example OpenShift project that belongs th the team-alpha tenant + - team-bravo-1 # example OpenShift project that belongs to the team-bravo tenant + - team-bravo-2 # example OpenShift project that belongs th the team-bravo tenant ---- + -All the namespaces that are part of the Mesh must enforce mTLS in strict mode. This forces Istio to only accept connections with a client-certificate present and allows the `istio-proxy` to validate the origin using `AuthorizationPolicy`. +All the OpenShift projects that are part of the mesh must enforce mTLS in strict mode. This forces Istio to only accept connections with a client-certificate present and allows the {smproductshortname} sidecar to validate the origin using `AuthorizationPolicy`. + . For {serverlessproductname} internal components to work, the several `AuthorizationPolicies` are necessary in the `knative-serving` and the `knative-eventing` namespace: + [source,yaml] ---- ---- apiVersion: security.istio.io/v1beta1 kind: AuthorizationPolicy metadata: @@ -230,13 +232,12 @@ spec: - from: - source: namespaces: [ "knative-serving" ] ---- ---- -These policies: +These policies restrict the access rules for the network communication between OpenShift Serverless' system components. In detail, they enforce the following rules: - Deny all traffic that is not explicitly allowed in `knative-serving` and `knative-eventing` - Allow traffic from `istio-system` and `knative-serving` to activator - Allow traffic from `knative-serving` to autoscaler -- Allow health probes for kafka components in `knative-eventing` +- Allow health probes for Apache Kafka components in `knative-eventing` - Allow internal traffic for channel based brokers in `knative-eventing` + Make sure to apply all those rules to your cluster with: @@ -246,18 +247,18 @@ Make sure to apply all those rules to your cluster with: $ oc apply -f ---- -. With this set up in place, cluster administrators can use their own `AuthorizationPolicies` to define which namespaces (tenants) can communicate with each other. Each namespace needs: -- One AuthorizationPolicy limiting directly incoming traffic in the tenants namespace -- One AuthorizationPolicy limiting incoming traffic via activator in the `knative-serving` namespace -- One AuthorizationPolicy allowing Kubernetes to call PreStopHooks on Knative Services +. With this set up in place, cluster administrators can use their own `AuthorizationPolicies` to define which OpenShift projects can communicate with each other. Every OpenShift project of a tenant needs: +- One `AuthorizationPolicy` limiting directly incoming traffic to the tenants OpenShift project +- One `AuthorizationPolicy` limiting incoming traffic via the activator component of {serverlessproductname} that runs in the `knative-serving` OpenShift project +- One `AuthorizationPolicy` allowing Kubernetes to call `PreStopHooks` on Knative Services + As it is a cumbersome task to create all those policies by hand, you can use our link:https://github.com/openshift-knative/knative-istio-authz-chart[helm based generator] to create the necessary resources for each tenant: + [source,terminal] .Create resources per tenant with helm ---- -helm template oci://quay.io/openshift-knative/knative-istio-authz-onboarding --version 1.31.0 --set "name=tenant-1" --set "namespaces={ns1, ns2}" -helm template oci://quay.io/openshift-knative/knative-istio-authz-onboarding --version 1.31.0 --set "name=tenant-2" --set "namespaces={ns3, ns4}" +helm template oci://quay.io/openshift-knative/knative-istio-authz-onboarding --version 1.31.0 --set "name=team-alpha" --set "namespaces={team-alpha-1,team-alpha-2}" > team-alpha.yaml +helm template oci://quay.io/openshift-knative/knative-istio-authz-onboarding --version 1.31.0 --set "name=team-bravo" --set "namespaces={team-bravo-1,team-bravo-2}" > team-bravo.yaml ---- + And apply the generated resources to your cluster: @@ -282,13 +283,13 @@ We can then use curl to verify the connectivity: + [source,terminal] ---- -# Tenant 1 -cat <<-EOF | oc apply -f - +# Team Alpha +cat < tenant-1 via cluster local domain (allowed) -oc exec deployment/curl -n tenant-1 -it -- curl -v http://test-webapp.tenant-1:80 +# Test team-alpha-1 -> team-alpha-1 via cluster local domain (allowed) +oc exec deployment/curl -n team-alpha-1 -it -- curl -v http://test-webapp.team-alpha-1:80 HTTP/1.1 200 OK content-length: 18 @@ -374,9 +373,9 @@ x-envoy-upstream-service-time: 9 Hello Serverless! -# Test tenant-1 -> tenant-1 via external domain (allowed) -EXTERNAL_URL=$(oc get ksvc -n tenant-1 test-webapp -o custom-columns=:.status.url --no-headers) -oc exec deployment/curl -n tenant-1 -it -- curl -ik $EXTERNAL_URL +# Test team-alpha-1 -> team-alpha-1 via external domain (allowed) +EXTERNAL_URL=$(oc get ksvc -n team-alpha-1 test-webapp -o custom-columns=:.status.url --no-headers) +oc exec deployment/curl -n team-alpha-1 -it -- curl -ik $EXTERNAL_URL HTTP/2 200 content-length: 18 @@ -388,14 +387,14 @@ x-envoy-upstream-service-time: 3629 Hello Serverless! -# Test tenant-1 -> tenant-2 via cluster local domain (not allowed) -oc exec deployment/curl -n tenant-1 -it -- curl -v http://test-webapp.tenant-2:80 +# Test team-alpha-1 -> team-bravo-1 via cluster local domain (not allowed) +oc exec deployment/curl -n team-alpha-1 -it -- curl -v http://test-webapp.team-bravo-1:80 -* processing: http://test-webapp.tenant-2:80 +* processing: http://test-webapp.team-bravo-1:80 * Trying 172.30.73.216:80... -* Connected to test-webapp.tenant-2 (172.30.73.216) port 80 +* Connected to test-webapp.team-bravo-1 (172.30.73.216) port 80 > GET / HTTP/1.1 -> Host: test-webapp.tenant-2 +> Host: test-webapp.team-bravo-1 > User-Agent: curl/8.2.0 > Accept: */* > @@ -406,13 +405,13 @@ oc exec deployment/curl -n tenant-1 -it -- curl -v http://test-webapp.tenant-2:8 < server: envoy < x-envoy-upstream-service-time: 6 < -* Connection #0 to host test-webapp.tenant-2 left intact +* Connection #0 to host test-webapp.team-bravo-1 left intact RBAC: access denied -# Test tenant-1 -> tenant-2 via external domain (allowed) -EXTERNAL_URL=$(oc get ksvc -n tenant-2 test-webapp -o custom-columns=:.status.url --no-headers) -oc exec deployment/curl -n tenant-1 -it -- curl -ik $EXTERNAL_URL +# Test team-alpha-1 -> team-bravo-1 via external domain (allowed) +EXTERNAL_URL=$(oc get ksvc -n team-bravo-1 test-webapp -o custom-columns=:.status.url --no-headers) +oc exec deployment/curl -n team-alpha-1 -it -- curl -ik $EXTERNAL_URL HTTP/2 200 content-length: 18 @@ -430,9 +429,9 @@ Delete the resources that were created for verification: + [source,terminal] ---- -oc delete deployment/curl -n tenant-1 -oc delete ksvc/test-webapp -n tenant-1 -oc delete ksvc/test-webapp -n tenant-2 +oc delete deployment/curl -n team-alpha-1 +oc delete ksvc/test-webapp -n team-alpha-1 +oc delete ksvc/test-webapp -n team-bravo-1 ---- diff --git a/modules/serverless-common/pages/service-mesh/common-service-mesh-setup.adoc b/modules/serverless-common/pages/service-mesh/common-service-mesh-setup.adoc index 7492f15b..dbc27e0a 100644 --- a/modules/serverless-common/pages/service-mesh/common-service-mesh-setup.adoc +++ b/modules/serverless-common/pages/service-mesh/common-service-mesh-setup.adoc @@ -3,26 +3,22 @@ // Metadata: :description: Setup {serverlessproductname} with {SMProductName} -// TODO - .Assumptions and limitations This page describes the integration of {serverlessproductname} with {smproductname}. In this setup, it is assumed that: -* All Knative internal components, as well as Knative Services, are part of the {smproductshortname} and have istio-sidecars injected. This means, that strict mTLS is enforced within the whole mesh. All requests to Knative Services require an mTLS connection (the client must send its certificate), except calls coming from OpenShift Routing. -* {serverlessproductname} with Kourier disabled can only target *one* istio service mesh. Multiple meshes can be present in the cluster, but {serverlessproductname} will only be available on one of them. -* The mesh that {serverlessproductname} is part of has to be distinct and ideally only for {serverlessproductname} workloads, as additional configuration like `Gateways` might interfere with the automated configuration by {serverlessproductname}. Other namespaces can be part of this Mesh, but must not configure the istio custom resource `Gateway` that could overlap with {serverlessproductname} `knative-local-gateway` and `knative-ingress-gateway`. -* {smproductname} only allows one `Gateway` to claim a wildcard host binding (`hosts: *`) on the same port (`port: 443`). So the `knative-ingress-gateway` and `knative-local-gateway` have to be unique in the Mesh. If another Mesh is already binding this configuration, a separate Mesh has to be created for {serverlessproductname} workloads. -* Cluster external Knative Services are expected to be called via OpenShift Ingress using OpenShift Routes. It is not supported to access istio directly, e.g. by exposing the `istio-ingressgateway` using a Service with type `NodePort` or `LoadBalancer`. -* Changing the target `ServiceMeshMemberRole`, that {serverlessproductname} is part of (meaning moving {serverlessproductname} to another mesh) is not supported. The only way to change the targeted Service Mesh is to uninstall and reinstall {serverlessproductname}. +* All Knative internal components, as well as Knative Services, are part of the {smproductshortname} and have sidecars injection enabled. This means, that strict mTLS is enforced within the whole mesh. All requests to Knative Services require an mTLS connection (the client must send its certificate), except calls coming from OpenShift Routing. +* {serverlessproductname} with Kourier disabled can only target *one* service mesh. Multiple meshes can be present in the cluster, but {serverlessproductname} will only be available on one of them. +* The mesh that {serverlessproductname} is part of has to be distinct and ideally only for {serverlessproductname} workloads, as additional configuration like `Gateways` might interfere with the automated configuration by {serverlessproductname}. Other namespaces can be part of this mesh, but must not configure the {smproductshortname} custom resource `Gateway` that could overlap with {serverlessproductname} `knative-local-gateway` and `knative-ingress-gateway`. +* {smproductname} only allows one `Gateway` to claim a wildcard host binding (`hosts: *`) on the same port (`port: 443`). So the `knative-ingress-gateway` and `knative-local-gateway` have to be unique in the mesh. If another mesh is already binding this configuration, a separate mesh has to be created for {serverlessproductname} workloads. +* Cluster external Knative Services are expected to be called via OpenShift Ingress using OpenShift Routes. It is not supported to access {smproductshortname} directly, e.g. by exposing the `istio-ingressgateway` using a Service with type `NodePort` or `LoadBalancer`. +* Changing the target `ServiceMeshMemberRole`, that {serverlessproductname} is part of (meaning moving {serverlessproductname} to another mesh) is not supported. The only way to change the targeted Service mesh is to uninstall and reinstall {serverlessproductname}. .Prerequisites * You have access to an {product-title} account with cluster administrator access. -* You have created a project or have access to a project with the appropriate roles and permissions to create applications and other workloads in {product-title}. - * Install the OpenShift CLI (`oc`). * Install the {SMProductName} Operator. @@ -34,7 +30,7 @@ Using {ServerlessProductName} with {SMProductShortName} is only supported with { .Installing and configuring {smproductname} -. Create a `ServiceMeshControlPlane` resource in the `istio-system` namespace. Make sure to use the following settings, to make Service Mesh work with Serverless: +. Create a `ServiceMeshControlPlane` resource in the `istio-system` namespace. Make sure to use the following settings, to make Service mesh work with Serverless: + [IMPORTANT] ==== @@ -75,7 +71,7 @@ spec: <1> This enforces strict mTLS in the mesh. Only calls using a valid client-certificate are allowed. <2> OpenShift Serverless has a graceful termination for Knative Services of 30 seconds. The istio-proxy needs to have a longer termination duration to make sure no requests are dropped. <3> Defining a specific selector for the `ingress-gateway` to target only the Knative gateway. -<4> Those ports are called by Kubernetes and cluster monitoring which are not part of the mesh and cannot be called using mTLS, thus those ports are excluded from the Service Mesh. +<4> Those ports are called by Kubernetes and cluster monitoring which are not part of the mesh and cannot be called using mTLS, thus those ports are excluded from the Service mesh. . Add the namespaces that you would like to integrate with {SMProductShortName} to the `ServiceMeshMemberRoll` object as members: @@ -91,7 +87,7 @@ spec: members: <1> - knative-serving - knative-eventing - - + - ---- <1> A list of namespaces to be integrated with {SMProductShortName}. + @@ -248,7 +244,7 @@ spec: "sidecar.istio.io/inject": "true" "sidecar.istio.io/rewriteAppHTTPProbers": "true" ---- -<1> Enables Eventing istio controller to create a `DestinationRule` for each InMemoryChannel or KafkaChannel service. +<1> Enables Eventing Istio controller to create a `DestinationRule` for each InMemoryChannel or KafkaChannel service. <2> Enables sidecar injection for Knative Eventing pods. . Apply the `KnativeEventing` resource: @@ -393,7 +389,7 @@ spec: containers: - image: ---- -<1> A namespace that is part of the Service Mesh member roll. +<1> A namespace that is part of the Service mesh member roll. <2> Instructs Knative Serving to generate an {product-title} pass-through enabled route, so that the certificates you have generated are served through the ingress gateway directly. <3> Injects {SMProductShortName} sidecars into the Knative service pods. + From 05ce8da4849728a7f5d88a93d50732f50b304064 Mon Sep 17 00:00:00 2001 From: Reto Lehmann Date: Tue, 29 Aug 2023 07:55:26 +0200 Subject: [PATCH 15/20] extract text out of Serving image --- .../multi-tenancy-service-mesh-Serving.drawio.svg | 2 +- .../service-mesh/common-service-mesh-network-isolation.adoc | 6 ++++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/modules/serverless-common/assets/images/service-mesh/multi-tenancy-service-mesh-Serving.drawio.svg b/modules/serverless-common/assets/images/service-mesh/multi-tenancy-service-mesh-Serving.drawio.svg index 63f07145..8d977857 100644 --- a/modules/serverless-common/assets/images/service-mesh/multi-tenancy-service-mesh-Serving.drawio.svg +++ b/modules/serverless-common/assets/images/service-mesh/multi-tenancy-service-mesh-Serving.drawio.svg @@ -1,4 +1,4 @@ -







tenant-1
*.tenant-1.svc.cluster.local
tenant-1...







tenant-2
 *.tenant-2.svc.cluster.local
tenant-2...






    knative-serving
knative-serving...






istio-system
istio-system...
Istio Ingress
Gateway
Istio Ingress...
Activator
+ IstioProxy
Activator...
KSVC IstioProxy + QueueProxy
KSVC IstioPro...
KSVC
IstioProxy + QueueProxy
KSVC...
AuthorizationPolicy
from:
    namespaces: ["tenant-1"]
to:
  hosts: ["*.tenant-1.svc.cluster.local"]
AuthorizationPolicy...
AuthorizationPolicy
from:
    namespaces: ["tenant-2"]
to:
  hosts: ["*.tenant-2.svc.cluster.local"]
AuthorizationPolicy...
Legend:
all calls coming from tenant-2 pod
Legend:...
tenant-2 can not call tenant-1 directly. Rejected in tenant-1 istio-proxy.

tenant-2 can not call tenant-1 via ingress-gateway. Rejected in tenant-1 istio-proxy.

tenant-2 can not call tenant-1 via activator. Rejected in activator istio-proxy, as this is the last known place of the "real" source.
Note: as ingress-gateway can always call activator, this includes calls from tenant-2 via ingress-gateway to activator.
tenant-2 can not call tenant-1 directly. Rejected in tenant-1 istio-proxy....
Activator is the last place we know the "real" source
Activator is...Text is not SVG - cannot display \ No newline at end of file +







tenant-1  🔒 (4)
tenant-1  🔒 (4)...







tenant-2
tenant-2...






    knative-serving  🔒 (4)
knative-serving  🔒 (4)...






istio-system
istio-system...
Istio Ingress
Gateway
Istio Ingress...
Activator
+ IstioProxy
Activator...
KSVC IstioProxy + QueueProxy
KSVC IstioPro...
KSVC
IstioProxy + QueueProxy
KSVC...
(1)
(1)
(3)
(3)
(3)
(3)
(2)
(2)
(3)
(3)
(2)
(2)Text is not SVG - cannot display \ No newline at end of file diff --git a/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc b/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc index fb62bc21..1d3d7a82 100644 --- a/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc +++ b/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc @@ -16,6 +16,12 @@ The high-level architecture consists of `AuthorizationPolicies` in the `knative- image::service-mesh/multi-tenancy-service-mesh-Serving.drawio.svg[] +. Workloads in project tenant-2 can not call workloads in project tenant-1 directly. Requests are rejected in the Istio proxies of tenant-1 workloads. +. Workloads in project tenant-2 can not call workloads in project tenant-1 via ingress-gateway. Requests are rejected in the Istio proxies of tenant-1 workloads. +. Workloads in project tenant-2 can not call workloads in project tenant-1 via activator. Requests are rejected in the Istio proxy of the activator component. Note: as ingress-gateway is allowed to call activator, this includes requests from project tenant-2 via ingress-gateway to activator. +. 🔒 `AuthorizationPolicies` are applied in this project + + *Knative Eventing* image::service-mesh/multi-tenancy-service-mesh-Eventing.drawio.svg[] From 9b83905f5827519fbbdf5e8e199d2129776b6243 Mon Sep 17 00:00:00 2001 From: Reto Lehmann Date: Tue, 29 Aug 2023 08:27:12 +0200 Subject: [PATCH 16/20] add pre-install verification steps --- .../common-service-mesh-setup.adoc | 42 +++++++++++++++---- 1 file changed, 35 insertions(+), 7 deletions(-) diff --git a/modules/serverless-common/pages/service-mesh/common-service-mesh-setup.adoc b/modules/serverless-common/pages/service-mesh/common-service-mesh-setup.adoc index dbc27e0a..ff57a948 100644 --- a/modules/serverless-common/pages/service-mesh/common-service-mesh-setup.adoc +++ b/modules/serverless-common/pages/service-mesh/common-service-mesh-setup.adoc @@ -3,18 +3,46 @@ // Metadata: :description: Setup {serverlessproductname} with {SMProductName} -.Assumptions and limitations +This page describes the integration of {serverlessproductname} with {smproductname}. -This page describes the integration of {serverlessproductname} with {smproductname}. In this setup, it is assumed that: +.Assumptions and limitations -* All Knative internal components, as well as Knative Services, are part of the {smproductshortname} and have sidecars injection enabled. This means, that strict mTLS is enforced within the whole mesh. All requests to Knative Services require an mTLS connection (the client must send its certificate), except calls coming from OpenShift Routing. -* {serverlessproductname} with Kourier disabled can only target *one* service mesh. Multiple meshes can be present in the cluster, but {serverlessproductname} will only be available on one of them. -* The mesh that {serverlessproductname} is part of has to be distinct and ideally only for {serverlessproductname} workloads, as additional configuration like `Gateways` might interfere with the automated configuration by {serverlessproductname}. Other namespaces can be part of this mesh, but must not configure the {smproductshortname} custom resource `Gateway` that could overlap with {serverlessproductname} `knative-local-gateway` and `knative-ingress-gateway`. -* {smproductname} only allows one `Gateway` to claim a wildcard host binding (`hosts: *`) on the same port (`port: 443`). So the `knative-ingress-gateway` and `knative-local-gateway` have to be unique in the mesh. If another mesh is already binding this configuration, a separate mesh has to be created for {serverlessproductname} workloads. -* Cluster external Knative Services are expected to be called via OpenShift Ingress using OpenShift Routes. It is not supported to access {smproductshortname} directly, e.g. by exposing the `istio-ingressgateway` using a Service with type `NodePort` or `LoadBalancer`. +* All Knative internal components, as well as Knative Services, are part of the {smproductshortname} and have sidecars injection enabled. This means, that strict mTLS is enforced within the whole mesh. All requests to Knative Services require an mTLS connection (the client must send its certificate), except calls coming from OpenShift Routing. +* {serverlessproductname} with {smproductshortname} integration can only target *one* service mesh. Multiple meshes can be present in the cluster, but {serverlessproductname} will only be available on one of them. * Changing the target `ServiceMeshMemberRole`, that {serverlessproductname} is part of (meaning moving {serverlessproductname} to another mesh) is not supported. The only way to change the targeted Service mesh is to uninstall and reinstall {serverlessproductname}. +.Pre-install verification + +Before you install and configure the {smproductshortname} integration with {serverlessproductname}, please run the following pre-install verifications to make sure, the integration will work properly. + +. Check for conflicting gateways ++ +The mesh that {serverlessproductname} is part of has to be distinct and ideally only for {serverlessproductname} workloads, as additional configuration like `Gateways` might interfere with the {serverlessproductname} gateways `knative-local-gateway` and `knative-ingress-gateway`. {smproductname} only allows one `Gateway` to claim a wildcard host binding (`hosts: ["*"]`) on the same port (`port: 443`). If another `Gateway` is already binding this configuration, a separate mesh has to be created for {serverlessproductname} workloads. To check if an existing `Gateway` is already binding those, run: ++ +[source,terminal] +---- +$ oc get gateway -A -o jsonpath='{range .items[*]}{@.metadata.namespace}{"/"}{@.metadata.name}{" "}{@.spec.servers}{"\n"}{end}' | column -t +knative-serving/knative-ingress-gateway [{"hosts":["*"],"port":{"name":"https","number":443,"protocol":"HTTPS"},"tls":{"credentialName":"wildcard-certs","mode":"SIMPLE"}}] +knative-serving/knative-local-gateway [{"hosts":["*"],"port":{"name":"http","number":8081,"protocol":"HTTP"}}] +---- ++ +This command should not return a `Gateway` that binds `port: 443` and `hosts: ["*"]`, except the `Gateways` in `knative-serving` and `Gateways`, that are part of another {smproductshortname} instance. + +. Check if {smproductshortname} is exposed as type `NodePort` or `LoadBalancer` ++ +Cluster external Knative Services are expected to be called via OpenShift Ingress using OpenShift Routes. It is not supported to access {smproductshortname} directly, e.g. by exposing the `istio-ingressgateway` using a Service with type `NodePort` or `LoadBalancer`. To check how your `istio-ingressgateway` is exposed, run: ++ +[source,terminal] +---- +$ oc get svc -A | grep istio-ingressgateway +istio-system istio-ingressgateway ClusterIP 172.30.46.146 none> 15021/TCP,80/TCP,443/TCP 9m50s +---- ++ +This command should not return a `Service` of type `NodePort` or `LoadBalancer`. + + + .Prerequisites * You have access to an {product-title} account with cluster administrator access. From e9fd1638400be4bc2dff9a7a1fb0bd1591cb1d18 Mon Sep 17 00:00:00 2001 From: Reto Lehmann Date: Thu, 31 Aug 2023 08:36:27 +0200 Subject: [PATCH 17/20] add page for Serving with OCP ingress sharding --- modules/ROOT/nav.adoc | 10 +- ...i-tenancy-service-mesh-Eventing.drawio.svg | 0 ...ti-tenancy-service-mesh-Serving.drawio.svg | 0 ...common-service-mesh-network-isolation.adoc | 0 .../common-service-mesh-setup.adoc | 1 - ...eventing-service-mesh-containersource.adoc | 0 .../eventing-service-mesh-sinkbinding.adoc | 0 .../serving-with-ingress-sharding.adoc | 123 ++++++++++++++++++ 8 files changed, 129 insertions(+), 5 deletions(-) rename modules/{serverless-common => serverless}/assets/images/service-mesh/multi-tenancy-service-mesh-Eventing.drawio.svg (100%) rename modules/{serverless-common => serverless}/assets/images/service-mesh/multi-tenancy-service-mesh-Serving.drawio.svg (100%) rename modules/{serverless-common => serverless}/pages/service-mesh/common-service-mesh-network-isolation.adoc (100%) rename modules/{serverless-common => serverless}/pages/service-mesh/common-service-mesh-setup.adoc (99%) rename modules/{serverless-common => serverless}/pages/service-mesh/eventing-service-mesh-containersource.adoc (100%) rename modules/{serverless-common => serverless}/pages/service-mesh/eventing-service-mesh-sinkbinding.adoc (100%) create mode 100644 modules/serverless/pages/serving/serving-with-ingress-sharding.adoc diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index d83e2ca4..d425021f 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -1,10 +1,12 @@ * xref:index.adoc[Overview] * Serverless ** Using OpenShift Serverless with OpenShift Service Mesh -*** xref:serverless-common:service-mesh/common-service-mesh-setup.adoc[Setup Serverless with OpenShift Service Mesh] -*** xref:serverless-common:service-mesh/common-service-mesh-network-isolation.adoc[Use Service Mesh to isolate network-traffic] -*** xref:serverless-common:service-mesh/eventing-service-mesh-containersource.adoc[Eventing: Using ContainerSource with OpenShift Service Mesh] -*** xref:serverless-common:service-mesh/eventing-service-mesh-sinkbinding.adoc[Eventing: Using SinkBinding with OpenShift Service Mesh] +*** xref:serverless:service-mesh/common-service-mesh-setup.adoc[Setup Serverless with OpenShift Service Mesh] +*** xref:serverless:service-mesh/common-service-mesh-network-isolation.adoc[Use Service Mesh to isolate network-traffic] +*** xref:serverless:service-mesh/eventing-service-mesh-containersource.adoc[Eventing: Using ContainerSource with OpenShift Service Mesh] +*** xref:serverless:service-mesh/eventing-service-mesh-sinkbinding.adoc[Eventing: Using SinkBinding with OpenShift Service Mesh] +** Serving +*** xref:serverless:serving/serving-with-ingress-sharding.adoc[Use Serving with OpenShift ingress sharding] * Serverless Logic ** xref:serverless-logic:about.adoc[About OpenShift Serverless Logic] ** User Guides diff --git a/modules/serverless-common/assets/images/service-mesh/multi-tenancy-service-mesh-Eventing.drawio.svg b/modules/serverless/assets/images/service-mesh/multi-tenancy-service-mesh-Eventing.drawio.svg similarity index 100% rename from modules/serverless-common/assets/images/service-mesh/multi-tenancy-service-mesh-Eventing.drawio.svg rename to modules/serverless/assets/images/service-mesh/multi-tenancy-service-mesh-Eventing.drawio.svg diff --git a/modules/serverless-common/assets/images/service-mesh/multi-tenancy-service-mesh-Serving.drawio.svg b/modules/serverless/assets/images/service-mesh/multi-tenancy-service-mesh-Serving.drawio.svg similarity index 100% rename from modules/serverless-common/assets/images/service-mesh/multi-tenancy-service-mesh-Serving.drawio.svg rename to modules/serverless/assets/images/service-mesh/multi-tenancy-service-mesh-Serving.drawio.svg diff --git a/modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc b/modules/serverless/pages/service-mesh/common-service-mesh-network-isolation.adoc similarity index 100% rename from modules/serverless-common/pages/service-mesh/common-service-mesh-network-isolation.adoc rename to modules/serverless/pages/service-mesh/common-service-mesh-network-isolation.adoc diff --git a/modules/serverless-common/pages/service-mesh/common-service-mesh-setup.adoc b/modules/serverless/pages/service-mesh/common-service-mesh-setup.adoc similarity index 99% rename from modules/serverless-common/pages/service-mesh/common-service-mesh-setup.adoc rename to modules/serverless/pages/service-mesh/common-service-mesh-setup.adoc index ff57a948..8d539719 100644 --- a/modules/serverless-common/pages/service-mesh/common-service-mesh-setup.adoc +++ b/modules/serverless/pages/service-mesh/common-service-mesh-setup.adoc @@ -42,7 +42,6 @@ istio-system istio-ingressgateway ClusterIP 172.30.46.146 none> 15021/TCP, This command should not return a `Service` of type `NodePort` or `LoadBalancer`. - .Prerequisites * You have access to an {product-title} account with cluster administrator access. diff --git a/modules/serverless-common/pages/service-mesh/eventing-service-mesh-containersource.adoc b/modules/serverless/pages/service-mesh/eventing-service-mesh-containersource.adoc similarity index 100% rename from modules/serverless-common/pages/service-mesh/eventing-service-mesh-containersource.adoc rename to modules/serverless/pages/service-mesh/eventing-service-mesh-containersource.adoc diff --git a/modules/serverless-common/pages/service-mesh/eventing-service-mesh-sinkbinding.adoc b/modules/serverless/pages/service-mesh/eventing-service-mesh-sinkbinding.adoc similarity index 100% rename from modules/serverless-common/pages/service-mesh/eventing-service-mesh-sinkbinding.adoc rename to modules/serverless/pages/service-mesh/eventing-service-mesh-sinkbinding.adoc diff --git a/modules/serverless/pages/serving/serving-with-ingress-sharding.adoc b/modules/serverless/pages/serving/serving-with-ingress-sharding.adoc new file mode 100644 index 00000000..6ee2adf6 --- /dev/null +++ b/modules/serverless/pages/serving/serving-with-ingress-sharding.adoc @@ -0,0 +1,123 @@ += Use Serving with {product-title} ingress sharding +:compat-mode!: +// Metadata: +:description: Use Serving with {product-title} ingress sharding + +This page describes how {serverless} Serving can be used with {product-title} ingress sharding. + +.Prerequisites + +* You have access to an {product-title} account with cluster administrator access. + +* You have an existing {product-title} routing setup that uses ingress shards + + +.Setting up {product-title} ingress shards + +{serverless} can be configured to match specific ingress shards with different domains with a label selector. +The following is an example of how your ingress shard could be configured. +The relevant fields are marked: + +[source,yaml] +---- +apiVersion: operator.openshift.io/v1 +kind: IngressController +metadata: + name: ingress-dev <1> + namespace: openshift-ingress-operator +spec: + routeSelector: + matchLabels: + router: dev <2> + domain: "dev.serverless.cluster.example.com" <3> + #... other settings ... +--- +apiVersion: operator.openshift.io/v1 +kind: IngressController +metadata: + name: ingress-prod <1> + namespace: openshift-ingress-operator +spec: + routeSelector: + matchLabels: + router: prod <2> + domain: "prod.serverless.cluster.example.com" <3> + #... other settings ... +---- +<1> The names of your ingress shard, here as an example `dev` and `prod` for different stages +<2> A label selector to match the ingress shard +<3> A custom domain for the ingress shard + +Replace these values with your own configuration. + +.Configuring custom domains in the `KnativeServing` CustomResource + +To match the ingress shards, you need to configure `KnativeServing` to use the same domains and labels as your ingress shards. +For this you need to create or edit your `KnativeServing` resource and add the `spec.config.domain` field: +[source,yaml] +---- +spec: + config: + domain: <1> + dev.serverless.cluster.example.com: | + selector: + router: dev + prod.serverless.cluster.example.com: | + selector: + router: prod +---- +<1> These values have to match to the ones in the ingress shard configuration. + +.Targeting a specific ingress shard in the Knative Service + +You can now target a specific ingress shard in your Knative `Service` resources using a label: +[source,yaml] +---- +apiVersion: serving.knative.dev/v1 +kind: Service +metadata: + name: hello-dev + labels: + router: dev <1> +spec: + template: + spec: + containers: + - image: docker.io/openshift/hello-openshift +--- +apiVersion: serving.knative.dev/v1 +kind: Service +metadata: + name: hello-prod + labels: + router: prod <1> +spec: + template: + spec: + containers: + - image: docker.io/openshift/hello-openshift +---- +<1> This have to match the configuration in `KnativeServing` + +.Verification + +With this setup, your Knative Services should use the correct route and the selected ingress shard: +[source,terminal] +---- +$ oc get ksvc +NAME URL LATESTCREATED LATESTREADY READY REASON +hello-dev https://hello-dev-default.dev.serverless.cluster.example.com hello-dev-00001 hello-dev-00001 True +hello-prod https://hello-prod-default.prod.serverless.cluster.example.com hello-prod-00001 hello-prod-00001 True +---- +[source,terminal] +---- +$ oc get route -n knative-serving-ingress -o jsonpath='{range .items[*]}{@.metadata.name}{" "}{@.spec.host}{" "}{@.status.ingress[*].routerName}{"\n"}{end}' +route-19e6628b-77af-4da0-9b4c-1224934b2250-323461616533 hello-prod-default.prod.serverless.cluster.example.com ingress-prod +route-cb5085d9-b7da-4741-9a56-96c88c6adaaa-373065343266 hello-dev-default.dev.serverless.cluster.example.com ingress-dev +---- + +[NOTE] +==== +Please be aware that even with OpenShift ingress sharding in place, the {serverless} traffic is still routed through a single Knative Ingress Gateway and the activator component in the `knative-serving` project. +Please also take a look at xref:./../service-mesh/common-service-mesh-network-isolation.adoc[] for more information about isolating the network traffic. +==== From 4501ea901261c488148610c63ad2082dda9d3679 Mon Sep 17 00:00:00 2001 From: Reto Lehmann Date: Tue, 12 Sep 2023 15:04:39 +0200 Subject: [PATCH 18/20] add tabs to show verification steps with kn cli --- ...common-service-mesh-network-isolation.adoc | 143 ++++++++++++++---- 1 file changed, 116 insertions(+), 27 deletions(-) diff --git a/modules/serverless/pages/service-mesh/common-service-mesh-network-isolation.adoc b/modules/serverless/pages/service-mesh/common-service-mesh-network-isolation.adoc index 1d3d7a82..dd44a9d9 100644 --- a/modules/serverless/pages/service-mesh/common-service-mesh-network-isolation.adoc +++ b/modules/serverless/pages/service-mesh/common-service-mesh-network-isolation.adoc @@ -285,12 +285,32 @@ The helm chart has several options that can be passed to configure the generated This verification is assuming that we have two tenants with one namespace each, all part of the `ServiceMeshMemberRoll`, configured with resources listed above. We can then use curl to verify the connectivity: -. Deploy Knative Services in both tenants namespaces and a `curl` pod to run test commands: +. Deploy Knative Services in both tenants namespaces: ++ +[tabs] +==== +Using the Knative CLI:: + [source,terminal] ---- # Team Alpha -cat < team-alpha-1 via cluster local domain (allowed) +oc exec deployment/curl -n team-alpha-1 -it -- curl -v http://test-webapp.team-alpha-1:80 + +HTTP/1.1 200 OK +content-length: 18 +content-type: text/plain; charset=utf-8 +date: Wed, 26 Jul 2023 12:49:59 GMT +server: envoy +x-envoy-upstream-service-time: 9 + +Hello Serverless! + + +# Test team-alpha-1 -> team-alpha-1 via external domain (allowed) +oc exec deployment/curl -n team-alpha-1 -it -- curl -ik $(kn service describe test-webapp -o url -n team-alpha-1) + +HTTP/2 200 +content-length: 18 +content-type: text/plain; charset=utf-8 +date: Wed, 26 Jul 2023 12:55:30 GMT +server: istio-envoy +x-envoy-upstream-service-time: 3629 + +Hello Serverless! + + +# Test team-alpha-1 -> team-bravo-1 via cluster local domain (not allowed) +oc exec deployment/curl -n team-alpha-1 -it -- curl -v http://test-webapp.team-bravo-1:80 + +* processing: http://test-webapp.team-bravo-1:80 +* Trying 172.30.73.216:80... +* Connected to test-webapp.team-bravo-1 (172.30.73.216) port 80 +> GET / HTTP/1.1 +> Host: test-webapp.team-bravo-1 +> User-Agent: curl/8.2.0 +> Accept: */* +> +< HTTP/1.1 403 Forbidden +< content-length: 19 +< content-type: text/plain +< date: Wed, 26 Jul 2023 12:55:49 GMT +< server: envoy +< x-envoy-upstream-service-time: 6 +< +* Connection #0 to host test-webapp.team-bravo-1 left intact +RBAC: access denied + + +# Test team-alpha-1 -> team-bravo-1 via external domain (allowed) +oc exec deployment/curl -n team-alpha-1 -it -- curl -ik $(kn service describe test-webapp -o url -n team-bravo-1) + +HTTP/2 200 +content-length: 18 +content-type: text/plain; charset=utf-8 +date: Wed, 26 Jul 2023 12:56:22 GMT +server: istio-envoy +x-envoy-upstream-service-time: 2856 + +Hello Serverless! +---- +Using OC client:: ++ [source,terminal] ---- # Test team-alpha-1 -> team-alpha-1 via cluster local domain (allowed) @@ -428,6 +518,7 @@ x-envoy-upstream-service-time: 2856 Hello Serverless! ---- +==== . Cleanup + @@ -439,5 +530,3 @@ oc delete deployment/curl -n team-alpha-1 oc delete ksvc/test-webapp -n team-alpha-1 oc delete ksvc/test-webapp -n team-bravo-1 ---- - - From e6866aff0de2057b7bfc747d5348156483fa424a Mon Sep 17 00:00:00 2001 From: Pierangelo Di Pilato Date: Mon, 18 Sep 2023 10:31:59 +0200 Subject: [PATCH 19/20] Update eventing diagram to use textual legend and a tenant-sliced version of the diagram Signed-off-by: Pierangelo Di Pilato --- .../multi-tenancy-service-mesh-Eventing.drawio.svg | 2 +- .../common-service-mesh-network-isolation.adoc | 7 +++++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/modules/serverless/assets/images/service-mesh/multi-tenancy-service-mesh-Eventing.drawio.svg b/modules/serverless/assets/images/service-mesh/multi-tenancy-service-mesh-Eventing.drawio.svg index 430ed33e..2ab99e8e 100644 --- a/modules/serverless/assets/images/service-mesh/multi-tenancy-service-mesh-Eventing.drawio.svg +++ b/modules/serverless/assets/images/service-mesh/multi-tenancy-service-mesh-Eventing.drawio.svg @@ -1,4 +1,4 @@ -







tenant-1
*.tenant-1.svc.cluster.local
tenant-1...







tenant-2
 *.tenant-2.svc.cluster.local
tenant-2...






    knative-eventing
knative-eventing...
Receiver
Receiver
Sender
Sender
1)
1)
AuthorizationPolicy
from:
    namespaces: ["tenant-1"]
to:
  hosts: ["*.tenant-1.svc.cluster.local"]
  paths: ["/tenant-1/*"]
AuthorizationPolicy...
AuthorizationPolicy
from:
    namespaces: ["tenant-2"]
to:
  hosts: ["*.tenant-2.svc.cluster.local"]
  paths: ["/tenant-2/*"]
AuthorizationPolicy...
Receiver is the last place we know the "real" source.

Sender sends "Kn-Namespace" header with the resource's namespace as value
Receiver is the last place w...
Legend
Legend
tenant-1: all shown paths are allowed
tenant-1: all shown paths are allow...
tenant-2: 
1) tenant-2 can call "receiver" directly and send events to resources in tenant-1 namespace.
2) tenant-2 can not call tenant-1 directly. Rejected in tenant-1 istio-proxy.
3) sources, triggers, and subscribers in tenant-2 namespaces can not call tenant-1 workloads. Rejected in tenant-1 istio-proxy.
tenant-2:...
3)
3)
AuthorizationPolicy
from:
    namespaces: ["knative-eventing"]
when:
    key: request.headers[Kn-Namespace]
    values: ["tenant-1"]
AuthorizationPolicy...
AuthorizationPolicy
from:
    namespaces: ["knative-eventing"]
when:
    key: request.headers[Kn-Namespace]
    values: ["tenant-2"]
AuthorizationPolicy...
ksvc
ksvc
svc
svc
ksvc
ksvc
svc
svcText is not SVG - cannot display \ No newline at end of file +






🔒 (6)

tenant-2
*.tenant-2.svc.cluster.local
🔒 (6)...






🔒 (6)

tenant-1
 *.tenant-1.svc.cluster.local
🔒 (6)...






    knative-eventing 🔒 (6)
knative-eventing 🔒 (6)...
Receiver
Receiver
Sender
Sender
1)
1)
2)
2)
AuthorizationPolicy
from:
    namespaces: ["tenant-1"]
to:
  hosts: ["*.tenant-1.svc.cluster.local"]
  paths: ["/tenant-1/*"]
AuthorizationPolicy...
AuthorizationPolicy
from:
    namespaces: ["tenant-2"]
to:
  hosts: ["*.tenant-2.svc.cluster.local"]
  paths: ["/tenant-2/*"]
AuthorizationPolicy...
4)
4)
AuthorizationPolicy
from:
    namespaces: ["knative-eventing"]
when:
    key: request.headers[Kn-Namespace]
    values: ["tenant-2"]
AuthorizationPolicy...
AuthorizationPolicy
from:
    namespaces: ["knative-eventing"]
when:
    key: request.headers[Kn-Namespace]
    values: ["tenant-1"]
AuthorizationPolicy...
5)
5)
ksvc
ksvc
svc
svc
ksvc
ksvc
svc
svc
tenant-1
tenant-1
tenant-2
tenant-2
tenant-1
tenant-1
tenant-2
tenant-2
3)
3)
Sender sends "Kn-Namespace" header with the resource's namespace as value, so that tenant's proxies can allow or deny the requests
Sender sends "Kn-Namespace" head...Text is not SVG - cannot display \ No newline at end of file diff --git a/modules/serverless/pages/service-mesh/common-service-mesh-network-isolation.adoc b/modules/serverless/pages/service-mesh/common-service-mesh-network-isolation.adoc index 1d3d7a82..df8362ab 100644 --- a/modules/serverless/pages/service-mesh/common-service-mesh-network-isolation.adoc +++ b/modules/serverless/pages/service-mesh/common-service-mesh-network-isolation.adoc @@ -26,6 +26,13 @@ image::service-mesh/multi-tenancy-service-mesh-Serving.drawio.svg[] image::service-mesh/multi-tenancy-service-mesh-Eventing.drawio.svg[] +. Workloads in tenant-1 project can send events to Eventing brokers, channels, and sinks endpoints in the tenant-1 project. +. Eventing sources, triggers and subscriptions in project tenant-1 can send events to workloads in project tenant-1. +. Workloads in tenant-1 project can _not_ send events to Eventing brokers, channels, and sinks endpoints in the tenant-2 project. +. Eventing sources, triggers and subscriptions in project tenant-2 can _not_ send events to workloads in project tenant-1. +. Workloads in project tenant-1 can not call workloads in project tenant-2 as well as workloads in project tenant-2 can not call workloads in project tenant-1. +. 🔒 `AuthorizationPolicies` are applied in this project + .Prerequisites * You have access to an {product-title} account with cluster administrator access. From 7ccf02e64356620dd70c4397f757cb8b78d006c3 Mon Sep 17 00:00:00 2001 From: Reto Lehmann Date: Mon, 18 Sep 2023 10:54:35 +0200 Subject: [PATCH 20/20] add authorization policy symbol to tenant-2 --- .../service-mesh/multi-tenancy-service-mesh-Serving.drawio.svg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/serverless/assets/images/service-mesh/multi-tenancy-service-mesh-Serving.drawio.svg b/modules/serverless/assets/images/service-mesh/multi-tenancy-service-mesh-Serving.drawio.svg index 8d977857..58821456 100644 --- a/modules/serverless/assets/images/service-mesh/multi-tenancy-service-mesh-Serving.drawio.svg +++ b/modules/serverless/assets/images/service-mesh/multi-tenancy-service-mesh-Serving.drawio.svg @@ -1,4 +1,4 @@ -







tenant-1  🔒 (4)
tenant-1  🔒 (4)...







tenant-2
tenant-2...






    knative-serving  🔒 (4)
knative-serving  🔒 (4)...






istio-system
istio-system...
Istio Ingress
Gateway
Istio Ingress...
Activator
+ IstioProxy
Activator...
KSVC IstioProxy + QueueProxy
KSVC IstioPro...
KSVC
IstioProxy + QueueProxy
KSVC...
(1)
(1)
(3)
(3)
(3)
(3)
(2)
(2)
(3)
(3)
(2)
(2)Text is not SVG - cannot display \ No newline at end of file +







tenant-1  🔒 (4)
tenant-1  🔒 (4)...







tenant-2  🔒 (4)
tenant-2  🔒 (4)...






    knative-serving  🔒 (4)
knative-serving  🔒 (4)...






istio-system
istio-system...
Istio Ingress
Gateway
Istio Ingress...
Activator
+ IstioProxy
Activator...
KSVC IstioProxy + QueueProxy
KSVC IstioPro...
KSVC
IstioProxy + QueueProxy
KSVC...
(1)
(1)
(3)
(3)
(3)
(3)
(2)
(2)
(3)
(3)
(2)
(2)Text is not SVG - cannot display \ No newline at end of file