diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index e90ae53c..d425021f 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -1,9 +1,12 @@ * xref:index.adoc[Overview] -* Serverless Eventing -** Using Eventing with OpenShift Service Mesh -*** xref:serverless-eventing:service-mesh/eventing-service-mesh-setup.adoc[Setup Eventing with OpenShift Service Mesh] -*** xref:serverless-eventing:service-mesh/eventing-service-mesh-containersource.adoc[Using ContainerSource with OpenShift Service Mesh] -*** xref:serverless-eventing:service-mesh/eventing-service-mesh-sinkbinding.adoc[Using SinkBinding with OpenShift Service Mesh] +* Serverless +** Using OpenShift Serverless with OpenShift Service Mesh +*** xref:serverless:service-mesh/common-service-mesh-setup.adoc[Setup Serverless with OpenShift Service Mesh] +*** xref:serverless:service-mesh/common-service-mesh-network-isolation.adoc[Use Service Mesh to isolate network-traffic] +*** xref:serverless:service-mesh/eventing-service-mesh-containersource.adoc[Eventing: Using ContainerSource with OpenShift Service Mesh] +*** xref:serverless:service-mesh/eventing-service-mesh-sinkbinding.adoc[Eventing: Using SinkBinding with OpenShift Service Mesh] +** Serving +*** xref:serverless:serving/serving-with-ingress-sharding.adoc[Use Serving with OpenShift ingress sharding] * Serverless Logic ** xref:serverless-logic:about.adoc[About OpenShift Serverless Logic] ** User Guides diff --git a/modules/serverless/assets/images/service-mesh/multi-tenancy-service-mesh-Eventing.drawio.svg b/modules/serverless/assets/images/service-mesh/multi-tenancy-service-mesh-Eventing.drawio.svg new file mode 100644 index 00000000..2ab99e8e --- /dev/null +++ b/modules/serverless/assets/images/service-mesh/multi-tenancy-service-mesh-Eventing.drawio.svg @@ -0,0 +1,4 @@ + + + +






🔒 (6)

tenant-2
*.tenant-2.svc.cluster.local
🔒 (6)...






🔒 (6)

tenant-1
 *.tenant-1.svc.cluster.local
🔒 (6)...






    knative-eventing 🔒 (6)
knative-eventing 🔒 (6)...
Receiver
Receiver
Sender
Sender
1)
1)
2)
2)
AuthorizationPolicy
from:
    namespaces: ["tenant-1"]
to:
  hosts: ["*.tenant-1.svc.cluster.local"]
  paths: ["/tenant-1/*"]
AuthorizationPolicy...
AuthorizationPolicy
from:
    namespaces: ["tenant-2"]
to:
  hosts: ["*.tenant-2.svc.cluster.local"]
  paths: ["/tenant-2/*"]
AuthorizationPolicy...
4)
4)
AuthorizationPolicy
from:
    namespaces: ["knative-eventing"]
when:
    key: request.headers[Kn-Namespace]
    values: ["tenant-2"]
AuthorizationPolicy...
AuthorizationPolicy
from:
    namespaces: ["knative-eventing"]
when:
    key: request.headers[Kn-Namespace]
    values: ["tenant-1"]
AuthorizationPolicy...
5)
5)
ksvc
ksvc
svc
svc
ksvc
ksvc
svc
svc
tenant-1
tenant-1
tenant-2
tenant-2
tenant-1
tenant-1
tenant-2
tenant-2
3)
3)
Sender sends "Kn-Namespace" header with the resource's namespace as value, so that tenant's proxies can allow or deny the requests
Sender sends "Kn-Namespace" head...
Text is not SVG - cannot display
\ No newline at end of file diff --git a/modules/serverless/assets/images/service-mesh/multi-tenancy-service-mesh-Serving.drawio.svg b/modules/serverless/assets/images/service-mesh/multi-tenancy-service-mesh-Serving.drawio.svg new file mode 100644 index 00000000..58821456 --- /dev/null +++ b/modules/serverless/assets/images/service-mesh/multi-tenancy-service-mesh-Serving.drawio.svg @@ -0,0 +1,4 @@ + + + +







tenant-1  🔒 (4)
tenant-1  🔒 (4)...







tenant-2  🔒 (4)
tenant-2  🔒 (4)...






    knative-serving  🔒 (4)
knative-serving  🔒 (4)...






istio-system
istio-system...
Istio Ingress
Gateway
Istio Ingress...
Activator
+ IstioProxy
Activator...
KSVC IstioProxy + QueueProxy
KSVC IstioPro...
KSVC
IstioProxy + QueueProxy
KSVC...
(1)
(1)
(3)
(3)
(3)
(3)
(2)
(2)
(3)
(3)
(2)
(2)
Text is not SVG - cannot display
\ No newline at end of file diff --git a/modules/serverless/pages/service-mesh/common-service-mesh-network-isolation.adoc b/modules/serverless/pages/service-mesh/common-service-mesh-network-isolation.adoc new file mode 100644 index 00000000..a6e4a7c3 --- /dev/null +++ b/modules/serverless/pages/service-mesh/common-service-mesh-network-isolation.adoc @@ -0,0 +1,539 @@ += Use {smproductname} to isolate network traffic with {serverlessproductname} +:compat-mode!: +// Metadata: +:description: Use {smproductname} to isolate network traffic with {serverlessproductname} + +{smproductshortname} can be used to isolate network-traffic between tenants on a shared OpenShift cluster using {smproductshortname} `AuthorizationPolicy` resources. +{serverlessproductname} can also leverage this, using several {smproductshortname} resources. + +.Tenants +A tenant is a group of one or multiple OpenShift projects that can access each other over the network on a shared OpenShift cluster. + +.High-level architecture +The high-level architecture consists of `AuthorizationPolicies` in the `knative-serving`, `knative-eventing` and the tenants namespaces, while all components are part of the {smproductshortname}. The injected {smproductshortname} sidecars take care of enforcing those rules to isolate network traffic between tenants. + +*Knative Serving* + +image::service-mesh/multi-tenancy-service-mesh-Serving.drawio.svg[] + +. Workloads in project tenant-2 can not call workloads in project tenant-1 directly. Requests are rejected in the Istio proxies of tenant-1 workloads. +. Workloads in project tenant-2 can not call workloads in project tenant-1 via ingress-gateway. Requests are rejected in the Istio proxies of tenant-1 workloads. +. Workloads in project tenant-2 can not call workloads in project tenant-1 via activator. Requests are rejected in the Istio proxy of the activator component. Note: as ingress-gateway is allowed to call activator, this includes requests from project tenant-2 via ingress-gateway to activator. +. 🔒 `AuthorizationPolicies` are applied in this project + + +*Knative Eventing* + +image::service-mesh/multi-tenancy-service-mesh-Eventing.drawio.svg[] + +. Workloads in tenant-1 project can send events to Eventing brokers, channels, and sinks endpoints in the tenant-1 project. +. Eventing sources, triggers and subscriptions in project tenant-1 can send events to workloads in project tenant-1. +. Workloads in tenant-1 project can _not_ send events to Eventing brokers, channels, and sinks endpoints in the tenant-2 project. +. Eventing sources, triggers and subscriptions in project tenant-2 can _not_ send events to workloads in project tenant-1. +. Workloads in project tenant-1 can not call workloads in project tenant-2 as well as workloads in project tenant-2 can not call workloads in project tenant-1. +. 🔒 `AuthorizationPolicies` are applied in this project + +.Prerequisites + +* You have access to an {product-title} account with cluster administrator access. + +* You have set up {serverlessproductname} and {smproductname} as documented in xref:./common-service-mesh-setup.adoc[]. + +* You have created one or multiple OpenShift projects for each tenant. + + +.Securing the {smproductshortname} + +. As in the set-up documentation, make sure that all your tenants OpenShift projects are part of the same `ServiceMeshMemberRoll` object as members: ++ +[source,yaml] +---- +apiVersion: maistra.io/v1 +kind: ServiceMeshMemberRoll +metadata: + name: default + namespace: istio-system +spec: + members: + - knative-serving # static value, needs to be here, see setup page + - knative-eventing # static value, needs to be here, see setup page + - team-alpha-1 # example OpenShift project that belongs to the team-alpha tenant + - team-alpha-2 # example OpenShift project that belongs th the team-alpha tenant + - team-bravo-1 # example OpenShift project that belongs to the team-bravo tenant + - team-bravo-2 # example OpenShift project that belongs th the team-bravo tenant +---- ++ +All the OpenShift projects that are part of the mesh must enforce mTLS in strict mode. This forces Istio to only accept connections with a client-certificate present and allows the {smproductshortname} sidecar to validate the origin using `AuthorizationPolicy`. ++ +. For {serverlessproductname} internal components to work, the several `AuthorizationPolicies` are necessary in the `knative-serving` and the `knative-eventing` namespace: ++ +[source,yaml] +---- +apiVersion: security.istio.io/v1beta1 +kind: AuthorizationPolicy +metadata: + name: deny-all-by-default + namespace: knative-eventing +spec: { } +--- +apiVersion: security.istio.io/v1beta1 +kind: AuthorizationPolicy +metadata: + name: deny-all-by-default + namespace: knative-serving +spec: { } +--- +apiVersion: security.istio.io/v1beta1 +kind: AuthorizationPolicy +metadata: + name: allow-mt-channel-based-broker-ingress-to-imc-dispatcher + namespace: knative-eventing +spec: + action: ALLOW + selector: + matchLabels: + app.kubernetes.io/component: "imc-dispatcher" + rules: + - from: + - source: + namespaces: [ "knative-eventing" ] + principals: [ "cluster.local/ns/knative-eventing/sa/mt-broker-ingress" ] + to: + - operation: + methods: [ "POST" ] +--- +apiVersion: security.istio.io/v1beta1 +kind: AuthorizationPolicy +metadata: + name: allow-mt-channel-based-broker-ingress-to-kafka-channel + namespace: knative-eventing +spec: + action: ALLOW + selector: + matchLabels: + app.kubernetes.io/component: "kafka-channel-receiver" + rules: + - from: + - source: + namespaces: [ "knative-eventing" ] + principals: [ "cluster.local/ns/knative-eventing/sa/mt-broker-ingress" ] + to: + - operation: + methods: [ "POST" ] +--- +apiVersion: security.istio.io/v1beta1 +kind: AuthorizationPolicy +metadata: + name: allow-kafka-channel-to-mt-channel-based-broker-filter + namespace: knative-eventing +spec: + action: ALLOW + selector: + matchLabels: + app.kubernetes.io/component: "broker-filter" + rules: + - from: + - source: + namespaces: [ "knative-eventing" ] + principals: [ "cluster.local/ns/knative-eventing/sa/knative-kafka-channel-data-plane" ] + to: + - operation: + methods: [ "POST" ] +--- +apiVersion: security.istio.io/v1beta1 +kind: AuthorizationPolicy +metadata: + name: allow-imc-to-mt-channel-based-broker-filter + namespace: knative-eventing +spec: + action: ALLOW + selector: + matchLabels: + app.kubernetes.io/component: "broker-filter" + rules: + - from: + - source: + namespaces: [ "knative-eventing" ] + principals: [ "cluster.local/ns/knative-eventing/sa/imc-dispatcher" ] + to: + - operation: + methods: [ "POST" ] +--- +apiVersion: security.istio.io/v1beta1 +kind: AuthorizationPolicy +metadata: + name: allow-probe-kafka-broker-receiver + namespace: knative-eventing +spec: + action: ALLOW + selector: + matchLabels: + app.kubernetes.io/component: "kafka-broker-receiver" + rules: + - from: + - source: + namespaces: [ "knative-eventing" ] + principals: [ "cluster.local/ns/knative-eventing/sa/kafka-controller" ] + to: + - operation: + methods: [ "GET" ] +--- +apiVersion: security.istio.io/v1beta1 +kind: AuthorizationPolicy +metadata: + name: allow-probe-kafka-sink-receiver + namespace: knative-eventing +spec: + action: ALLOW + selector: + matchLabels: + app.kubernetes.io/component: "kafka-sink-receiver" + rules: + - from: + - source: + namespaces: [ "knative-eventing" ] + principals: [ "cluster.local/ns/knative-eventing/sa/kafka-controller" ] + to: + - operation: + methods: [ "GET" ] +--- +apiVersion: security.istio.io/v1beta1 +kind: AuthorizationPolicy +metadata: + name: allow-probe-kafka-channel-receiver + namespace: knative-eventing +spec: + action: ALLOW + selector: + matchLabels: + app.kubernetes.io/component: "kafka-channel-receiver" + rules: + - from: + - source: + namespaces: [ "knative-eventing" ] + principals: [ "cluster.local/ns/knative-eventing/sa/kafka-controller" ] + to: + - operation: + methods: [ "GET" ] +--- +apiVersion: security.istio.io/v1beta1 +kind: AuthorizationPolicy +metadata: + name: allow-traffic-to-activator + namespace: knative-serving +spec: + selector: + matchLabels: + app: activator + action: ALLOW + rules: + - from: + - source: + namespaces: [ "knative-serving", "istio-system" ] +--- +apiVersion: security.istio.io/v1beta1 +kind: AuthorizationPolicy +metadata: + name: allow-traffic-to-autoscaler + namespace: knative-serving +spec: + selector: + matchLabels: + app: autoscaler + action: ALLOW + rules: + - from: + - source: + namespaces: [ "knative-serving" ] +---- +These policies restrict the access rules for the network communication between OpenShift Serverless' system components. In detail, they enforce the following rules: +- Deny all traffic that is not explicitly allowed in `knative-serving` and `knative-eventing` +- Allow traffic from `istio-system` and `knative-serving` to activator +- Allow traffic from `knative-serving` to autoscaler +- Allow health probes for Apache Kafka components in `knative-eventing` +- Allow internal traffic for channel based brokers in `knative-eventing` ++ +Make sure to apply all those rules to your cluster with: ++ +[source,terminal] +---- +$ oc apply -f +---- + +. With this set up in place, cluster administrators can use their own `AuthorizationPolicies` to define which OpenShift projects can communicate with each other. Every OpenShift project of a tenant needs: +- One `AuthorizationPolicy` limiting directly incoming traffic to the tenants OpenShift project +- One `AuthorizationPolicy` limiting incoming traffic via the activator component of {serverlessproductname} that runs in the `knative-serving` OpenShift project +- One `AuthorizationPolicy` allowing Kubernetes to call `PreStopHooks` on Knative Services ++ +As it is a cumbersome task to create all those policies by hand, you can use our link:https://github.com/openshift-knative/knative-istio-authz-chart[helm based generator] to create the necessary resources for each tenant: ++ +[source,terminal] +.Create resources per tenant with helm +---- +helm template oci://quay.io/openshift-knative/knative-istio-authz-onboarding --version 1.31.0 --set "name=team-alpha" --set "namespaces={team-alpha-1,team-alpha-2}" > team-alpha.yaml +helm template oci://quay.io/openshift-knative/knative-istio-authz-onboarding --version 1.31.0 --set "name=team-bravo" --set "namespaces={team-bravo-1,team-bravo-2}" > team-bravo.yaml +---- ++ +And apply the generated resources to your cluster: ++ +[source,terminal] +---- +$ oc apply -f +---- ++ +[NOTE] +==== +The helm chart has several options that can be passed to configure the generated resources. Please refer to the link:https://github.com/openshift-knative/knative-istio-authz-chart/blob/main/values.yaml[values.yaml] for a full reference. +==== + + +.Verifying the configuration + +This verification is assuming that we have two tenants with one namespace each, all part of the `ServiceMeshMemberRoll`, configured with resources listed above. +We can then use curl to verify the connectivity: + +. Deploy Knative Services in both tenants namespaces: ++ +[tabs] +==== +Using the Knative CLI:: ++ +[source,terminal] +---- +# Team Alpha +kn service create test-webapp -n team-alpha-1 \ + --annotation-service serving.knative.openshift.io/enablePassthrough=true \ + --annotation-revision sidecar.istio.io/inject=true \ + --env RESPONSE="Hello Serverless" \ + --image docker.io/openshift/hello-openshift + +# Team Bravo +kn service create test-webapp -n team-bravo-1 \ + --annotation-service serving.knative.openshift.io/enablePassthrough=true \ + --annotation-revision sidecar.istio.io/inject=true \ + --env RESPONSE="Hello Serverless" \ + --image docker.io/openshift/hello-openshift +---- +Using YAML:: ++ +[source,yaml] +---- +apiVersion: serving.knative.dev/v1 +kind: Service +metadata: + name: test-webapp + namespace: team-alpha-1 + annotations: + serving.knative.openshift.io/enablePassthrough: "true" +spec: + template: + metadata: + annotations: + sidecar.istio.io/inject: 'true' + spec: + containers: + - image: docker.io/openshift/hello-openshift + env: + - name: RESPONSE + value: "Hello Serverless!" +--- +apiVersion: serving.knative.dev/v1 +kind: Service +metadata: + name: test-webapp + namespace: team-bravo-1 + annotations: + serving.knative.openshift.io/enablePassthrough: "true" +spec: + template: + metadata: + annotations: + sidecar.istio.io/inject: 'true' + spec: + containers: + - image: docker.io/openshift/hello-openshift + env: + - name: RESPONSE + value: "Hello Serverless!" +---- +==== + +. Deploy a `curl` pod to test the connections: ++ +[source,terminal] +---- +cat < team-alpha-1 via cluster local domain (allowed) +oc exec deployment/curl -n team-alpha-1 -it -- curl -v http://test-webapp.team-alpha-1:80 + +HTTP/1.1 200 OK +content-length: 18 +content-type: text/plain; charset=utf-8 +date: Wed, 26 Jul 2023 12:49:59 GMT +server: envoy +x-envoy-upstream-service-time: 9 + +Hello Serverless! + + +# Test team-alpha-1 -> team-alpha-1 via external domain (allowed) +oc exec deployment/curl -n team-alpha-1 -it -- curl -ik $(kn service describe test-webapp -o url -n team-alpha-1) + +HTTP/2 200 +content-length: 18 +content-type: text/plain; charset=utf-8 +date: Wed, 26 Jul 2023 12:55:30 GMT +server: istio-envoy +x-envoy-upstream-service-time: 3629 + +Hello Serverless! + + +# Test team-alpha-1 -> team-bravo-1 via cluster local domain (not allowed) +oc exec deployment/curl -n team-alpha-1 -it -- curl -v http://test-webapp.team-bravo-1:80 + +* processing: http://test-webapp.team-bravo-1:80 +* Trying 172.30.73.216:80... +* Connected to test-webapp.team-bravo-1 (172.30.73.216) port 80 +> GET / HTTP/1.1 +> Host: test-webapp.team-bravo-1 +> User-Agent: curl/8.2.0 +> Accept: */* +> +< HTTP/1.1 403 Forbidden +< content-length: 19 +< content-type: text/plain +< date: Wed, 26 Jul 2023 12:55:49 GMT +< server: envoy +< x-envoy-upstream-service-time: 6 +< +* Connection #0 to host test-webapp.team-bravo-1 left intact +RBAC: access denied + + +# Test team-alpha-1 -> team-bravo-1 via external domain (allowed) +oc exec deployment/curl -n team-alpha-1 -it -- curl -ik $(kn service describe test-webapp -o url -n team-bravo-1) + +HTTP/2 200 +content-length: 18 +content-type: text/plain; charset=utf-8 +date: Wed, 26 Jul 2023 12:56:22 GMT +server: istio-envoy +x-envoy-upstream-service-time: 2856 + +Hello Serverless! +---- +Using OC client:: ++ +[source,terminal] +---- +# Test team-alpha-1 -> team-alpha-1 via cluster local domain (allowed) +oc exec deployment/curl -n team-alpha-1 -it -- curl -v http://test-webapp.team-alpha-1:80 + +HTTP/1.1 200 OK +content-length: 18 +content-type: text/plain; charset=utf-8 +date: Wed, 26 Jul 2023 12:49:59 GMT +server: envoy +x-envoy-upstream-service-time: 9 + +Hello Serverless! + + +# Test team-alpha-1 -> team-alpha-1 via external domain (allowed) +EXTERNAL_URL=$(oc get ksvc -n team-alpha-1 test-webapp -o custom-columns=:.status.url --no-headers) +oc exec deployment/curl -n team-alpha-1 -it -- curl -ik $EXTERNAL_URL + +HTTP/2 200 +content-length: 18 +content-type: text/plain; charset=utf-8 +date: Wed, 26 Jul 2023 12:55:30 GMT +server: istio-envoy +x-envoy-upstream-service-time: 3629 + +Hello Serverless! + + +# Test team-alpha-1 -> team-bravo-1 via cluster local domain (not allowed) +oc exec deployment/curl -n team-alpha-1 -it -- curl -v http://test-webapp.team-bravo-1:80 + +* processing: http://test-webapp.team-bravo-1:80 +* Trying 172.30.73.216:80... +* Connected to test-webapp.team-bravo-1 (172.30.73.216) port 80 +> GET / HTTP/1.1 +> Host: test-webapp.team-bravo-1 +> User-Agent: curl/8.2.0 +> Accept: */* +> +< HTTP/1.1 403 Forbidden +< content-length: 19 +< content-type: text/plain +< date: Wed, 26 Jul 2023 12:55:49 GMT +< server: envoy +< x-envoy-upstream-service-time: 6 +< +* Connection #0 to host test-webapp.team-bravo-1 left intact +RBAC: access denied + + +# Test team-alpha-1 -> team-bravo-1 via external domain (allowed) +EXTERNAL_URL=$(oc get ksvc -n team-bravo-1 test-webapp -o custom-columns=:.status.url --no-headers) +oc exec deployment/curl -n team-alpha-1 -it -- curl -ik $EXTERNAL_URL + +HTTP/2 200 +content-length: 18 +content-type: text/plain; charset=utf-8 +date: Wed, 26 Jul 2023 12:56:22 GMT +server: istio-envoy +x-envoy-upstream-service-time: 2856 + +Hello Serverless! +---- +==== + +. Cleanup ++ +Delete the resources that were created for verification: ++ +[source,terminal] +---- +oc delete deployment/curl -n team-alpha-1 +oc delete ksvc/test-webapp -n team-alpha-1 +oc delete ksvc/test-webapp -n team-bravo-1 +---- diff --git a/modules/serverless-eventing/pages/service-mesh/eventing-service-mesh-setup.adoc b/modules/serverless/pages/service-mesh/common-service-mesh-setup.adoc similarity index 56% rename from modules/serverless-eventing/pages/service-mesh/eventing-service-mesh-setup.adoc rename to modules/serverless/pages/service-mesh/common-service-mesh-setup.adoc index 08810339..8d539719 100644 --- a/modules/serverless-eventing/pages/service-mesh/eventing-service-mesh-setup.adoc +++ b/modules/serverless/pages/service-mesh/common-service-mesh-setup.adoc @@ -1,28 +1,105 @@ -= Setup Eventing with {SMProductName} += Setup {serverlessproductname} with {SMProductName} :compat-mode!: // Metadata: -:description: Setup Eventing with {SMProductName} +:description: Setup {serverlessproductname} with {SMProductName} + +This page describes the integration of {serverlessproductname} with {smproductname}. + +.Assumptions and limitations + +* All Knative internal components, as well as Knative Services, are part of the {smproductshortname} and have sidecars injection enabled. This means, that strict mTLS is enforced within the whole mesh. All requests to Knative Services require an mTLS connection (the client must send its certificate), except calls coming from OpenShift Routing. +* {serverlessproductname} with {smproductshortname} integration can only target *one* service mesh. Multiple meshes can be present in the cluster, but {serverlessproductname} will only be available on one of them. +* Changing the target `ServiceMeshMemberRole`, that {serverlessproductname} is part of (meaning moving {serverlessproductname} to another mesh) is not supported. The only way to change the targeted Service mesh is to uninstall and reinstall {serverlessproductname}. + + +.Pre-install verification + +Before you install and configure the {smproductshortname} integration with {serverlessproductname}, please run the following pre-install verifications to make sure, the integration will work properly. + +. Check for conflicting gateways ++ +The mesh that {serverlessproductname} is part of has to be distinct and ideally only for {serverlessproductname} workloads, as additional configuration like `Gateways` might interfere with the {serverlessproductname} gateways `knative-local-gateway` and `knative-ingress-gateway`. {smproductname} only allows one `Gateway` to claim a wildcard host binding (`hosts: ["*"]`) on the same port (`port: 443`). If another `Gateway` is already binding this configuration, a separate mesh has to be created for {serverlessproductname} workloads. To check if an existing `Gateway` is already binding those, run: ++ +[source,terminal] +---- +$ oc get gateway -A -o jsonpath='{range .items[*]}{@.metadata.namespace}{"/"}{@.metadata.name}{" "}{@.spec.servers}{"\n"}{end}' | column -t +knative-serving/knative-ingress-gateway [{"hosts":["*"],"port":{"name":"https","number":443,"protocol":"HTTPS"},"tls":{"credentialName":"wildcard-certs","mode":"SIMPLE"}}] +knative-serving/knative-local-gateway [{"hosts":["*"],"port":{"name":"http","number":8081,"protocol":"HTTP"}}] +---- ++ +This command should not return a `Gateway` that binds `port: 443` and `hosts: ["*"]`, except the `Gateways` in `knative-serving` and `Gateways`, that are part of another {smproductshortname} instance. + +. Check if {smproductshortname} is exposed as type `NodePort` or `LoadBalancer` ++ +Cluster external Knative Services are expected to be called via OpenShift Ingress using OpenShift Routes. It is not supported to access {smproductshortname} directly, e.g. by exposing the `istio-ingressgateway` using a Service with type `NodePort` or `LoadBalancer`. To check how your `istio-ingressgateway` is exposed, run: ++ +[source,terminal] +---- +$ oc get svc -A | grep istio-ingressgateway +istio-system istio-ingressgateway ClusterIP 172.30.46.146 none> 15021/TCP,80/TCP,443/TCP 9m50s +---- ++ +This command should not return a `Service` of type `NodePort` or `LoadBalancer`. -// TODO .Prerequisites * You have access to an {product-title} account with cluster administrator access. -* You have created a project or have access to a project with the appropriate roles and permissions to create applications and other workloads in {product-title}. +* Install the OpenShift CLI (`oc`). -* Install the {SMProductName} Operator and create a `ServiceMeshControlPlane` resource in the `istio-system` namespace. If you want to use mTLS functionality, you must also set the `spec.security.dataPlane.mtls` field for the `ServiceMeshControlPlane` resource to `true`. +* Install the {SMProductName} Operator. + [IMPORTANT] ==== Using {ServerlessProductName} with {SMProductShortName} is only supported with {SMProductName} version 2.0.5 or later. ==== -* Install the {ServerlessOperatorName}. +.Installing and configuring {smproductname} -* Install the OpenShift CLI (`oc`). +. Create a `ServiceMeshControlPlane` resource in the `istio-system` namespace. Make sure to use the following settings, to make Service mesh work with Serverless: ++ +[IMPORTANT] +==== +If you have an existing `ServiceMeshControlPlane`, make sure that you have the same configuration applied. +==== ++ +[source,yaml] +---- +apiVersion: maistra.io/v2 +kind: ServiceMeshControlPlane +metadata: + name: basic + namespace: istio-system +spec: + profiles: + - default + security: + dataPlane: + mtls: true <1> + techPreview: + meshConfig: + defaultConfig: + terminationDrainDuration: 35s <2> + gateways: + ingress: + service: + metadata: + labels: + knative: ingressgateway <3> + proxy: + networking: + trafficControl: + inbound: + excludedPorts: <4> + - 8444 # metrics + - 8022 # serving: wait-for-drain k8s pre-stop hook +---- +<1> This enforces strict mTLS in the mesh. Only calls using a valid client-certificate are allowed. +<2> OpenShift Serverless has a graceful termination for Knative Services of 30 seconds. The istio-proxy needs to have a longer termination duration to make sure no requests are dropped. +<3> Defining a specific selector for the `ingress-gateway` to target only the Knative gateway. +<4> Those ports are called by Kubernetes and cluster monitoring which are not part of the mesh and cannot be called using mTLS, thus those ports are excluded from the Service mesh. -.Procedure . Add the namespaces that you would like to integrate with {SMProductShortName} to the `ServiceMeshMemberRoll` object as members: + @@ -37,7 +114,7 @@ spec: members: <1> - knative-serving - knative-eventing - - + - ---- <1> A list of namespaces to be integrated with {SMProductShortName}. + @@ -55,7 +132,7 @@ $ oc apply -f . Create the necessary gateways so that {SMProductShortName} can accept traffic: + -.Example `knative-local-gateway` object using HTTP +.Example `knative-local-gateway` object using `ISTIO_MUTUAL` (mTLS) [source,yaml] ---- apiVersion: networking.istio.io/v1alpha3 @@ -65,7 +142,7 @@ metadata: namespace: knative-serving spec: selector: - istio: ingressgateway + knative: ingressgateway servers: - port: number: 443 @@ -84,12 +161,14 @@ metadata: namespace: knative-serving spec: selector: - istio: ingressgateway + knative: ingressgateway servers: - port: number: 8081 - name: http - protocol: HTTP <2> + name: https + protocol: HTTPS <2> + tls: + mode: ISTIO_MUTUAL <2> hosts: - "*" --- @@ -110,30 +189,7 @@ spec: targetPort: 8081 ---- <1> Add the name of the secret that contains the wildcard certificate. -<2> The `knative-local-gateway` serves HTTP traffic. Using HTTP means that traffic coming from outside of {SMProductShortName}, but using an internal hostname, such as `example.default.svc.cluster.local`, is not encrypted. You can set up encryption for this path by creating another wildcard certificate and an additional gateway that uses a different `protocol` spec. -+ -.Example `knative-local-gateway` object using HTTPS -[source,yaml] ----- -apiVersion: networking.istio.io/v1alpha3 -kind: Gateway -metadata: - name: knative-local-gateway - namespace: knative-serving -spec: - selector: - istio: ingressgateway - servers: - - port: - number: 443 - name: https - protocol: HTTPS - hosts: - - "*" - tls: - mode: SIMPLE - credentialName: ----- +<2> The `knative-local-gateway` serves HTTPS traffic and expects all clients to send requests using mTLS. This means, that only traffic coming from withing {SMProductShortName} is possible. Workloads from outside the {smproductshortname} must use the external domain via OpenShift Routing. . Apply the `Gateway` resources: + @@ -142,7 +198,11 @@ spec: $ oc apply -f ---- -. Install Knative Serving by creating the following `KnativeServing` custom resource, which also enables the Istio integration: +.Installing and configuring {serverless} + +. First, install the {serverless} Operator. + +. Then, install Knative Serving by creating the following `KnativeServing` custom resource, which also enables the Istio integration: + [source,yaml] ---- @@ -164,9 +224,14 @@ spec: annotations: "sidecar.istio.io/inject": "true" "sidecar.istio.io/rewriteAppHTTPProbers": "true" + config: + istio: <3> + gateway.knative-serving.knative-ingress-gateway: istio-ingressgateway..svc.cluster.local + local-gateway.knative-serving.knative-local-gateway: knative-local-gateway..svc.cluster.local ---- <1> Enables Istio integration. <2> Enables sidecar injection for Knative Serving data plane pods. +<3> Optional: if your istio is *NOT* running in `istio-system`, set those two flags with the correct namespace. . Apply the `KnativeServing` resource: + @@ -206,7 +271,7 @@ spec: "sidecar.istio.io/inject": "true" "sidecar.istio.io/rewriteAppHTTPProbers": "true" ---- -<1> Enables Eventing istio controller to create a `DestinationRule` for each InMemoryChannel or KafkaChannel service. +<1> Enables Eventing Istio controller to create a `DestinationRule` for each InMemoryChannel or KafkaChannel service. <2> Enables sidecar injection for Knative Eventing pods. . Apply the `KnativeEventing` resource: @@ -351,9 +416,14 @@ spec: containers: - image: ---- -<1> A namespace that is part of the Service Mesh member roll. +<1> A namespace that is part of the Service mesh member roll. <2> Instructs Knative Serving to generate an {product-title} pass-through enabled route, so that the certificates you have generated are served through the ingress gateway directly. <3> Injects {SMProductShortName} sidecars into the Knative service pods. ++ +[IMPORTANT] +==== +Please note, that you have to always add the annotation from the example above to all your Knative `Service` to make them work with {SMPRODUCTSHORTNAME}. +==== . Apply the `Service` resource: + diff --git a/modules/serverless-eventing/pages/service-mesh/eventing-service-mesh-containersource.adoc b/modules/serverless/pages/service-mesh/eventing-service-mesh-containersource.adoc similarity index 100% rename from modules/serverless-eventing/pages/service-mesh/eventing-service-mesh-containersource.adoc rename to modules/serverless/pages/service-mesh/eventing-service-mesh-containersource.adoc diff --git a/modules/serverless-eventing/pages/service-mesh/eventing-service-mesh-sinkbinding.adoc b/modules/serverless/pages/service-mesh/eventing-service-mesh-sinkbinding.adoc similarity index 100% rename from modules/serverless-eventing/pages/service-mesh/eventing-service-mesh-sinkbinding.adoc rename to modules/serverless/pages/service-mesh/eventing-service-mesh-sinkbinding.adoc diff --git a/modules/serverless/pages/serving/serving-with-ingress-sharding.adoc b/modules/serverless/pages/serving/serving-with-ingress-sharding.adoc new file mode 100644 index 00000000..6ee2adf6 --- /dev/null +++ b/modules/serverless/pages/serving/serving-with-ingress-sharding.adoc @@ -0,0 +1,123 @@ += Use Serving with {product-title} ingress sharding +:compat-mode!: +// Metadata: +:description: Use Serving with {product-title} ingress sharding + +This page describes how {serverless} Serving can be used with {product-title} ingress sharding. + +.Prerequisites + +* You have access to an {product-title} account with cluster administrator access. + +* You have an existing {product-title} routing setup that uses ingress shards + + +.Setting up {product-title} ingress shards + +{serverless} can be configured to match specific ingress shards with different domains with a label selector. +The following is an example of how your ingress shard could be configured. +The relevant fields are marked: + +[source,yaml] +---- +apiVersion: operator.openshift.io/v1 +kind: IngressController +metadata: + name: ingress-dev <1> + namespace: openshift-ingress-operator +spec: + routeSelector: + matchLabels: + router: dev <2> + domain: "dev.serverless.cluster.example.com" <3> + #... other settings ... +--- +apiVersion: operator.openshift.io/v1 +kind: IngressController +metadata: + name: ingress-prod <1> + namespace: openshift-ingress-operator +spec: + routeSelector: + matchLabels: + router: prod <2> + domain: "prod.serverless.cluster.example.com" <3> + #... other settings ... +---- +<1> The names of your ingress shard, here as an example `dev` and `prod` for different stages +<2> A label selector to match the ingress shard +<3> A custom domain for the ingress shard + +Replace these values with your own configuration. + +.Configuring custom domains in the `KnativeServing` CustomResource + +To match the ingress shards, you need to configure `KnativeServing` to use the same domains and labels as your ingress shards. +For this you need to create or edit your `KnativeServing` resource and add the `spec.config.domain` field: +[source,yaml] +---- +spec: + config: + domain: <1> + dev.serverless.cluster.example.com: | + selector: + router: dev + prod.serverless.cluster.example.com: | + selector: + router: prod +---- +<1> These values have to match to the ones in the ingress shard configuration. + +.Targeting a specific ingress shard in the Knative Service + +You can now target a specific ingress shard in your Knative `Service` resources using a label: +[source,yaml] +---- +apiVersion: serving.knative.dev/v1 +kind: Service +metadata: + name: hello-dev + labels: + router: dev <1> +spec: + template: + spec: + containers: + - image: docker.io/openshift/hello-openshift +--- +apiVersion: serving.knative.dev/v1 +kind: Service +metadata: + name: hello-prod + labels: + router: prod <1> +spec: + template: + spec: + containers: + - image: docker.io/openshift/hello-openshift +---- +<1> This have to match the configuration in `KnativeServing` + +.Verification + +With this setup, your Knative Services should use the correct route and the selected ingress shard: +[source,terminal] +---- +$ oc get ksvc +NAME URL LATESTCREATED LATESTREADY READY REASON +hello-dev https://hello-dev-default.dev.serverless.cluster.example.com hello-dev-00001 hello-dev-00001 True +hello-prod https://hello-prod-default.prod.serverless.cluster.example.com hello-prod-00001 hello-prod-00001 True +---- +[source,terminal] +---- +$ oc get route -n knative-serving-ingress -o jsonpath='{range .items[*]}{@.metadata.name}{" "}{@.spec.host}{" "}{@.status.ingress[*].routerName}{"\n"}{end}' +route-19e6628b-77af-4da0-9b4c-1224934b2250-323461616533 hello-prod-default.prod.serverless.cluster.example.com ingress-prod +route-cb5085d9-b7da-4741-9a56-96c88c6adaaa-373065343266 hello-dev-default.dev.serverless.cluster.example.com ingress-dev +---- + +[NOTE] +==== +Please be aware that even with OpenShift ingress sharding in place, the {serverless} traffic is still routed through a single Knative Ingress Gateway and the activator component in the `knative-serving` project. +Please also take a look at xref:./../service-mesh/common-service-mesh-network-isolation.adoc[] for more information about isolating the network traffic. +====