Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
Add ingressClassName to Policy
  • Loading branch information
soneillf5 committed Jul 24, 2021
1 parent 8177a32 commit 165d3f2
Show file tree
Hide file tree
Showing 18 changed files with 288 additions and 11 deletions.
2 changes: 1 addition & 1 deletion cmd/nginx-ingress/main.go
Expand Up @@ -79,7 +79,7 @@ var (
For Kubernetes < 1.18, the Ingress Controller only processes resources that belong to its class -
i.e have the annotation "kubernetes.io/ingress.class" (for Ingress resources)
or field "ingressClassName" (for VirtualServer/VirtualServerRoute/TransportServer resources) equal to the class.
or field "ingressClassName" equal to the class.
Additionally, the Ingress Controller processes resources that do not have the class set,
which can be disabled by setting the "-use-ingress-class-only" flag
Expand Down
2 changes: 2 additions & 0 deletions deployments/common/crds/k8s.nginx.org_policies.yaml
Expand Up @@ -76,6 +76,8 @@ spec:
type: integer
verifyServer:
type: boolean
ingressClassName:
type: string
ingressMTLS:
description: 'IngressMTLS defines an Ingress MTLS policy. policy status: preview'
type: object
Expand Down
2 changes: 1 addition & 1 deletion deployments/helm-chart/README.md
Expand Up @@ -172,7 +172,7 @@ Parameter | Description | Default
`controller.volumeMounts` | The volumeMounts of the Ingress controller pods. | []
`controller.resources` | The resources of the Ingress controller pods. | {}
`controller.replicaCount` | The number of replicas of the Ingress controller deployment. | 1
`controller.ingressClass` | A class of the Ingress controller. For Kubernetes >= 1.18, a corresponding IngressClass resource with the name equal to the class must be deployed. Otherwise, the Ingress Controller will fail to start. The Ingress controller only processes resources that belong to its class - i.e. have the "ingressClassName" field resource equal to the class. For Kubernetes < 1.18, the Ingress Controller only processes resources that belong to its class - i.e have the annotation "kubernetes.io/ingress.class" (for Ingress resources) or field "ingressClassName" (for VirtualServer/VirtualServerRoute/TransportServer resources) equal to the class. Additionally, the Ingress Controller processes resources that do not have the class set, which can be disabled by setting the `controller.useIngressClassOnly` parameter to `true`. The Ingress Controller processes all the VirtualServer/VirtualServerRoute/TransportServer resources that do not have the "ingressClassName" field for all versions of kubernetes. | nginx
`controller.ingressClass` | A class of the Ingress controller. For Kubernetes >= 1.18, a corresponding IngressClass resource with the name equal to the class must be deployed. Otherwise, the Ingress Controller will fail to start. The Ingress controller only processes resources that belong to its class - i.e. have the "ingressClassName" field resource equal to the class. For Kubernetes < 1.18, the Ingress Controller only processes resources that belong to its class - i.e have the annotation "kubernetes.io/ingress.class" (for Ingress resources) or field "ingressClassName" equal to the class. Additionally, the Ingress Controller processes resources that do not have the class set, which can be disabled by setting the `controller.useIngressClassOnly` parameter to `true`. The Ingress Controller processes all the VirtualServer/VirtualServerRoute/TransportServer resources that do not have the "ingressClassName" field for all versions of kubernetes. | nginx
`controller.useIngressClassOnly` | Ignore Ingress resources without the `"kubernetes.io/ingress.class"` annotation. For kubernetes versions >= 1.18 this flag will be IGNORED. | false
`controller.setAsDefaultIngress` | New Ingresses without an `"ingressClassName"` field specified will be assigned the class specified in `controller.ingressClass`. Only for kubernetes versions >= 1.18. | false
`controller.watchNamespace` | Namespace to watch for Ingress resources. By default the Ingress controller watches all namespaces. | ""
Expand Down
2 changes: 2 additions & 0 deletions deployments/helm-chart/crds/k8s.nginx.org_policies.yaml
Expand Up @@ -76,6 +76,8 @@ spec:
type: integer
verifyServer:
type: boolean
ingressClassName:
type: string
ingressMTLS:
description: 'IngressMTLS defines an Ingress MTLS policy. policy status: preview'
type: object
Expand Down
4 changes: 2 additions & 2 deletions deployments/helm-chart/values.yaml
Expand Up @@ -125,11 +125,11 @@ controller:

## For Kubernetes < 1.18, the Ingress Controller only processes resources that belong to its class -
## i.e have the annotation "kubernetes.io/ingress.class" (for Ingress resources)
## or field "ingressClassName" (for VirtualServer/VirtualServerRoute/TransportServer resources) equal to the class.
## or field "ingressClassName" equal to the class.
## Additionally, the Ingress Controller processes resources that do not have the class set,
## which can be disabled by setting the controller.useIngressClassOnly parameter to true.

## The Ingress Controller processes all the VirtualServer/VirtualServerRoute/TransportServer resources that do not have the "ingressClassName" field for all versions of kubernetes.
## The Ingress Controller processes all the resources that do not have the "ingressClassName" field for all versions of kubernetes.
ingressClass: nginx

## For kubernetes versions >= 1.18 this flag will be IGNORED.
Expand Down
Expand Up @@ -94,10 +94,10 @@ Below we describe the available command-line arguments:
For Kubernetes >= 1.18, a corresponding IngressClass resource with the name equal to the class must be deployed. Otherwise, the Ingress Controller will fail to start.
The Ingress controller only processes resources that belong to its class - i.e. have the "ingressClassName" field resource equal to the class.
For Kubernetes < 1.18, the Ingress Controller only processes resources that belong to its class - i.e have the annotation "kubernetes.io/ingress.class" (for Ingress resources) or field "ingressClassName" (for VirtualServer/VirtualServerRoute/TransportServer resources) equal to the class.
For Kubernetes < 1.18, the Ingress Controller only processes resources that belong to its class - i.e have the annotation "kubernetes.io/ingress.class" (for Ingress resources) or field "ingressClassName" equal to the class.
Additionally, the Ingress Controller processes resources that do not have the class set, which can be disabled by setting the "-use-ingress-class-only" flag.
The Ingress Controller processes all the VirtualServer/VirtualServerRoute/TransportServer resources that do not have the "ingressClassName" field.
The Ingress Controller processes all the resources that do not have the "ingressClassName" field.
(default "nginx")
Expand Down
5 changes: 3 additions & 2 deletions docs/content/configuration/policy-resource.md
Expand Up @@ -64,12 +64,13 @@ spec:
{{% table %}}
|Field | Description | Type | Required |
| ---| ---| ---| --- |
|``accessControl`` | The access control policy based on the client IP address. | [accessControl](#accesscontrol) | No |
|``accessControl`` | The access control policy based on the client IP address. | [accessControl](#accesscontrol) | No |
|``ingressClassName`` | Specifies which Ingress Controller must handle the Policy resource. | ``string`` | No |
|``rateLimit`` | The rate limit policy controls the rate of processing requests per a defined key. | [rateLimit](#ratelimit) | No |
|``jwt`` | The JWT policy configures NGINX Plus to authenticate client requests using JSON Web Tokens. | [jwt](#jwt) | No |
|``ingressMTLS`` | The IngressMTLS policy configures client certificate verification. | [ingressMTLS](#ingressmtls) | No |
|``egressMTLS`` | The EgressMTLS policy configures upstreams authentication and certificate verification. | [egressMTLS](#egressmtls) | No |
|``waf`` | The WAF policy configures WAF and log configuration policies for [NGINX AppProtect](/nginx-ingress-controller/app-protect/installation/) | [WAF](#waf) | No |
|``waf`` | The WAF policy configures WAF and log configuration policies for [NGINX AppProtect](/nginx-ingress-controller/app-protect/installation/) | [WAF](#waf) | No |
{{% /table %}}

\* A policy must include exactly one policy.
Expand Down
Expand Up @@ -23,7 +23,7 @@ The smooth coexistence of multiple Ingress Controllers in one cluster is provide
* Every Ingress Controller must only handle Ingress resources for its particular class.
* For Kubernetes < 1.18, Ingress resources should be annotated with the `kubernetes.io/ingress.class` annotation set to the value, which corresponds to the class of the Ingress Controller the user wants to use.
* When using versions of Kubernetes >= 1.18, Ingress resources should have the `ingressClassName` field set to the value, which corresponds to the class of the Ingress Controller the user wants to use.
* VirtualServer, VirtualServerRoute and TransportServer resources should have the `ingressClassName` field set to the value, which corresponds to the class of the Ingress Controller the user wants to use.
* VirtualServer, VirtualServerRoute, Policy and TransportServer resources should have the `ingressClassName` field set to the value, which corresponds to the class of the Ingress Controller the user wants to use.

### Configuring Ingress Class

Expand All @@ -32,7 +32,7 @@ The default Ingress class of NGINX Ingress Controller is `nginx`, which means th
**Notes**:
* For Kubernetes < 1.18, if the class is not set in an Ingress configuration resource, the Ingress Controller will handle the resource. This is controlled via the `-use-ingress-class-only` argument.
* For Kubernetes >= 1.18, if the class is not set in an Ingress resource, Kubernetes will set it to the class of the default Ingress Controller. To make the Ingress Controller the default one, the `ingressclass.kubernetes.io/is-default-class` must be set on the IngressClass resource. See Step 3 *Create an IngressClass resource* of the [Create Common Resources](/nginx-ingress-controller/installation/installation-with-manifests/#create-common-resources) section.
* For VirtualServer, VirtualServerRoute and TransportServer resources the Ingress Controller will always handle resources with an empty class.
* For VirtualServer, VirtualServerRoute, Policy and TransportServer resources the Ingress Controller will always handle resources with an empty class.

## Running NGINX Ingress Controller and Another Ingress Controller

Expand Down
10 changes: 9 additions & 1 deletion internal/k8s/controller.go
Expand Up @@ -298,6 +298,7 @@ func NewLoadBalancerController(input NewLoadBalancerControllerInput) *LoadBalanc
policyLister: lbc.policyLister,
keyFunc: keyFunc,
confClient: input.ConfClient,
hasCorrectIngressClass: lbc.HasCorrectIngressClass,
}

lbc.configuration = NewConfiguration(
Expand Down Expand Up @@ -794,7 +795,7 @@ func (lbc *LoadBalancerController) syncPolicy(task task) {

glog.V(2).Infof("Adding, Updating or Deleting Policy: %v\n", key)

if polExists {
if polExists && lbc.HasCorrectIngressClass(obj) {
pol := obj.(*conf_v1.Policy)
err := validation.ValidatePolicy(pol, lbc.isNginxPlus, lbc.enablePreviewPolicies, lbc.appProtectEnabled)
if err != nil {
Expand Down Expand Up @@ -2487,6 +2488,11 @@ func (lbc *LoadBalancerController) getPolicies(policies []conf_v1.PolicyReferenc

policy := policyObj.(*conf_v1.Policy)

if !lbc.HasCorrectIngressClass(policy) {
errors = append(errors, fmt.Errorf("referenced policy %s has incorrect ingress class: %s (controller ingress class: %s)", policyKey, policy.Spec.IngressClass, lbc.ingressClass))
continue
}

err = validation.ValidatePolicy(policy, lbc.isNginxPlus, lbc.enablePreviewPolicies, lbc.appProtectEnabled)
if err != nil {
errors = append(errors, fmt.Errorf("Policy %s is invalid: %w", policyKey, err))
Expand Down Expand Up @@ -3038,6 +3044,8 @@ func (lbc *LoadBalancerController) HasCorrectIngressClass(obj interface{}) bool
class = obj.Spec.IngressClass
case *conf_v1alpha1.TransportServer:
class = obj.Spec.IngressClass
case *conf_v1.Policy:
class = obj.Spec.IngressClass
case *networking.Ingress:
isIngress = true
class = obj.Annotations[ingressClassKey]
Expand Down
20 changes: 20 additions & 0 deletions internal/k8s/controller_test.go
Expand Up @@ -711,6 +711,19 @@ func TestGetPolicies(t *testing.T) {
},
}

validPolicyIngressClass := &conf_v1.Policy{
ObjectMeta: meta_v1.ObjectMeta{
Name: "valid-policy-ingress-class",
Namespace: "default",
},
Spec: conf_v1.PolicySpec{
IngressClass: "test-class",
AccessControl: &conf_v1.AccessControl{
Allow: []string{"127.0.0.1"},
},
},
}

invalidPolicy := &conf_v1.Policy{
ObjectMeta: meta_v1.ObjectMeta{
Name: "invalid-policy",
Expand All @@ -726,6 +739,8 @@ func TestGetPolicies(t *testing.T) {
switch key {
case "default/valid-policy":
return validPolicy, true, nil
case "default/valid-policy-ingress-class":
return validPolicyIngressClass, true, nil
case "default/invalid-policy":
return invalidPolicy, true, nil
case "nginx-ingress/valid-policy":
Expand Down Expand Up @@ -754,13 +769,18 @@ func TestGetPolicies(t *testing.T) {
Name: "some-policy", // will make lister return error
Namespace: "nginx-ingress",
},
{
Name: "valid-policy-ingress-class",
Namespace: "default",
},
}

expectedPolicies := []*conf_v1.Policy{validPolicy}
expectedErrors := []error{
errors.New("Policy default/invalid-policy is invalid: spec: Invalid value: \"\": must specify exactly one of: `accessControl`, `rateLimit`, `ingressMTLS`, `egressMTLS`, `jwt`, `oidc`, `waf`"),
errors.New("Policy nginx-ingress/valid-policy doesn't exist"),
errors.New("Failed to get policy nginx-ingress/some-policy: GetByKey error"),
errors.New("referenced policy default/valid-policy-ingress-class has incorrect ingress class: test-class (controller ingress class: )"),
}

result, errors := lbc.getPolicies(policyRefs, "default")
Expand Down
6 changes: 6 additions & 0 deletions internal/k8s/status.go
Expand Up @@ -44,6 +44,7 @@ type statusUpdater struct {
transportServerLister cache.Store
policyLister cache.Store
confClient k8s_nginx.Interface
hasCorrectIngressClass func(interface{}) bool
}

func (su *statusUpdater) UpdateExternalEndpointsForResources(resource []Resource) error {
Expand Down Expand Up @@ -638,6 +639,11 @@ func (su *statusUpdater) UpdatePolicyStatus(pol *v1.Policy, state string, reason
return nil
}

if !su.hasCorrectIngressClass(polLatest) {
glog.V(3).Infof("ignoring policy with incorrect ingress class")
return nil
}

polCopy := polLatest.(*v1.Policy)

if !hasPolicyStatusChanged(polCopy, state, reason, message) {
Expand Down
1 change: 1 addition & 0 deletions pkg/apis/configuration/v1/types.go
Expand Up @@ -343,6 +343,7 @@ type PolicyStatus struct {
// The spec includes multiple fields, where each field represents a different policy.
// Only one policy (field) is allowed.
type PolicySpec struct {
IngressClass string `json:"ingressClassName"`
AccessControl *AccessControl `json:"accessControl"`
RateLimit *RateLimit `json:"rateLimit"`
JWTAuth *JWTAuth `json:"jwt"`
Expand Down
10 changes: 10 additions & 0 deletions tests/data/policy-ingress-class/policy-ingress-class.yaml
@@ -0,0 +1,10 @@
apiVersion: k8s.nginx.org/v1
kind: Policy
metadata:
name: rate-limit-primary
spec:
ingressClasName: nginx
rateLimit:
rate: 1r/s
key: ${binary_remote_addr}
zoneSize: 10M
10 changes: 10 additions & 0 deletions tests/data/policy-ingress-class/policy-other-ingress-class.yaml
@@ -0,0 +1,10 @@
apiVersion: k8s.nginx.org/v1
kind: Policy
metadata:
name: rate-limit-primary
spec:
ingressClassName: other
rateLimit:
rate: 1r/s
key: ${binary_remote_addr}
zoneSize: 10M
9 changes: 9 additions & 0 deletions tests/data/policy-ingress-class/policy.yaml
@@ -0,0 +1,9 @@
apiVersion: k8s.nginx.org/v1
kind: Policy
metadata:
name: rate-limit-primary
spec:
rateLimit:
rate: 1r/s
key: ${binary_remote_addr}
zoneSize: 10M
23 changes: 23 additions & 0 deletions tests/data/policy-ingress-class/virtual-server-policy.yaml
@@ -0,0 +1,23 @@
apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
name: virtual-server
spec:
ingressClassName: nginx
host: virtual-server.example.com
policies:
- name: rate-limit-primary
upstreams:
- name: backend2
service: backend2-svc
port: 80
- name: backend1
service: backend1-svc
port: 80
routes:
- path: "/backend1"
action:
pass: backend1
- path: "/backend2"
action:
pass: backend2
21 changes: 21 additions & 0 deletions tests/data/policy-ingress-class/virtual-server.yaml
@@ -0,0 +1,21 @@
apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
name: virtual-server
spec:
ingressClassName: nginx
host: virtual-server.example.com
upstreams:
- name: backend2
service: backend2-svc
port: 80
- name: backend1
service: backend1-svc
port: 80
routes:
- path: "/backend1"
action:
pass: backend1
- path: "/backend2"
action:
pass: backend2

0 comments on commit 165d3f2

Please sign in to comment.