first-vela-app cannot bind load balancer in AKS with default nginx ingress controller

[heavenwing]

Describe the bug
I tried KubeVela quick-start in AKS, I follow this document to enable ingress. But after I deployed first-vela-app, I got below error:

services:
  - healthy: true
    name: express-server
    traits:
    - healthy: true
      message: |
        No loadBalancer found, visiting by using 'vela port-forward first-vela-app'
      type: ingress-1-20
    workloadDefinition:
      apiVersion: apps/v1
      kind: Deployment
  status: running

To Reproduce
Steps to reproduce the behavior:

1. Create AKS cluster
2. Enable ingress follow up this document, and verify ingress is working
3. Deploy quick-start app into the AKS cluster
4. See error

Expected behavior
curl -H "Host:testsvc.example.com" http://<your ip address>/ will get correct html contents

Screenshots

KubeVela Version
1.1.3

Cluster information

• Kubernetes Version 1.21.2
• Kubernetes Network CNI

[wonderflow]

Thanks for reporting. Could you help provide the ingress object status?

[heavenwing]

complete ingress yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    app.oam.dev/last-applied-configuration: '{"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"core.oam.dev/v1beta1\",\"kind\":\"Application\",\"metadata\":{\"annotations\":{},\"name\":\"first-vela-app\",\"namespace\":\"default\"},\"spec\":{\"components\":[{\"name\":\"express-server\",\"properties\":{\"image\":\"crccheck/hello-world\",\"port\":8000},\"traits\":[{\"properties\":{\"domain\":\"testsvc.example.com\",\"http\":{\"/\":8000}},\"type\":\"ingress-1-20\"}],\"type\":\"webservice\"}]}}\n","oam.dev/kubevela-version":"v1.1.3"},"labels":{"app.oam.dev/app-revision-hash":"f7282e2c2debed00","app.oam.dev/appRevision":"first-vela-app-v2","app.oam.dev/component":"express-server","app.oam.dev/name":"first-vela-app","app.oam.dev/resourceType":"TRAIT","app.oam.dev/revision":"express-server-v1","trait.oam.dev/resource":"ingress","trait.oam.dev/type":"ingress-1-20"},"name":"express-server","namespace":"default","ownerReferences":[{"apiVersion":"core.oam.dev/v1beta1","blockOwnerDeletion":true,"controller":true,"kind":"ResourceTracker","name":"first-vela-app-v2-default","uid":"2756da8c-fdc2-4ea5-b1e1-e22a375e2cb1"}]},"spec":{"rules":[{"host":"testsvc.example.com","http":{"paths":[{"backend":{"service":{"name":"express-server","port":{"number":8000}}},"path":"/","pathType":"ImplementationSpecific"}]}}]}}'
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"core.oam.dev/v1beta1","kind":"Application","metadata":{"annotations":{},"name":"first-vela-app","namespace":"default"},"spec":{"components":[{"name":"express-server","properties":{"image":"crccheck/hello-world","port":8000},"traits":[{"properties":{"domain":"testsvc.example.com","http":{"/":8000}},"type":"ingress-1-20"}],"type":"webservice"}]}}
    oam.dev/kubevela-version: v1.1.3
  creationTimestamp: "2021-10-09T07:35:37Z"
  generation: 1
  labels:
    app.oam.dev/app-revision-hash: f7282e2c2debed00
    app.oam.dev/appRevision: first-vela-app-v2
    app.oam.dev/component: express-server
    app.oam.dev/name: first-vela-app
    app.oam.dev/resourceType: TRAIT
    app.oam.dev/revision: express-server-v1
    trait.oam.dev/resource: ingress
    trait.oam.dev/type: ingress-1-20
  name: express-server
  namespace: default
  ownerReferences:
  - apiVersion: core.oam.dev/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: ResourceTracker
    name: first-vela-app-v2-default
    uid: 2756da8c-fdc2-4ea5-b1e1-e22a375e2cb1
  resourceVersion: "1260581"
  uid: 79afcbdd-9c79-481c-b0db-ba42f9cf12a4
spec:
  rules:
  - host: testsvc.example.com
    http:
      paths:
      - backend:
          service:
            name: express-server
            port:
              number: 8000
        path: /
        pathType: ImplementationSpecific
status:
  loadBalancer: {}

complete description of the ingress:

Name:             express-server
Namespace:        default
Address:          
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host                 Path  Backends
  ----                 ----  --------
  testsvc.example.com  
                       /   express-server:8000 (10.2.1.33:8000)
Annotations:           app.oam.dev/last-applied-configuration:
                         {"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"ap...
                       oam.dev/kubevela-version: v1.1.3
Events:                <none>

[wonderflow]

It seems the ingress didn't create any loadBalancer because of lacking default backend. Could you help verify if it works when we add the defaultBackend section? You could help try to apply the following yaml, thanks.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: express-server
  namespace: default
spec:
  defaultBackend:
    service:
      name: express-server
      port:
        number: 8000
  rules:
  - host: testsvc.example.com
    http:
      paths:
      - backend:
          service:
            name: express-server
            port:
              number: 8000
        path: /
        pathType: ImplementationSpecific

[wonderflow]

Another possible reason is we need to add the annotation in the ingress object:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  name: express-server
  namespace: default
spec:
  rules:
  - host: testsvc.example.com
    http:
      paths:
      - backend:
          service:
            name: express-server
            port:
              number: 8000
        path: /
        pathType: ImplementationSpecific

We don't have an AKS cluster, if you don't mind, please help verify it, thanks

[heavenwing]

You are right. The reason is 'kubernetes.io/ingress.class: nginx' missing.
The ingress yaml from document also have this annotations. But we can miss it In theory.

[wonderflow]

miss

Thanks for the verification, we can fix it by adding some parameters.

[wonderflow]

@heavenwing Apply this yaml can fix the problem, we will fix it in #2445 :

# Code generated by KubeVela templates. DO NOT EDIT. Please edit the original cue file.
# Definition source cue file: vela-templates/definitions/internal/ingress-1-20.cue
apiVersion: core.oam.dev/v1beta1
kind: TraitDefinition
metadata:
  annotations:
    definition.oam.dev/description: Enable public web traffic for the component, the ingress API matches K8s v1.20+.
  name: ingress-1-20
  namespace: {{.Values.systemDefinitionNamespace}}
spec:
  podDisruptive: false
  schematic:
    cue:
      template: |
        // trait template can have multiple outputs in one trait
        outputs: service: {
        	apiVersion: "v1"
        	kind:       "Service"
        	metadata: name: context.name
        	spec: {
        		selector: "app.oam.dev/component": context.name
        		ports: [
        			for k, v in parameter.http {
        				port:       v
        				targetPort: v
        			},
        		]
        	}
        }
        outputs: ingress: {
        	apiVersion: "networking.k8s.io/v1"
        	kind:       "Ingress"
        	metadata: {
        		name: context.name
        		annotations: "kubernetes.io/ingress.class": parameter.class
        	}
        	spec: rules: [{
        		host: parameter.domain
        		http: paths: [
        			for k, v in parameter.http {
        				path:     k
        				pathType: "ImplementationSpecific"
        				backend: service: {
        					name: context.name
        					port: number: v
        				}
        			},
        		]
        	}]
        }
        parameter: {
        	// +usage=Specify the domain you want to expose
        	domain: string

        	// +usage=Specify the mapping relationship between the http path and the workload port
        	http: [string]: int

        	// +usage=Specify the class of ingress to use
        	class?: *"nginx" | string
        }
  status:
    customStatus: |-
      let igs = context.outputs.ingress.status.loadBalancer.ingress
      if igs == _|_ {
        message: "No loadBalancer found, visiting by using 'vela port-forward " + context.appName + "'\n"
      }
      if len(igs) > 0 {
        if igs[0].ip != _|_ {
      	  message: "Visiting URL: " + context.outputs.ingress.spec.rules[0].host + ", IP: " + igs[0].ip
        }
        if igs[0].ip == _|_ {
      	  message: "Visiting URL: " + context.outputs.ingress.spec.rules[0].host
        }
      }
    healthPolicy: 'isHealth: len(context.outputs.service.spec.clusterIP) > 0'

[heavenwing]

above yaml have a little issue:
need to replace this line

namespace: {{.Values.systemDefinitionNamespace}}

into

namespace: vela-system

And, for app yaml, I have to set class parameter, I mistaken think I can ignore it firstly

apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: first-vela-app
spec:
  components:
    - name: express-server
      type: webservice
      properties:
        image: crccheck/hello-world
        port: 8000
      traits:
        - type: ingress-1-20
          properties:
            class: nginx
            domain: testsvc.example.com
            http:
              "/": 8000

[wonderflow]

@heavenwing Sorry, we should remove ? , so it can be ignored

class: *"nginx" | string