测试higress gateway-api时，换端口发现测试不成功

[alexzzh]

• 按照 https://github.com/alibaba/higress/blob/main/samples/gateway-api/demo.yaml 文档测试，可以访问成功，但把 Gateway 资源的port: 80 换成 port: 1180后，提示gateway 状态不正常，不能修改这里代理端口(80)吗? 理论上应该可以且支持配置多个不同的监听端口吧。

apiVersion: gateway.networking.k8s.io/v1
    Message:               Waiting for controller
    Reason:                Pending
    Status:                Unknown
    Type:                  Programmed
    Last Transition Time:  2024-02-27T09:02:17Z
    Message:               failed to assign to any requested addresses: port 1180 not found for hostname "higress-gateway.higress-system.svc.cluster.local"
    Observed Generation:   1
    Reason:                AddressNotAssigned
    Status:                False
    Type:                  Ready
    Last Transition Time:  2024-02-27T09:02:17Z
    Message:               Resources not yet deployed to the cluster
    Observed Generation:   1
    Reason:                ResourcesPending
    Status:                False
    Type:                  Scheduled

• 由于业务需要，我们需要通过gateway api定义多个port并做不同的配置，想问下higress的gateway api是否可以满足下方的需求?:
• 例如

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: higress-gateway
  namespace: higress-system
spec:
  gatewayClassName: higress-gateway
  listeners:
  - name: ashttp1
    hostname: "*.gateway-api.com"
    port: 180
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: All
  - name: ashttp2
    hostname: "*.gateway-api.com"
    port: 443
    protocol: HTTPS
    xxx - 证书上下文
    allowedRoutes:
      namespaces:
        from: All
  - name: default
    hostname: "*.gateway-api.com"
    port: 445
    protocol: HTTPS
    xxx - 证书上下文
    allowedRoutes:
      namespaces:
        from: All

感谢！

[johnlanni]

可以转成issue哈，方便后面解决跟进

[johnlanni]

应该是loadbalancer service没有对应端口，需要手动设置下

[alexzzh]

使用 kubectl edit svc higress-gateway -n higress-system 调整 higress-gateway svc 的 port 确实可以解决这个问题，如下，可同时支持配置gateway资源监听 11111、22222、443三个端口

zhangzh@ubuntu:~/data/codedir/higress$ kubectl get svc higress-gateway -n higress-system
NAME              TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                                         AGE
higress-gateway   LoadBalancer   10.97.218.56   <pending>     11111:30257/TCP,22222:30258/TCP,443:30875/TCP   46h

但这里引出另一个问题: 如果我们的业务经常会增加不同的监听端口，那对于higress来说，不仅仅要调整kind: Gateway 资源的port， 还要同步调整 higress-gateway service的port，相当于要调整2处，只能这么做吗?

[alexzzh]

应该是loadbalancer service没有对应端口，需要手动设置下

另外，在测试 https://higress.io/zh-cn/blog/release-1.3/ 文档中提到的https效果时，发现证书并没有下发到envoy gateway，导致访问443端口失败

• Gatewayclass 和 HTTPRoute 不变，调整 gateway 使其支持https

 apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: higress-gateway
  namespace: higress-system
spec:
  gatewayClassName: higress-gateway
  listeners:
  - name: http1
    port: 11111
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: All
  - name: http2
    port: 22222
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: All
  - name: foobar-https
    hostname: "*.foobar.com"
    port: 443
    protocol: HTTPS
    allowedRoutes:
      namespaces:
        from: All
    tls:
      certificateRefs:
      - kind: Secret
        name: wildcard-foobar-com
      mode: Terminate

• 在 higress-system 中创建 secret 如下：

kubectl create secret generic  wildcard-foobar-com --from-file=./certs/server.crt --from-file=./certs/server.key -n higress-system

$ kubectl describe secret wildcard-foobar-com  -n higress-system
Name:         wildcard-foobar-com
Namespace:    higress-system
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
server.crt:  1635 bytes
server.key:  1704 bytes

• apply gateway 没有报错，但查看envoy gateway 的证书还是空，应该是证书没有下发到envoy.

[johnlanni]

rsa密钥长度至少需要2048bi t

[alexzzh]

secret没有名称不匹配导致该问题

2024-02-28T07:30:32.489566Z	info	ads	XDS: Incremental Pushing:2024-02-28T07:14:20Z/180 ConnectedEndpoints:1 Version:2024-02-28T07:14:20Z/180
2024-02-28T07:30:32.489702Z	warn	ads	failed to fetch key and certificate for kubernetes-gateway://higress-system/wildcard-foobar-com: found secret, but didn't have expected keys (cert and key) or (tls.crt and tls.key); found: server.crt, server.key
2024-02-28T07:30:32.489768Z	info	ads	SDS: PUSH for node:higress-gateway-7bb6b7555-q2qc5.higress-system resources:0 size:0B cached:0/1

• 根据这个提示，创建secret使用tls.crt, tls.key,则控制平面不再报着个错误，数据平面的envoy也能查到证书配置，
  但是证书和私钥内容明显不对，这样肯定还是不通。

证书文件
$ cat tls.crt
-----BEGIN CERTIFICATE-----
MIIEijCCAnKgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwEzERMA8GA1UEAwwIZW52
b3ktY2EwHhcNMjQwMjI2MDcyNTU0WhcNMzQwMjIzMDcyNTU0WjAQMQ4wDAYDVQQD
DAVjZXJ0czCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKzaww03obY3
vE/tTd7R9EV4yLD2Otm8a8ULQe8lg5JKcEjr/xeZj4Lk9sQSLg7HXltYEB0q3BWZ
s6RM5rZvskNeQ8qLXLxbXQ/MOsbqJa0JowG0RKvbUWmm0dNhPfnX1ofXio0ZViWy
/AsT8WYvVDzUwVa7tjdwPYAnYKsOVJ5WX4+JMHUanCbsZghYwe1/fi7J6cXlaDAD
jt/eQbwQRSqfVl4hlrjyAn94vmxm7+p+zqUl6wzNmxZLymBj87zrBprcKhHLP+TP
STwN6A3gHwfPJZD2bs7nqsbxLnVCXtxqvXHQHZdWdkDtwHLMonrDjMvmuJO+fNru
hfJXbSO/WJUCAwEAAaOB6jCB5zAJBgNVHRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIG
QDAzBglghkgBhvhCAQ0EJhYkT3BlblNTTCBHZW5lcmF0ZWQgU2VydmVyIENlcnRp
ZmljYXRlMB0GA1UdDgQWBBS9V54AUeXNf1T81gVObRbRUnnc6DBOBgNVHSMERzBF
gBS+COr2QXur+Hx7Zt0aGe6A0hbCP6EXpBUwEzERMA8GA1UEAwwIZW52b3ktY2GC
FCv+Op2cKwFLxjFoJtvcP3aOVYphMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAK
BggrBgEFBQcDATANBgkqhkiG9w0BAQsFAAOCAgEAkQr87SFN65wmF01yRPbY5H7t
0LCopOyqN2c/vY3DhIIEeA56fWr/KDQNJarWQE8roRPgReQS9dyHI7PkxIS6Wck9
XHDpEMGUgPnBAZ9VGwfcQKoWbkOCTdwxg4DdS7FwuqX3vk6EidMg9waTDvQbLS8I
n1TZJPbbbEUdamIR3+C285nE6d9cw2dtjrifT34I6Fi8TsAp/xMoVd5ucPAaLyuJ
xXvo8/fIELF4oP6OAo0CunIMk1fn/02649AhRdQBuVOhCFELMKNupXbJhPbLDH9X
RQnhHzo1yr/4EINWh5G16H2h+i0ZP3NLQyxCFN/qS5qDh27xuvfkRgKhIviaoZmE
mSAyN4SQqoOfcWumDYQ0UUU308xqQRQcHzoZPpeG+M4Vw+61TlEJCucUcGQa04RB
IxxjUuz6aioqdOcpn2/m+CgRcoEiRN2IETeox0EyfDwEM9+zfId5pHQcv9z8LZ/Z
OjCI8yPtKVaLitbfmlx5LA+waJyFtoBJ6GoUDtyM80uNELc/FBZdJrJ2bXgF8zvJ
9OG+oWFzL7BSAv0H6h/ogAjl52+0pZ2D2wrIXv/m6PWukpIL8NlHWlfT7fKvc3Mf
l2ka97C+cgJDHNnV8iTR7vQ9ZwantMFzzwvuUBPzTkkvUnjuLCx4TKlhIamUMeGt
s8zUbl/lpzHspHuLql8=
-----END CERTIFICATE-----

私钥文件
$ cat tls.key
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCs2sMNN6G2N7xP
7U3e0fRFeMiw9jrZvGvFC0HvJYOSSnBI6/8XmY+C5PbEEi4Ox15bWBAdKtwVmbOk
TOa2b7JDXkPKi1y8W10PzDrG6iWtCaMBtESr21FpptHTYT3519aH14qNGVYlsvwL
E/FmL1Q81MFWu7Y3cD2AJ2CrDlSeVl+PiTB1Gpwm7GYIWMHtf34uyenF5WgwA47f
3kG8EEUqn1ZeIZa48gJ/eL5sZu/qfs6lJesMzZsWS8pgY/O86waa3CoRyz/kz0k8
DegN4B8HzyWQ9m7O56rG8S51Ql7car1x0B2XVnZA7cByzKJ6w4zL5riTvnza7oXy
V20jv1iVAgMBAAECggEAAs7A2U+chFGucNvpZmOhpJk8JUeymAPL+lTe+svwsakc
3spah98aKiWe7ntfHSVW3oP1KwwdrAollVccbj0tN5Pp5PnThUm0nsOR9tYpy4A7
EhWEAgibb2xivxy2vX6ogOda3ETbldws5N8rjh3Zr+1CwK5m3MqpbRH2q4Cdlj7J
DEPAs/1Xr1NY3g4s9tQpP1KAsIlc6uo1Tp4dT+jHQi4ErugjSl18k7SjcaLOPqGe
zmzafwTIBw60wAf1E68xurRLnk9Krd2cZ+mUqkIRZoh3QCI7yiu9f+gA8u23/PB1
dYUH/zQ5NdojgVfMofsaKCHE3/nGZJ5xS6jUM/prUQKBgQDnTGaYjxuMO0WyAxTF
VHtwzeF0nS2CEcbYttuEP+wQMlKTj0pgz5oNDQhS0isDa0XgvaNr/JU3n02viitQ
KyLqqtXHvfzgfM2JopFbM+Z3iJXejIQfXApzCXVqKBqJhES60yEPaUMXNc1pXpaq
Mbh9TLUhk4MmgvJcOqw3GoYCGQKBgQC/UIe5/VAQmf9lpQ9QpbtJQ6Do0xeBC888
ZQgMPTtfqqz9Q8LXhN986oLm6DZr2cx75t6+GiJAofvd0OFvlwHY3Z8jDbmxfsiR
Lf3LTAfs2oVxrh1vsndUcX0BCn/+EbK4bzCN8WtpIDSTU2LTqedQnOlxEKtQ9U3U
Ucq/hYbx3QKBgQDiEWuJmv4K3vfs2FQZ1svb4m6E4wdSxPs8X+VIao3Y+sZ5ESla
XC6uqNlE0cRW0Cd5VQEAj9xshBiTBRH2hmLXSGgwlUIvS+w8pRc7NAwDMw2Aeoi8
eeKAs3L+cfoWBKCElc0PWgvICL8fSOMrr/MpDKtRCPO71Wr6uCIQrazjKQKBgQCd
K/qliP9K9ElG50jit5Vaxk2emTV5FgumEVaAP5nUqXI/C3k59/ZnOH5cVy/GgQMm
U3Bw+yVitfEmtWFPfN7OnJwNf0wsDv2VkBQt/YORSaFR2HpkdljkrevFfCvUvhN2
xBo+srTMg+HNmzbYr3/S0WZTNh+Yfj73jE6l83v+nQKBgBvf4CzSjFdV8F/fiAeP
EAWVix9L1z+WE4H3LoDpplmlDtsrSbQ/JhWBWzxI6KC+cc5fQ2QVpyJE3l8U9qz4
hJY7LmtWVCCUH68HPJ6hjF38DWEfJRCYKXBW+4TGRIW0cpSADNLGWUW3yLj2I9yz
0E3PdBVCOlocpQdCKm6YBQok
-----END PRIVATE KEY-----

// 数据平面的 envoy 的certificate_chain 的 inline_bytes 内容是tls.crt的base64编码， private_key 的内容明显不对, 好奇怪
{
   "@type": "type.googleapis.com/envoy.admin.v3.SecretsConfigDump",
   "dynamic_active_secrets": [
    {
     "name": "kubernetes-gateway://higress-system/wildcard-foobar-com",
     "version_info": "2024-02-28T07:14:20Z/180",
     "last_updated": "2024-02-28T07:32:18.959Z",
     "secret": {
      "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret",
      "name": "kubernetes-gateway://higress-system/wildcard-foobar-com",
      "tls_certificate": {
       "certificate_chain": {
        "inline_bytes": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVpakNDQW5LZ0F3SUJBZ0lDRUFBd0RRWUpLb1pJaHZjTkFRRUxCUUF3RXpFUk1BOEdBMVVFQXd3SVpXNTIKYjNrdFkyRXdIaGNOTWpRd01qSTJNRGN5TlRVMFdoY05NelF3TWpJek1EY3lOVFUwV2pBUU1RNHdEQVlEVlFRRApEQVZqWlhKMGN6Q0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUt6YXd3MDNvYlkzCnZFL3RUZDdSOUVWNHlMRDJPdG04YThVTFFlOGxnNUpLY0Vqci94ZVpqNExrOXNRU0xnN0hYbHRZRUIwcTNCV1oKczZSTTVyWnZza05lUThxTFhMeGJYUS9NT3NicUphMEpvd0cwUkt2YlVXbW0wZE5oUGZuWDFvZlhpbzBaVmlXeQovQXNUOFdZdlZEelV3VmE3dGpkd1BZQW5ZS3NPVko1V1g0K0pNSFVhbkNic1pnaFl3ZTEvZmk3SjZjWGxhREFECmp0L2VRYndRUlNxZlZsNGhscmp5QW45NHZteG03K3ArenFVbDZ3ek5teFpMeW1Cajg3enJCcHJjS2hITFArVFAKU1R3TjZBM2dId2ZQSlpEMmJzN25xc2J4TG5WQ1h0eHF2WEhRSFpkV2RrRHR3SExNb25yRGpNdm11Sk8rZk5ydQpoZkpYYlNPL1dKVUNBd0VBQWFPQjZqQ0I1ekFKQmdOVkhSTUVBakFBTUJFR0NXQ0dTQUdHK0VJQkFRUUVBd0lHClFEQXpCZ2xnaGtnQmh2aENBUTBFSmhZa1QzQmxibE5UVENCSFpXNWxjbUYwWldRZ1UyVnlkbVZ5SUVObGNuUnAKWm1sallYUmxNQjBHQTFVZERnUVdCQlM5VjU0QVVlWE5mMVQ4MWdWT2JSYlJVbm5jNkRCT0JnTlZIU01FUnpCRgpnQlMrQ09yMlFYdXIrSHg3WnQwYUdlNkEwaGJDUDZFWHBCVXdFekVSTUE4R0ExVUVBd3dJWlc1MmIza3RZMkdDCkZDditPcDJjS3dGTHhqRm9KdHZjUDNhT1ZZcGhNQTRHQTFVZER3RUIvd1FFQXdJRm9EQVRCZ05WSFNVRUREQUsKQmdnckJnRUZCUWNEQVRBTkJna3Foa2lHOXcwQkFRc0ZBQU9DQWdFQWtRcjg3U0ZONjV3bUYwMXlSUGJZNUg3dAowTENvcE95cU4yYy92WTNEaElJRWVBNTZmV3IvS0RRTkphcldRRThyb1JQZ1JlUVM5ZHlISTdQa3hJUzZXY2s5ClhIRHBFTUdVZ1BuQkFaOVZHd2ZjUUtvV2JrT0NUZHd4ZzREZFM3Rnd1cVgzdms2RWlkTWc5d2FURHZRYkxTOEkKbjFUWkpQYmJiRVVkYW1JUjMrQzI4NW5FNmQ5Y3cyZHRqcmlmVDM0STZGaThUc0FwL3hNb1ZkNXVjUEFhTHl1Sgp4WHZvOC9mSUVMRjRvUDZPQW8wQ3VuSU1rMWZuLzAyNjQ5QWhSZFFCdVZPaENGRUxNS051cFhiSmhQYkxESDlYClJRbmhIem8xeXIvNEVJTldoNUcxNkgyaCtpMFpQM05MUXl4Q0ZOL3FTNXFEaDI3eHV2ZmtSZ0toSXZpYW9abUUKbVNBeU40U1Fxb09mY1d1bURZUTBVVVUzMDh4cVFSUWNIem9aUHBlRytNNFZ3KzYxVGxFSkN1Y1VjR1FhMDRSQgpJeHhqVXV6NmFpb3FkT2NwbjIvbStDZ1Jjb0VpUk4ySUVUZW94MEV5ZkR3RU05K3pmSWQ1cEhRY3Y5ejhMWi9aCk9qQ0k4eVB0S1ZhTGl0YmZtbHg1TEErd2FKeUZ0b0JKNkdvVUR0eU04MHVORUxjL0ZCWmRKckoyYlhnRjh6dkoKOU9HK29XRnpMN0JTQXYwSDZoL29nQWpsNTIrMHBaMkQyd3JJWHYvbTZQV3VrcElMOE5sSFdsZlQ3Zkt2YzNNZgpsMmthOTdDK2NnSkRITm5WOGlUUjd2UTlad2FudE1Genp3dnVVQlB6VGtrdlVuanVMQ3g0VEtsaElhbVVNZUd0CnM4elVibC9scHpIc3BIdUxxbDg9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"
       },
       "private_key": {
        "inline_bytes": "W3JlZGFjdGVkXQ=="
       }
      }
     }
    }

[johnlanni]

private key 是为了安全隐藏的 不会把真实的输出在configdump里

[alexzzh]

private key 是为了安全隐藏的 不会把真实的输出在configdump里

理解了

[alexzzh]

• 如果使用 curl -k https://ip:port/foo 访问https资源提示被reset，有可能是因为访问时没有使用正确的域名，需要将 hostname: "*.foobar.com" 给移除，以提供不带域名的https支持

- name: foobar-https
      hostname: "*.foobar.com"
      port: 443
      protocol: HTTPS
        allowedRoutes:
        namespaces:
          from: All
      tls:
        certificateRefs:
        - kind: Secret
          name: wildcard-foobar-com
        mode: Terminate