K8s安装，hostNetwork模式部署时，支持配置监听端口，以避免冲突 #734
Assignees
Labels
Comments
|
我想认领此issue
在helm values.yaml中追加
|
|
@Uncle-Justice hostnetwork模式下，容器监听什么端口，宿主机上就暴露什么端口，所以只需要配置 httpPort 和 httpsPort 即可，不过代码中可能有一些 hardcord 依赖了 80和443，实现时可以检查下。 |
|
@johnlanni 请问hostNetwork模式下，对gateway的访问只通过 以及进入node中查看端口映射情况，也只观察到gateway对应的LoadBalancer产生的DNAT规则，而我预期的情况是在hostetwork下应该出现 root@higress-control-plane:/# iptables -nvL -t nat | grep DNAT
0 0 DNAT tcp -- * * 0.0.0.0/0 172.19.0.1 tcp dpt:53 to:127.0.0.11:34179
98 8216 DNAT udp -- * * 0.0.0.0/0 172.19.0.1 udp dpt:53 to:127.0.0.11:41293
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* higress-system/higress-gateway:http2 */ tcp to:172.19.0.2:80
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* higress-system/higress-controller:http */ tcp to:10.244.0.5:8888
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* higress-system/higress-controller:grpc */ tcp to:10.244.0.5:15051
1 60 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* higress-system/higress-controller:https-dns */ tcp to:10.244.0.5:15012
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */ tcp to:10.244.0.2:53
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes:https */ tcp to:172.19.0.2:6443
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* higress-system/higress-controller:http-monitoring */ tcp to:10.244.0.5:15014
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:metrics */ tcp to:10.244.0.2:9153
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:metrics */ tcp to:10.244.0.4:9153
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* higress-system/higress-controller:grpc-xds */ tcp to:10.244.0.5:15010
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */ tcp to:10.244.0.4:53
32 3255 DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */ udp to:10.244.0.4:53
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* higress-system/higress-controller:https-webhook */ tcp to:10.244.0.5:15017
39 3939 DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */ udp to:10.244.0.2:53
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* higress-system/higress-gateway:https */ tcp to:172.19.0.2:443原因可能是因为 gateway pod的信息看上去没什么问题，ip确实是与node ip一致 root@higress-control-plane:/# kubectl describe pod higress-gateway -n higress-system
Name: higress-gateway-7cbb7cd79d-gzfql
Namespace: higress-system
Priority: 0
Service Account: higress-gateway
Node: higress-control-plane/172.19.0.2
Start Time: Sun, 04 Feb 2024 05:37:55 +0000
Labels: app=higress-gateway
higress=higress-system-higress-gateway
pod-template-hash=7cbb7cd79d
sidecar.istio.io/inject=false
Annotations: prometheus.io/path: /stats/prometheus
prometheus.io/port: 15020
prometheus.io/scrape: true
sidecar.istio.io/inject: false
Status: Running
IP: 172.19.0.2
IPs:
IP: 172.19.0.2
Controlled By: ReplicaSet/higress-gateway-7cbb7cd79d
Containers:
higress-gateway:
Container ID: containerd://85054bb67dc88d63502c7905799790ad95b8f6fb683878a5b64c11fca26406fc
Image: higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/gateway:sha-87c39d3
Image ID: higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/gateway@sha256:6184a7d584f4bc59cc9de37afcdafa0a727921a886db64bfaccff705c7cafa3a
Ports: 15090/TCP, 80/TCP, 443/TCP
Host Ports: 15090/TCP, 80/TCP, 443/TCP
Args:
proxy
router
--domain
$(POD_NAMESPACE).svc.cluster.local
--proxyLogLevel=warning
--proxyComponentLogLevel=misc:error
--log_output_level=all:info
--serviceCluster=higress-gateway
State: Running
Started: Sun, 04 Feb 2024 05:42:40 +0000
Ready: True
Restart Count: 0
Readiness: http-get http://:15021/healthz/ready delay=1s timeout=3s period=2s #success=1 #failure=30
Environment:
NODE_NAME: (v1:spec.nodeName)
POD_NAME: higress-gateway-7cbb7cd79d-gzfql (v1:metadata.name)
POD_NAMESPACE: higress-system (v1:metadata.namespace)
INSTANCE_IP: (v1:status.podIP)
HOST_IP: (v1:status.hostIP)
SERVICE_ACCOUNT: (v1:spec.serviceAccountName)
PILOT_XDS_SEND_TIMEOUT: 60s
PROXY_XDS_VIA_AGENT: true
ENABLE_INGRESS_GATEWAY_SDS: false
JWT_POLICY: third-party-jwt
ISTIO_META_HTTP10: 1
ISTIO_META_CLUSTER_ID: Kubernetes
INSTANCE_NAME: higress-gateway
Mounts:
/etc/istio/config from config (rw)
/etc/istio/pod from podinfo (rw)
/etc/istio/proxy from proxy-socket (rw)
/var/lib/istio/data from istio-data (rw)
/var/run/secrets/istio from istio-ca-root-cert (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fhnwr (ro)
/var/run/secrets/tokens from istio-token (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
istio-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 43200
istio-ca-root-cert:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: higress-ca-root-cert
Optional: false
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: higress-config
Optional: false
istio-data:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
proxy-socket:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
podinfo:
Type: DownwardAPI (a volume populated by information about the pod)
Items:
metadata.labels -> labels
metadata.annotations -> annotations
requests.cpu -> cpu-request
limits.cpu -> cpu-limit
kube-api-access-fhnwr:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 48m default-scheduler Successfully assigned higress-system/higress-gateway-7cbb7cd79d-gzfql to higress-control-plane
Warning FailedMount 47m (x7 over 48m) kubelet MountVolume.SetUp failed for volume "istio-ca-root-cert" : configmap "higress-ca-root-cert" not found
Normal Pulling 47m kubelet Pulling image "higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/gateway:sha-87c39d3"
Normal Pulled 43m kubelet Successfully pulled image "higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/gateway:sha-87c39d3" in 3m40.802395633s
Normal Created 43m kubelet Created container higress-gateway
Normal Started 43m kubelet Started container higress-gateway
|
|
开启hostnetwork后，容器直接在宿主机上监听端口，你在宿主机上netstat -lntp没看到envoy监听的80/443吗，如果没看到，是不是因为没有创建ingress？或者那里没配置对 |
|
@johnlanni 对gateway容器的service创建ingress之后，的确观察到宿主机上出现了envoy在监听80/443，不过如果我把gateway容器监听的端口改成非80，netstat显示envoy依然监听的80 所以higress的逻辑是，higress-controller监听到了ingress资源创建后，才会触发envoy-gateway，使其正常工作，而不是gateway容器创建成功之后？ 如果不论gateway容器设置的监听端口是几，envoy-gateway实际监听的都是80端口，那你之前说的hardcode问题应该主要可能出现在控制器ingress与envoy交互的那部分？ |
|
higress是通过把ingress转换为istio gateway api来控制数据面的端口监听的，可以看下ingress转换实现里convertGateway部分的逻辑哈 |
|
手动修改 但是如果是用k8s安装，配置数据面的端口要与higress-gateway pod容器yaml设置的监听接口保持一致，应该如何实现呢？要在istio这一层面去实现读取gateway pod配置了哪些端口吗？感觉istio应该主要还是负责把新的配置推送到数据面吧？ |
|
@Uncle-Justice 通过环境变量来控制监听端口，并且在Helm模版中根据helm参数设置这个环境变量，并跟yaml中端口保持一致即可 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Originally posted by @johnlanni in #382 (comment)
The text was updated successfully, but these errors were encountered: