error opening relay circuit: NO_RESERVATION #411

renslk · 2022-12-21T10:51:50Z

What happened:
edge侧和cloud侧的通道无法建立

How to reproduce it (as minimally and precisely as possible):
04-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: edgemesh-agent-cfg
  namespace: kubeedge
  labels:
    k8s-app: kubeedge
    kubeedge: edgemesh-agent
data:
  edgemesh-agent.yaml: |
    # For more detailed configuration, please refer to: https://edgemesh.netlify.app/reference/config-items.html#edgemesh-agent-cfg
    modules:
      edgeProxy:
        enable: true
      edgeTunnel:
        enable: true
        relayNodes:
        - nodeName: cn-west-1-a-ed1-master01
          advertiseAddress:
          - <公网ip>
        #- nodeName: cloud-node01
        #  advertiseAddress:
        #  - 10.128.2.221
        #  - 3.3.3.3

公网ip:20006端口映射到cn-west-1-a-ed1-master01 20006端口
同时cn-west-1-a-ed1-master01也是部署cloud core节点

Anything else we need to know?:
master上的edgemesh-agent日志:

I1221 18:07:11.818744       1 server.go:55] Version: v1.12.0-dirty
I1221 18:07:11.818834       1 server.go:89] [1] Prepare agent to run
I1221 18:07:11.819139       1 netif.go:96] bridge device edgemesh0 already exists
I1221 18:07:11.819250       1 server.go:93] edgemesh-agent running on CloudMode
I1221 18:07:11.819260       1 server.go:96] [2] New clients
W1221 18:07:11.819282       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1221 18:07:11.820161       1 server.go:103] [3] Register beehive modules
W1221 18:07:11.820196       1 module.go:37] Module EdgeDNS is disabled, do not register
I1221 18:07:11.913786       1 server.go:66] Using userspace Proxier.
I1221 18:07:13.219336       1 module.go:34] Module EdgeProxy registered successfully
I1221 18:07:13.513875       1 module.go:159] I'm {12D3KooWL5cwYP6rRt2vccqk3VUFo2kzkvYfMzrNCDknkN8J8jxW: [/ip4/10.50.166.41/tcp/20006 /ip4/127.0.0.1/tcp/20006 /ip4/47.101.49.231/tcp/20006]}
I1221 18:07:13.514188       1 module.go:168] Run as a relay node
I1221 18:07:13.514497       1 module.go:181] Bootstrapping the DHT
I1221 18:07:13.913577       1 tunnel.go:63] Starting MDNS discovery service
I1221 18:07:13.913661       1 tunnel.go:76] Starting DHT discovery service
I1221 18:07:13.913827       1 module.go:34] Module EdgeTunnel registered successfully
I1221 18:07:13.913853       1 server.go:109] [4] Start all modules
I1221 18:07:13.913964       1 core.go:24] Starting module EdgeProxy
I1221 18:07:13.914009       1 core.go:24] Starting module EdgeTunnel
I1221 18:07:13.914303       1 tunnel.go:447] Starting relay finder
I1221 18:07:13.914891       1 config.go:135] "Starting endpoints config controller"
I1221 18:07:13.914968       1 shared_informer.go:240] Waiting for caches to sync for endpoints config
I1221 18:07:13.915017       1 config.go:317] "Starting service config controller"
I1221 18:07:13.915028       1 shared_informer.go:240] Waiting for caches to sync for service config
I1221 18:07:13.916090       1 loadbalancer.go:239] "Starting loadBalancer destinationRule controller"
I1221 18:07:13.916165       1 shared_informer.go:240] Waiting for caches to sync for loadBalancer destinationRule
I1221 18:07:14.216882       1 shared_informer.go:247] Caches are synced for loadBalancer destinationRule 
I1221 18:07:14.514761       1 shared_informer.go:247] Caches are synced for service config 
I1221 18:07:14.617696       1 shared_informer.go:247] Caches are synced for endpoints config 
I1221 18:07:14.612961       1 tunnel.go:118] [MDNS] Discovery found peer: {12D3KooWEME3ybEgNcj7LZhPYAv6sV3Uu99UHM6r5Q63ThRWpBzC: [/ip4/10.50.166.42/tcp/20006 /ip4/127.0.0.1/tcp/20006]}
I1221 18:07:14.622361       1 tunnel.go:130] [MDNS] New stream between peer {12D3KooWEME3ybEgNcj7LZhPYAv6sV3Uu99UHM6r5Q63ThRWpBzC: [/ip4/127.0.0.1/tcp/20006 /ip4/10.50.166.42/tcp/20006 /ip4/172.17.0.1/tcp/20006 /ip4/172.31.219.180/tcp/20006 /ip4/172.31.225.165/tcp/20006 /ip4/10.50.166.182/tcp/20006 /ip4/172.31.250.124/tcp/20006 /ip4/172.31.52.197/tcp/20006 /ip4/172.31.137.160/tcp/20006 /ip4/172.31.67.67/tcp/20006 /ip4/10.50.166.183/tcp/20006 /ip4/172.31.39.75/tcp/20006 /ip4/172.31.0.10/tcp/20006 /ip4/172.31.34.128/tcp/20006 /ip4/172.31.76.232/tcp/20006 /ip4/172.31.240.234/tcp/20006 /ip4/172.31.12.17/tcp/20006 /ip4/172.31.55.228/tcp/20006 /ip4/172.31.207.176/tcp/20006 /ip4/172.31.128.38/tcp/20006 /ip4/172.31.200.42/tcp/20006 /ip4/172.31.117.16/tcp/20006 /ip4/172.31.152.23/tcp/20006 /ip4/172.31.249.123/tcp/20006 /ip4/172.31.88.57/tcp/20006 /ip4/172.31.10.228/tcp/20006 /ip4/172.31.153.142/tcp/20006 /ip4/172.31.0.1/tcp/20006 /ip4/172.31.140.138/tcp/20006 /ip4/172.31.218.243/tcp/20006 /ip4/172.31.137.56/tcp/20006 /ip4/172.31.145.155/tcp/20006 /ip4/172.31.85.17/tcp/20006 /ip4/172.31.242.218/tcp/20006 /ip4/172.31.31.226/tcp/20006 /ip4/172.31.40.241/tcp/20006 /ip4/172.31.231.78/tcp/20006 /ip4/172.31.106.0/tcp/20006 /ip4/169.254.96.16/tcp/20006 /ip4/172.29.192.192/tcp/20006]} success
I1221 18:07:14.712683       1 tunnel.go:166] [MDNS] Discovery to cn-west-1-a-ed1-worker01 : {12D3KooWEME3ybEgNcj7LZhPYAv6sV3Uu99UHM6r5Q63ThRWpBzC: [/ip4/127.0.0.1/tcp/20006 /ip4/10.50.166.42/tcp/20006 /ip4/172.17.0.1/tcp/20006 /ip4/172.31.219.180/tcp/20006 /ip4/172.31.225.165/tcp/20006 /ip4/10.50.166.182/tcp/20006 /ip4/172.31.250.124/tcp/20006 /ip4/172.31.52.197/tcp/20006 /ip4/172.31.137.160/tcp/20006 /ip4/172.31.67.67/tcp/20006 /ip4/10.50.166.183/tcp/20006 /ip4/172.31.39.75/tcp/20006 /ip4/172.31.0.10/tcp/20006 /ip4/172.31.34.128/tcp/20006 /ip4/172.31.76.232/tcp/20006 /ip4/172.31.240.234/tcp/20006 /ip4/172.31.12.17/tcp/20006 /ip4/172.31.55.228/tcp/20006 /ip4/172.31.207.176/tcp/20006 /ip4/172.31.128.38/tcp/20006 /ip4/172.31.200.42/tcp/20006 /ip4/172.31.117.16/tcp/20006 /ip4/172.31.152.23/tcp/20006 /ip4/172.31.249.123/tcp/20006 /ip4/172.31.88.57/tcp/20006 /ip4/172.31.10.228/tcp/20006 /ip4/172.31.153.142/tcp/20006 /ip4/172.31.0.1/tcp/20006 /ip4/172.31.140.138/tcp/20006 /ip4/172.31.218.243/tcp/20006 /ip4/172.31.137.56/tcp/20006 /ip4/172.31.145.155/tcp/20006 /ip4/172.31.85.17/tcp/20006 /ip4/172.31.242.218/tcp/20006 /ip4/172.31.31.226/tcp/20006 /ip4/172.31.40.241/tcp/20006 /ip4/172.31.231.78/tcp/20006 /ip4/172.31.106.0/tcp/20006 /ip4/169.254.96.16/tcp/20006 /ip4/172.29.192.192/tcp/20006]}
I1221 18:07:14.713008       1 tunnel.go:118] [MDNS] Discovery found peer: {12D3KooWRhrUY71vdSzeVhWFGPv246MQ27J1XpmwxQXtKhqyoG1D: [/ip4/10.50.166.43/tcp/20006]}
I1221 18:07:14.713663       1 tunnel.go:130] [MDNS] New stream between peer {12D3KooWRhrUY71vdSzeVhWFGPv246MQ27J1XpmwxQXtKhqyoG1D: [/ip4/127.0.0.1/tcp/20006 /ip4/10.50.166.43/tcp/20006 /ip4/172.17.0.1/tcp/20006 /ip4/172.31.88.57/tcp/20006 /ip4/172.31.67.67/tcp/20006 /ip4/10.50.166.183/tcp/20006 /ip4/172.31.152.23/tcp/20006 /ip4/172.31.128.38/tcp/20006 /ip4/172.31.250.124/tcp/20006 /ip4/172.31.0.1/tcp/20006 /ip4/172.31.34.128/tcp/20006 /ip4/172.31.207.176/tcp/20006 /ip4/172.31.249.123/tcp/20006 /ip4/172.31.52.197/tcp/20006 /ip4/172.31.200.42/tcp/20006 /ip4/172.31.10.228/tcp/20006 /ip4/172.31.240.234/tcp/20006 /ip4/172.31.225.165/tcp/20006 /ip4/10.50.166.182/tcp/20006 /ip4/172.31.117.16/tcp/20006 /ip4/172.31.12.17/tcp/20006 /ip4/172.31.76.232/tcp/20006 /ip4/172.31.137.160/tcp/20006 /ip4/172.31.153.142/tcp/20006 /ip4/172.31.39.75/tcp/20006 /ip4/172.31.140.138/tcp/20006 /ip4/172.31.219.180/tcp/20006 /ip4/172.31.0.10/tcp/20006 /ip4/172.31.55.228/tcp/20006 /ip4/172.31.218.243/tcp/20006 /ip4/172.31.137.56/tcp/20006 /ip4/172.31.145.155/tcp/20006 /ip4/172.31.85.17/tcp/20006 /ip4/172.31.242.218/tcp/20006 /ip4/172.31.31.226/tcp/20006 /ip4/172.31.40.241/tcp/20006 /ip4/172.31.231.78/tcp/20006 /ip4/172.31.106.0/tcp/20006 /ip4/169.254.96.16/tcp/20006 /ip4/172.29.135.0/tcp/20006]} success
I1221 18:07:14.847561       1 tunnel.go:166] [MDNS] Discovery to cn-west-1-a-ed1-worker02 : {12D3KooWRhrUY71vdSzeVhWFGPv246MQ27J1XpmwxQXtKhqyoG1D: [/ip4/127.0.0.1/tcp/20006 /ip4/10.50.166.43/tcp/20006 /ip4/172.17.0.1/tcp/20006 /ip4/172.31.88.57/tcp/20006 /ip4/172.31.67.67/tcp/20006 /ip4/10.50.166.183/tcp/20006 /ip4/172.31.152.23/tcp/20006 /ip4/172.31.128.38/tcp/20006 /ip4/172.31.250.124/tcp/20006 /ip4/172.31.0.1/tcp/20006 /ip4/172.31.34.128/tcp/20006 /ip4/172.31.207.176/tcp/20006 /ip4/172.31.249.123/tcp/20006 /ip4/172.31.52.197/tcp/20006 /ip4/172.31.200.42/tcp/20006 /ip4/172.31.10.228/tcp/20006 /ip4/172.31.240.234/tcp/20006 /ip4/172.31.225.165/tcp/20006 /ip4/10.50.166.182/tcp/20006 /ip4/172.31.117.16/tcp/20006 /ip4/172.31.12.17/tcp/20006 /ip4/172.31.76.232/tcp/20006 /ip4/172.31.137.160/tcp/20006 /ip4/172.31.153.142/tcp/20006 /ip4/172.31.39.75/tcp/20006 /ip4/172.31.140.138/tcp/20006 /ip4/172.31.219.180/tcp/20006 /ip4/172.31.0.10/tcp/20006 /ip4/172.31.55.228/tcp/20006 /ip4/172.31.218.243/tcp/20006 /ip4/172.31.137.56/tcp/20006 /ip4/172.31.145.155/tcp/20006 /ip4/172.31.85.17/tcp/20006 /ip4/172.31.242.218/tcp/20006 /ip4/172.31.31.226/tcp/20006 /ip4/172.31.40.241/tcp/20006 /ip4/172.31.231.78/tcp/20006 /ip4/172.31.106.0/tcp/20006 /ip4/169.254.96.16/tcp/20006 /ip4/172.29.135.0/tcp/20006]}
I1221 18:07:15.112519       1 tunnel.go:175] Discovery service got a new stream from {12D3KooWRhrUY71vdSzeVhWFGPv246MQ27J1XpmwxQXtKhqyoG1D: [/ip4/10.50.166.41/tcp/41482]}
I1221 18:07:15.112644       1 tunnel.go:204] [MDNS] Discovery from cn-west-1-a-ed1-worker02 : {12D3KooWRhrUY71vdSzeVhWFGPv246MQ27J1XpmwxQXtKhqyoG1D: [/ip4/10.50.166.41/tcp/41482]}

在cloud侧的work节点上edgemesh-agent日志：

I1221 18:07:11.082813       1 server.go:55] Version: v1.12.0-dirty
I1221 18:07:11.082890       1 server.go:89] [1] Prepare agent to run
I1221 18:07:11.083138       1 netif.go:96] bridge device edgemesh0 already exists
I1221 18:07:11.083280       1 server.go:93] edgemesh-agent running on CloudMode
I1221 18:07:11.083304       1 server.go:96] [2] New clients
W1221 18:07:11.083324       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1221 18:07:11.178863       1 server.go:103] [3] Register beehive modules
W1221 18:07:11.178912       1 module.go:37] Module EdgeDNS is disabled, do not register
I1221 18:07:11.181652       1 server.go:66] Using userspace Proxier.
I1221 18:07:12.181417       1 module.go:34] Module EdgeProxy registered successfully
I1221 18:07:13.180717       1 module.go:159] I'm {12D3KooWEME3ybEgNcj7LZhPYAv6sV3Uu99UHM6r5Q63ThRWpBzC: [/ip4/10.50.166.42/tcp/20006 /ip4/127.0.0.1/tcp/20006]}
I1221 18:07:13.180847       1 module.go:181] Bootstrapping the DHT
I1221 18:07:13.277358       1 tunnel.go:387] [Bootstrap] bootstrapping to 12D3KooWL5cwYP6rRt2vccqk3VUFo2kzkvYfMzrNCDknkN8J8jxW
I1221 18:07:13.578548       1 tunnel.go:397] [Bootstrap] success bootstrapped with {12D3KooWL5cwYP6rRt2vccqk3VUFo2kzkvYfMzrNCDknkN8J8jxW: [/ip4/47.101.49.231/tcp/20006]}
I1221 18:07:13.680250       1 tunnel.go:63] Starting MDNS discovery service
I1221 18:07:13.680291       1 tunnel.go:76] Starting DHT discovery service
I1221 18:07:13.680378       1 module.go:34] Module EdgeTunnel registered successfully
I1221 18:07:13.680391       1 server.go:109] [4] Start all modules
I1221 18:07:13.680535       1 tunnel.go:447] Starting relay finder
I1221 18:07:13.680541       1 core.go:24] Starting module EdgeProxy
I1221 18:07:13.680617       1 core.go:24] Starting module EdgeTunnel
I1221 18:07:13.681127       1 loadbalancer.go:239] "Starting loadBalancer destinationRule controller"
I1221 18:07:13.681166       1 shared_informer.go:240] Waiting for caches to sync for loadBalancer destinationRule
I1221 18:07:13.681167       1 config.go:317] "Starting service config controller"
I1221 18:07:13.681195       1 shared_informer.go:240] Waiting for caches to sync for service config
I1221 18:07:13.681134       1 config.go:135] "Starting endpoints config controller"
I1221 18:07:13.681284       1 shared_informer.go:240] Waiting for caches to sync for endpoints config
I1221 18:07:13.978507       1 tunnel.go:118] [MDNS] Discovery found peer: {12D3KooWRhrUY71vdSzeVhWFGPv246MQ27J1XpmwxQXtKhqyoG1D: []}
I1221 18:07:13.977124       1 tunnel.go:175] Discovery service got a new stream from {12D3KooWL5cwYP6rRt2vccqk3VUFo2kzkvYfMzrNCDknkN8J8jxW: [/ip4/47.101.49.231/tcp/20006]}
I1221 18:07:13.979633       1 tunnel.go:204] [MDNS] Discovery from cn-west-1-a-ed1-master01 : {12D3KooWL5cwYP6rRt2vccqk3VUFo2kzkvYfMzrNCDknkN8J8jxW: [/ip4/47.101.49.231/tcp/20006]}
I1221 18:07:14.181566       1 shared_informer.go:247] Caches are synced for loadBalancer destinationRule 
I1221 18:07:14.477170       1 shared_informer.go:247] Caches are synced for endpoints config 
I1221 18:07:14.581479       1 shared_informer.go:247] Caches are synced for service config 
I1221 18:07:14.977479       1 tunnel.go:175] Discovery service got a new stream from {12D3KooWRhrUY71vdSzeVhWFGPv246MQ27J1XpmwxQXtKhqyoG1D: [/ip4/10.50.166.43/tcp/20006]}
I1221 18:07:14.977732       1 tunnel.go:204] [MDNS] Discovery from cn-west-1-a-ed1-worker02 : {12D3KooWRhrUY71vdSzeVhWFGPv246MQ27J1XpmwxQXtKhqyoG1D: [/ip4/10.50.166.43/tcp/20006]}
I1221 18:07:14.977740       1 tunnel.go:130] [MDNS] New stream between peer {12D3KooWRhrUY71vdSzeVhWFGPv246MQ27J1XpmwxQXtKhqyoG1D: [/ip4/127.0.0.1/tcp/20006 /ip4/10.50.166.43/tcp/20006 /ip4/172.17.0.1/tcp/20006 /ip4/172.31.88.57/tcp/20006 /ip4/172.31.67.67/tcp/20006 /ip4/10.50.166.183/tcp/20006 /ip4/172.31.152.23/tcp/20006 /ip4/172.31.128.38/tcp/20006 /ip4/172.31.250.124/tcp/20006 /ip4/172.31.0.1/tcp/20006 /ip4/172.31.34.128/tcp/20006 /ip4/172.31.207.176/tcp/20006 /ip4/172.31.249.123/tcp/20006 /ip4/172.31.52.197/tcp/20006 /ip4/172.31.200.42/tcp/20006 /ip4/172.31.10.228/tcp/20006 /ip4/172.31.240.234/tcp/20006 /ip4/172.31.225.165/tcp/20006 /ip4/10.50.166.182/tcp/20006 /ip4/172.31.117.16/tcp/20006 /ip4/172.31.12.17/tcp/20006 /ip4/172.31.76.232/tcp/20006 /ip4/172.31.137.160/tcp/20006 /ip4/172.31.153.142/tcp/20006 /ip4/172.31.39.75/tcp/20006 /ip4/172.31.140.138/tcp/20006 /ip4/172.31.219.180/tcp/20006 /ip4/172.31.0.10/tcp/20006 /ip4/172.31.55.228/tcp/20006 /ip4/172.31.218.243/tcp/20006 /ip4/172.31.137.56/tcp/20006 /ip4/172.31.145.155/tcp/20006 /ip4/172.31.85.17/tcp/20006 /ip4/172.31.242.218/tcp/20006 /ip4/172.31.31.226/tcp/20006 /ip4/172.31.40.241/tcp/20006 /ip4/172.31.231.78/tcp/20006 /ip4/172.31.106.0/tcp/20006 /ip4/169.254.96.16/tcp/20006 /ip4/172.29.135.0/tcp/20006]} success
I1221 18:07:15.130510       1 tunnel.go:166] [MDNS] Discovery to cn-west-1-a-ed1-worker02 : {12D3KooWRhrUY71vdSzeVhWFGPv246MQ27J1XpmwxQXtKhqyoG1D: [/ip4/127.0.0.1/tcp/20006 /ip4/10.50.166.43/tcp/20006 /ip4/172.17.0.1/tcp/20006 /ip4/172.31.88.57/tcp/20006 /ip4/172.31.67.67/tcp/20006 /ip4/10.50.166.183/tcp/20006 /ip4/172.31.152.23/tcp/20006 /ip4/172.31.128.38/tcp/20006 /ip4/172.31.250.124/tcp/20006 /ip4/172.31.0.1/tcp/20006 /ip4/172.31.34.128/tcp/20006 /ip4/172.31.207.176/tcp/20006 /ip4/172.31.249.123/tcp/20006 /ip4/172.31.52.197/tcp/20006 /ip4/172.31.200.42/tcp/20006 /ip4/172.31.10.228/tcp/20006 /ip4/172.31.240.234/tcp/20006 /ip4/172.31.225.165/tcp/20006 /ip4/10.50.166.182/tcp/20006 /ip4/172.31.117.16/tcp/20006 /ip4/172.31.12.17/tcp/20006 /ip4/172.31.76.232/tcp/20006 /ip4/172.31.137.160/tcp/20006 /ip4/172.31.153.142/tcp/20006 /ip4/172.31.39.75/tcp/20006 /ip4/172.31.140.138/tcp/20006 /ip4/172.31.219.180/tcp/20006 /ip4/172.31.0.10/tcp/20006 /ip4/172.31.55.228/tcp/20006 /ip4/172.31.218.243/tcp/20006 /ip4/172.31.137.56/tcp/20006 /ip4/172.31.145.155/tcp/20006 /ip4/172.31.85.17/tcp/20006 /ip4/172.31.242.218/tcp/20006 /ip4/172.31.31.226/tcp/20006 /ip4/172.31.40.241/tcp/20006 /ip4/172.31.231.78/tcp/20006 /ip4/172.31.106.0/tcp/20006 /ip4/169.254.96.16/tcp/20006 /ip4/172.29.135.0/tcp/20006]}
I1221 18:07:15.130732       1 tunnel.go:118] [MDNS] Discovery found peer: {12D3KooWL5cwYP6rRt2vccqk3VUFo2kzkvYfMzrNCDknkN8J8jxW: [/ip4/47.101.49.231/tcp/20006 /ip4/10.50.166.41/tcp/20006 /ip4/127.0.0.1/tcp/20006]}
I1221 18:07:15.177370       1 tunnel.go:130] [MDNS] New stream between peer {12D3KooWL5cwYP6rRt2vccqk3VUFo2kzkvYfMzrNCDknkN8J8jxW: [/ip4/127.0.0.1/tcp/20006 /ip4/10.50.166.41/tcp/20006 /ip4/172.17.0.1/tcp/20006 /ip4/172.31.10.228/tcp/20006 /ip4/172.31.88.57/tcp/20006 /ip4/172.31.39.75/tcp/20006 /ip4/172.31.12.17/tcp/20006 /ip4/172.31.0.1/tcp/20006 /ip4/172.31.200.42/tcp/20006 /ip4/172.31.250.124/tcp/20006 /ip4/172.31.117.16/tcp/20006 /ip4/172.31.225.165/tcp/20006 /ip4/10.50.166.182/tcp/20006 /ip4/172.31.52.197/tcp/20006 /ip4/172.31.0.10/tcp/20006 /ip4/172.31.67.67/tcp/20006 /ip4/10.50.166.183/tcp/20006 /ip4/172.31.34.128/tcp/20006 /ip4/172.31.137.160/tcp/20006 /ip4/172.31.153.142/tcp/20006 /ip4/172.31.55.228/tcp/20006 /ip4/172.31.76.232/tcp/20006 /ip4/172.31.152.23/tcp/20006 /ip4/172.31.240.234/tcp/20006 /ip4/172.31.128.38/tcp/20006 /ip4/172.31.207.176/tcp/20006 /ip4/172.31.140.138/tcp/20006 /ip4/172.31.219.180/tcp/20006 /ip4/172.31.249.123/tcp/20006 /ip4/172.31.218.243/tcp/20006 /ip4/172.31.137.56/tcp/20006 /ip4/172.31.145.155/tcp/20006 /ip4/172.31.85.17/tcp/20006 /ip4/172.31.242.218/tcp/20006 /ip4/172.31.31.226/tcp/20006 /ip4/172.31.40.241/tcp/20006 /ip4/172.31.231.78/tcp/20006 /ip4/172.31.106.0/tcp/20006 /ip4/169.254.96.16/tcp/20006 /ip4/172.29.214.128/tcp/20006]} success
I1221 18:07:15.190364       1 tunnel.go:166] [MDNS] Discovery to cn-west-1-a-ed1-master01 : {12D3KooWL5cwYP6rRt2vccqk3VUFo2kzkvYfMzrNCDknkN8J8jxW: [/ip4/127.0.0.1/tcp/20006 /ip4/10.50.166.41/tcp/20006 /ip4/172.17.0.1/tcp/20006 /ip4/172.31.10.228/tcp/20006 /ip4/172.31.88.57/tcp/20006 /ip4/172.31.39.75/tcp/20006 /ip4/172.31.12.17/tcp/20006 /ip4/172.31.0.1/tcp/20006 /ip4/172.31.200.42/tcp/20006 /ip4/172.31.250.124/tcp/20006 /ip4/172.31.117.16/tcp/20006 /ip4/172.31.225.165/tcp/20006 /ip4/10.50.166.182/tcp/20006 /ip4/172.31.52.197/tcp/20006 /ip4/172.31.0.10/tcp/20006 /ip4/172.31.67.67/tcp/20006 /ip4/10.50.166.183/tcp/20006 /ip4/172.31.34.128/tcp/20006 /ip4/172.31.137.160/tcp/20006 /ip4/172.31.153.142/tcp/20006 /ip4/172.31.55.228/tcp/20006 /ip4/172.31.76.232/tcp/20006 /ip4/172.31.152.23/tcp/20006 /ip4/172.31.240.234/tcp/20006 /ip4/172.31.128.38/tcp/20006 /ip4/172.31.207.176/tcp/20006 /ip4/172.31.140.138/tcp/20006 /ip4/172.31.219.180/tcp/20006 /ip4/172.31.249.123/tcp/20006 /ip4/172.31.218.243/tcp/20006 /ip4/172.31.137.56/tcp/20006 /ip4/172.31.145.155/tcp/20006 /ip4/172.31.85.17/tcp/20006 /ip4/172.31.242.218/tcp/20006 /ip4/172.31.31.226/tcp/20006 /ip4/172.31.40.241/tcp/20006 /ip4/172.31.231.78/tcp/20006 /ip4/172.31.106.0/tcp/20006 /ip4/169.254.96.16/tcp/20006 /ip4/172.29.214.128/tcp/20006]}

在edge node上的edgemesh-agent日志:

I1221 18:25:38.546320       1 server.go:55] Version: v1.12.0-dirty
I1221 18:25:38.546469       1 server.go:89] [1] Prepare agent to run
I1221 18:25:38.641896       1 netif.go:96] bridge device edgemesh0 already exists
I1221 18:25:38.642216       1 server.go:93] edgemesh-agent running on EdgeMode
I1221 18:25:38.642314       1 server.go:96] [2] New clients
I1221 18:25:38.645576       1 server.go:103] [3] Register beehive modules
I1221 18:25:38.647241       1 module.go:34] Module EdgeDNS registered successfully
I1221 18:25:38.648123       1 server.go:66] Using userspace Proxier.
I1221 18:25:39.544579       1 module.go:34] Module EdgeProxy registered successfully
I1221 18:25:39.944778       1 module.go:159] I'm {12D3KooWLSDF9t796mnj5ziZxryrEGt7DDuqDx6pQHf541hKpt8M: [/ip4/10.52.166.44/tcp/20006 /ip4/127.0.0.1/tcp/20006]}
I1221 18:25:39.944979       1 module.go:181] Bootstrapping the DHT
I1221 18:25:39.945064       1 tunnel.go:387] [Bootstrap] bootstrapping to 12D3KooWL5cwYP6rRt2vccqk3VUFo2kzkvYfMzrNCDknkN8J8jxW
I1221 18:25:40.361073       1 tunnel.go:397] [Bootstrap] success bootstrapped with {12D3KooWL5cwYP6rRt2vccqk3VUFo2kzkvYfMzrNCDknkN8J8jxW: [/ip4/47.101.49.231/tcp/20006]}
I1221 18:25:40.443652       1 tunnel.go:63] Starting MDNS discovery service
I1221 18:25:40.443692       1 tunnel.go:76] Starting DHT discovery service
I1221 18:25:40.443761       1 module.go:34] Module EdgeTunnel registered successfully
I1221 18:25:40.443776       1 server.go:109] [4] Start all modules
I1221 18:25:40.443835       1 core.go:24] Starting module EdgeDNS
I1221 18:25:40.443989       1 core.go:24] Starting module EdgeProxy
I1221 18:25:40.444304       1 core.go:24] Starting module EdgeTunnel
I1221 18:25:40.446827       1 tunnel.go:447] Starting relay finder
I1221 18:25:40.447551       1 dns.go:34] Runs CoreDNS v1.8.0 as a local dns
I1221 18:25:40.448751       1 config.go:317] "Starting service config controller"
I1221 18:25:40.448800       1 shared_informer.go:240] Waiting for caches to sync for service config
I1221 18:25:40.541404       1 loadbalancer.go:239] "Starting loadBalancer destinationRule controller"
I1221 18:25:40.541447       1 shared_informer.go:240] Waiting for caches to sync for loadBalancer destinationRule
I1221 18:25:40.541557       1 config.go:135] "Starting endpoints config controller"
I1221 18:25:40.541590       1 shared_informer.go:240] Waiting for caches to sync for endpoints config
I1221 18:25:40.749691       1 shared_informer.go:247] Caches are synced for service config 
I1221 18:25:40.842356       1 tunnel.go:118] [DHT] Discovery found peer: {12D3KooWRhrUY71vdSzeVhWFGPv246MQ27J1XpmwxQXtKhqyoG1D: [/ip4/10.50.166.43/tcp/20006 /ip4/127.0.0.1/tcp/20006 /ip4/47.101.49.231/tcp/20006/p2p/12D3KooWL5cwYP6rRt2vccqk3VUFo2kzkvYfMzrNCDknkN8J8jxW/p2p-circuit]}
I1221 18:25:40.843935       1 shared_informer.go:247] Caches are synced for loadBalancer destinationRule 
I1221 18:25:41.142316       1 shared_informer.go:247] Caches are synced for endpoints config 
.:53 on 169.254.96.16
I1221 18:25:41.243474       1 log.go:184] [INFO] plugin/reload: Running configuration MD5 = ab8e188d63b1d1a5168d68e78fe4128d
I1221 18:25:41.541656       1 log.go:184] [INFO] 169.254.96.16:34390 - 45572 "HINFO IN 1615312138878258316.8711688076870925944. udp 57 false 512" NXDOMAIN qr,rd,ra 132 0.297784761s
I1221 18:25:42.945534       1 proxier.go:895] "Opened iptables from-containers public port for service" servicePortName="prometheus/prometheus-operated-nodeport:web" protocol=TCP nodePort=31640

E1221 18:25:45.843453       1 tunnel.go:121] [DHT] New stream between peer {12D3KooWRhrUY71vdSzeVhWFGPv246MQ27J1XpmwxQXtKhqyoG1D: [/ip4/10.50.166.43/tcp/20006 /ip4/127.0.0.1/tcp/20006]} err: failed to dial 12D3KooWRhrUY71vdSzeVhWFGPv246MQ27J1XpmwxQXtKhqyoG1D:
  * [/ip4/47.101.49.231/tcp/20006/p2p/12D3KooWL5cwYP6rRt2vccqk3VUFo2kzkvYfMzrNCDknkN8J8jxW/p2p-circuit] error opening relay circuit: NO_RESERVATION (204)
  * [/ip4/10.50.166.43/tcp/20006] dial tcp4 0.0.0.0:20006->10.50.166.43:20006: i/o timeout
I1221 18:25:45.843673       1 tunnel.go:118] [DHT] Discovery found peer: {12D3KooWEME3ybEgNcj7LZhPYAv6sV3Uu99UHM6r5Q63ThRWpBzC: [/ip4/127.0.0.1/tcp/20006 /ip4/47.101.49.231/tcp/20006/p2p/12D3KooWL5cwYP6rRt2vccqk3VUFo2kzkvYfMzrNCDknkN8J8jxW/p2p-circuit /ip4/10.50.166.42/tcp/20006]}
I1221 18:25:45.847683       1 proxier.go:916] "Opened iptables from-non-local public port for service" servicePortName="default/d-2048-3-dao-2048" protocol=TCP nodePort=31193

E1221 18:25:50.844276       1 tunnel.go:121] [DHT] New stream between peer {12D3KooWEME3ybEgNcj7LZhPYAv6sV3Uu99UHM6r5Q63ThRWpBzC: [/ip4/10.50.166.42/tcp/20006 /ip4/127.0.0.1/tcp/20006]} err: failed to dial 12D3KooWEME3ybEgNcj7LZhPYAv6sV3Uu99UHM6r5Q63ThRWpBzC:
  * [/ip4/47.101.49.231/tcp/20006/p2p/12D3KooWL5cwYP6rRt2vccqk3VUFo2kzkvYfMzrNCDknkN8J8jxW/p2p-circuit] error opening relay circuit: NO_RESERVATION (204)
  * [/ip4/10.50.166.42/tcp/20006] dial tcp4 0.0.0.0:20006->10.50.166.42:20006: i/o timeout
I1221 18:25:50.844398       1 tunnel.go:118] [DHT] Discovery found peer: {12D3KooWBqeyjCMAKbJViTh1tEoyPxj7XWD8SeuoJZ1zCZgXq954: [/ip4/192.168.100.132/tcp/20006 /ip4/127.0.0.1/tcp/20006 /ip4/192.168.100.13/tcp/54476 /ip4/47.101.49.231/tcp/20006/p2p/12D3KooWL5cwYP6rRt2vccqk3VUFo2kzkvYfMzrNCDknkN8J8jxW/p2p-circuit]}
E1221 18:25:55.845680       1 tunnel.go:121] [DHT] New stream between peer {12D3KooWBqeyjCMAKbJViTh1tEoyPxj7XWD8SeuoJZ1zCZgXq954: [/ip4/192.168.100.132/tcp/20006 /ip4/127.0.0.1/tcp/20006 /ip4/192.168.100.13/tcp/54476]} err: failed to dial 12D3KooWBqeyjCMAKbJViTh1tEoyPxj7XWD8SeuoJZ1zCZgXq954:
  * [/ip4/47.101.49.231/tcp/20006/p2p/12D3KooWL5cwYP6rRt2vccqk3VUFo2kzkvYfMzrNCDknkN8J8jxW/p2p-circuit] error opening relay circuit: NO_RESERVATION (204)
  * [/ip4/192.168.100.132/tcp/20006] dial tcp4 0.0.0.0:20006->192.168.100.132:20006: i/o timeout
  * [/ip4/192.168.100.13/tcp/54476] dial tcp4 0.0.0.0:20006->192.168.100.13:54476: i/o timeout
I1221 18:25:55.845936       1 tunnel.go:118] [DHT] Discovery found peer: {12D3KooWEg5jkGp1aN49CiiX8SQeahsWCCsjPNr2WzRPeYecd9a4: [/ip4/192.168.1.101/tcp/20006 /ip4/127.0.0.1/tcp/20006 /ip4/192.168.100.12/tcp/24008 /ip4/47.101.49.231/tcp/20006/p2p/12D3KooWL5cwYP6rRt2vccqk3VUFo2kzkvYfMzrNCDknkN8J8jxW/p2p-circuit]}
E1221 18:26:00.846577       1 tunnel.go:121] [DHT] New stream between peer {12D3KooWEg5jkGp1aN49CiiX8SQeahsWCCsjPNr2WzRPeYecd9a4: [/ip4/192.168.1.101/tcp/20006 /ip4/127.0.0.1/tcp/20006 /ip4/192.168.100.12/tcp/24008]} err: failed to dial 12D3KooWEg5jkGp1aN49CiiX8SQeahsWCCsjPNr2WzRPeYecd9a4:
  * [/ip4/47.101.49.231/tcp/20006/p2p/12D3KooWL5cwYP6rRt2vccqk3VUFo2kzkvYfMzrNCDknkN8J8jxW/p2p-circuit] error opening relay circuit: NO_RESERVATION (204)
  * [/ip4/192.168.1.101/tcp/20006] dial tcp4 0.0.0.0:20006->192.168.1.101:20006: i/o timeout
  * [/ip4/192.168.100.12/tcp/24008] dial tcp4 0.0.0.0:20006->192.168.100.12:24008: i/o timeout
I1221 18:26:40.460462       1 tunnel.go:462] [Finder] find a relay: {12D3KooWL5cwYP6rRt2vccqk3VUFo2kzkvYfMzrNCDknkN8J8jxW: [/ip4/47.101.49.231/tcp/20006 /ip4/10.50.166.41/tcp/20006 /ip4/127.0.0.1/tcp/20006]}

在edge node上也telnet测试了公网ip 20006端口可以访问

Environment:

EdgeMesh version: v1.12.0
Kubernetes version (use kubectl version): v1.21.14
KubeEdge version(e.g. cloudcore --version and edgecore --version): 1.10.3

The text was updated successfully, but these errors were encountered:

Poorunga · 2022-12-24T03:07:47Z

对于edgemesh>=1.12版本

在架构上不再需要部署edgemesh-server了，内网穿透和中继的能力移植到了edgemesh-agent里面。如果某个节点或者某些节点具有公网IP，你可以让部署在这些节点上的edgemesh-agent成为中继。

edgemesh-agent在启动的时候，会多次尝试连接那些被配置成中继节点的relayNodes（我们也称之为bootstrap节点）。对比小于1.12版本的差异是，现在即使连接不上bootstrap节点，edgemesh-agent也不会挂掉，紧接着它会通过多播(组播)的方式发送mDNS协议的数据包，在同一个VLAN网络里发现其他edgemesh-agent并记录peer ID。

这边需要满足几个条件：

a. mDNS协议本身是基于UDP协议的，你需要确保你的网络放通了UDP数据包的传输

b. 由于mDNS是多播(组播)协议，因此要求你的节点在同一个网段里面；节点也必须也得在同一个VLAN里面，不同VLAN之间是隔离广播域的

c. edgemesh-agent的tunnel模块监听在20006，确保安全组/防火墙对20006端口放开

d. 所有节点应该具备内网IP（10.0.0.0/8、172.16.0.0/12、192.168.0.0/16），否则mDNS的数据包会被丢弃，导致不能互相发现

满足上述的条件后，edgemesh就能协助同一个边缘网络里应用的互访了；如果不满足上述条件，你就必须配置relayNodes来走中继的方式去通信。relayNodes设置方式详细材料请阅读：KubeEdge EdgeMesh 高可用架构详解。

renslk · 2022-12-28T02:34:06Z

对于edgemesh>=1.12版本

在架构上不再需要部署edgemesh-server了，内网穿透和中继的能力移植到了edgemesh-agent里面。如果某个节点或者某些节点具有公网IP，你可以让部署在这些节点上的edgemesh-agent成为中继。

edgemesh-agent在启动的时候，会多次尝试连接那些被配置成中继节点的relayNodes（我们也称之为bootstrap节点）。对比小于1.12版本的差异是，现在即使连接不上bootstrap节点，edgemesh-agent也不会挂掉，紧接着它会通过多播(组播)的方式发送mDNS协议的数据包，在同一个VLAN网络里发现其他edgemesh-agent并记录peer ID。

这边需要满足几个条件：

a. mDNS协议本身是基于UDP协议的，你需要确保你的网络放通了UDP数据包的传输

b. 由于mDNS是多播(组播)协议，因此要求你的节点在同一个网段里面；节点也必须也得在同一个VLAN里面，不同VLAN之间是隔离广播域的

c. edgemesh-agent的tunnel模块监听在20006，确保安全组/防火墙对20006端口放开

d. 所有节点应该具备内网IP（10.0.0.0/8、172.16.0.0/12、192.168.0.0/16），否则mDNS的数据包会被丢弃，导致不能互相发现

满足上述的条件后，edgemesh就能协助同一个边缘网络里应用的互访了；如果不满足上述条件，你就必须配置relayNodes来走中继的方式去通信。relayNodes设置方式详细材料请阅读：KubeEdge EdgeMesh 高可用架构详解。

我这里就是使用的中续节点配置了relaynode，其他环境也测试了可行，但是这个环境存在问题；
当前这个环境我已降到1.11.0使用了，1.11.0暂时可行

Poorunga · 2022-12-28T02:36:46Z

edgemesh v1.12升级到go-libp2p circuit-v2，NO_RESERVATION (204) 表示你需要等待一会，让relayNode ready了，这是 go-libp2p circuit-v2的限制，详见：https://github.com/libp2p/specs/blob/master/relay/circuit-v2.md

可在代码里配置：
https://github.com/kubeedge/edgemesh/blob/main/pkg/tunnel/module.go#L143

                libp2p.EnableAutoRelay(
			autorelay.WithPeerSource(func(numPeers int) <-chan peer.AddrInfo {
				return peerSource
			}, 15*time.Second),
			autorelay.WithMinCandidates(0),
			autorelay.WithMaxCandidates(c.MaxCandidates),
			autorelay.WithBackoff(30*time.Second),
		),

renslk · 2022-12-28T02:48:20Z

edgemesh v1.12升级到go-libp2p circuit-v2，NO_RESERVATION (204) 表示你需要等待一会，让relayNode ready了，这是 go-libp2p circuit-v2的限制，详见：https://github.com/libp2p/specs/blob/master/relay/circuit-v2.md

可在代码里配置： https://github.com/kubeedge/edgemesh/blob/main/pkg/tunnel/module.go#L143
                libp2p.EnableAutoRelay(
			autorelay.WithPeerSource(func(numPeers int) <-chan peer.AddrInfo {
				return peerSource
			}, 15*time.Second),
			autorelay.WithMinCandidates(0),
			autorelay.WithMaxCandidates(c.MaxCandidates),
			autorelay.WithBackoff(30*time.Second),
		),

收到，感谢，后续我验证一下

renslk added the kind/bug Categorizes issue or PR as related to a bug. label Dec 21, 2022

Poorunga closed this as completed Jan 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

error opening relay circuit: NO_RESERVATION #411

error opening relay circuit: NO_RESERVATION #411

renslk commented Dec 21, 2022

Poorunga commented Dec 24, 2022

renslk commented Dec 28, 2022

Poorunga commented Dec 28, 2022 •

edited

renslk commented Dec 28, 2022

error opening relay circuit: NO_RESERVATION #411

error opening relay circuit: NO_RESERVATION #411

Comments

renslk commented Dec 21, 2022

Poorunga commented Dec 24, 2022

renslk commented Dec 28, 2022

Poorunga commented Dec 28, 2022 • edited

renslk commented Dec 28, 2022

Poorunga commented Dec 28, 2022 •

edited