[Bug]: endpoint is always <nil>

[ivdok]

Contact Details

ivdokmc@gmail.com

What happened?

After installation of v0.20.1 Netmaker doesn't recognise endpoint IP, doesn't change it via WebUI, and external users' configs are created with <nil>:51821 as an endpoint.

Version

v0.20.1

What OS are you using?

Linux

Relevant log output

root@wgdashboard:~# curl -LO https://raw.githubusercontent.com/gravitl/netmaker/master/scripts/nm-quick.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 27368  100 27368    0     0   104k      0 --:--:-- --:--:-- --:--:--  103k
root@wgdashboard:~# chmod +x nm-quick.sh 
root@wgdashboard:~# ./nm-quick.sh 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
                                                                                         
 __   __     ______     ______   __    __     ______     __  __     ______     ______    
/\ "-.\ \   /\  ___\   /\__  _\ /\ "-./  \   /\  __ \   /\ \/ /    /\  ___\   /\  == \   
\ \ \-.  \  \ \  __\   \/_/\ \/ \ \ \-./\ \  \ \  __ \  \ \  _"-.  \ \  __\   \ \  __<   
 \ \_\\"\_\  \ \_____\    \ \_\  \ \_\ \ \_\  \ \_\ \_\  \ \_\ \_\  \ \_____\  \ \_\ \_\ 
  \/_/ \/_/   \/_____/     \/_/   \/_/  \/_/   \/_/\/_/   \/_/\/_/   \/_____/   \/_/ /_/ 
                                                                                                                                                                                                 

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
-----------------------------------------------------
Would you like to install Netmaker Community Edition (CE), or Netmaker Enterprise Edition (EE)?
EE will require you to create an account at https://dashboard.license.netmaker.io
-----------------------------------------------------
1) Community Edition
2) Enterprise Edition
#? 1
installing Netmaker CE
-----------Build Options-----------------------------
    EE or CE: ce
  Build Type: version
   Build Tag: v0.20.1
   Image Tag: v0.20.1
   Installer: v0.1.0
-----------------------------------------------------
checking dependencies...
Hit:1 http://us-east-2.ec2.archive.ubuntu.com/ubuntu jammy InRelease
Get:2 http://us-east-2.ec2.archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]                    
Get:3 http://us-east-2.ec2.archive.ubuntu.com/ubuntu jammy-backports InRelease [108 kB]                  
Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]                                
Hit:5 https://download.docker.com/linux/ubuntu jammy InRelease                         
Get:6 http://us-east-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [672 kB]
Get:7 http://us-east-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [182 kB]
Get:8 http://us-east-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [350 kB]
Get:9 http://us-east-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [52.8 kB]
Get:10 http://us-east-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [920 kB]
Get:11 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [732 kB]
Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [129 kB]
Fetched 3375 kB in 1s (2949 kB/s)                                 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
104 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: https://download.docker.com/linux/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
     git is installed
     wireguard is installed
     wireguard-tools is installed
     dnsutils is not installed. Attempting install.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following package was automatically installed and is no longer required:
  squashfs-tools
Use 'apt autoremove' to remove it.
The following NEW packages will be installed:
  dnsutils
0 upgraded, 1 newly installed, 0 to remove and 104 not upgraded.
Need to get 3924 B of archives.
After this operation, 58.4 kB of additional disk space will be used.
Get:1 http://us-east-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 dnsutils all 1:9.18.12-0ubuntu0.22.04.1 [3924 B]
Fetched 3924 B in 0s (293 kB/s)
Selecting previously unselected package dnsutils.
(Reading database ... 100423 files and directories currently installed.)
Preparing to unpack .../dnsutils_1%3a9.18.12-0ubuntu0.22.04.1_all.deb ...
Unpacking dnsutils (1:9.18.12-0ubuntu0.22.04.1) ...
Setting up dnsutils (1:9.18.12-0ubuntu0.22.04.1) ...
Scanning processes...                                                                                                                                                                                                                                                                                                       
Scanning linux images...                                                                                                                                                                                                                                                                                                    

Running kernel seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.
     dnsutils is installed
dpkg-query: no packages found matching jq
     jq is not installed. Attempting install.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following package was automatically installed and is no longer required:
  squashfs-tools
Use 'apt autoremove' to remove it.
The following additional packages will be installed:
  libjq1 libonig5
The following NEW packages will be installed:
  jq libjq1 libonig5
0 upgraded, 3 newly installed, 0 to remove and 104 not upgraded.
Need to get 357 kB of archives.
After this operation, 1087 kB of additional disk space will be used.
Get:1 http://us-east-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB]
Get:2 http://us-east-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB]
Get:3 http://us-east-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB]
Fetched 357 kB in 0s (11.7 MB/s)
Selecting previously unselected package libonig5:amd64.
(Reading database ... 100427 files and directories currently installed.)
Preparing to unpack .../libonig5_6.9.7.1-2build1_amd64.deb ...
Unpacking libonig5:amd64 (6.9.7.1-2build1) ...
Selecting previously unselected package libjq1:amd64.
Preparing to unpack .../libjq1_1.6-2.1ubuntu3_amd64.deb ...
Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ...
Selecting previously unselected package jq.
Preparing to unpack .../jq_1.6-2.1ubuntu3_amd64.deb ...
Unpacking jq (1.6-2.1ubuntu3) ...
Setting up libonig5:amd64 (6.9.7.1-2build1) ...
Setting up libjq1:amd64 (1.6-2.1ubuntu3) ...
Setting up jq (1.6-2.1ubuntu3) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for libc-bin (2.35-0ubuntu3) ...
Scanning processes...                                                                                                                                                                                                                                                                                                       
Scanning linux images...                                                                                                                                                                                                                                                                                                    

Running kernel seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.
     jq is installed
     docker.io is not installed. Attempting install.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 containerd.io : Conflicts: containerd
                 Conflicts: runc
E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages.
   FAILED TO INSTALL docker.io
   This may break functionality.
dpkg-query: no packages found matching docker-compose
     docker-compose is not installed. Attempting install.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  docker-scan-plugin squashfs-tools
Use 'apt autoremove' to remove them.
The following additional packages will be installed:
  containerd.io docker-ce docker-ce-cli python3-docker python3-dockerpty python3-docopt python3-dotenv python3-texttable python3-websocket
Suggested packages:
  aufs-tools cgroupfs-mount | cgroup-lite
Recommended packages:
  docker-buildx-plugin docker-compose-plugin docker.io
The following NEW packages will be installed:
  docker-compose python3-docker python3-dockerpty python3-docopt python3-dotenv python3-texttable python3-websocket
The following packages will be upgraded:
  containerd.io docker-ce docker-ce-cli
3 upgraded, 7 newly installed, 0 to remove and 101 not upgraded.
Need to get 64.7 MB of archives.
After this operation, 108 MB disk space will be freed.
Get:1 http://us-east-2.ec2.archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB]
Get:2 http://us-east-2.ec2.archive.ubuntu.com/ubuntu jammy/universe amd64 python3-docker all 5.0.3-1 [89.3 kB]
Get:3 http://us-east-2.ec2.archive.ubuntu.com/ubuntu jammy/universe amd64 python3-dockerpty all 0.4.1-2 [11.1 kB]
Get:4 http://us-east-2.ec2.archive.ubuntu.com/ubuntu jammy/universe amd64 python3-docopt all 0.6.2-4 [26.9 kB]
Get:5 http://us-east-2.ec2.archive.ubuntu.com/ubuntu jammy/universe amd64 python3-dotenv all 0.19.2-1 [20.5 kB]
Get:6 https://download.docker.com/linux/ubuntu jammy/stable amd64 containerd.io amd64 1.6.21-1 [28.3 MB]
Get:7 http://us-east-2.ec2.archive.ubuntu.com/ubuntu jammy/universe amd64 python3-texttable all 1.6.4-1 [11.4 kB]
Get:8 http://us-east-2.ec2.archive.ubuntu.com/ubuntu jammy/universe amd64 docker-compose all 1.29.2-1 [95.8 kB]
Get:9 https://download.docker.com/linux/ubuntu jammy/stable amd64 docker-ce-cli amd64 5:24.0.2-1~ubuntu.22.04~jammy [13.3 MB]
Get:10 https://download.docker.com/linux/ubuntu jammy/stable amd64 docker-ce amd64 5:24.0.2-1~ubuntu.22.04~jammy [22.9 MB]
Fetched 64.7 MB in 1s (57.7 MB/s)
(Reading database ... 100444 files and directories currently installed.)
Preparing to unpack .../0-containerd.io_1.6.21-1_amd64.deb ...
Unpacking containerd.io (1.6.21-1) over (1.6.8-1) ...
Preparing to unpack .../1-docker-ce-cli_5%3a24.0.2-1~ubuntu.22.04~jammy_amd64.deb ...
Unpacking docker-ce-cli (5:24.0.2-1~ubuntu.22.04~jammy) over (5:20.10.18~3-0~ubuntu-jammy) ...
Preparing to unpack .../2-docker-ce_5%3a24.0.2-1~ubuntu.22.04~jammy_amd64.deb ...
Unpacking docker-ce (5:24.0.2-1~ubuntu.22.04~jammy) over (5:20.10.18~3-0~ubuntu-jammy) ...
Selecting previously unselected package python3-websocket.
Preparing to unpack .../3-python3-websocket_1.2.3-1_all.deb ...
Unpacking python3-websocket (1.2.3-1) ...
Selecting previously unselected package python3-docker.
Preparing to unpack .../4-python3-docker_5.0.3-1_all.deb ...
Unpacking python3-docker (5.0.3-1) ...
Selecting previously unselected package python3-dockerpty.
Preparing to unpack .../5-python3-dockerpty_0.4.1-2_all.deb ...
Unpacking python3-dockerpty (0.4.1-2) ...
Selecting previously unselected package python3-docopt.
Preparing to unpack .../6-python3-docopt_0.6.2-4_all.deb ...
Unpacking python3-docopt (0.6.2-4) ...
Selecting previously unselected package python3-dotenv.
Preparing to unpack .../7-python3-dotenv_0.19.2-1_all.deb ...
Unpacking python3-dotenv (0.19.2-1) ...
Selecting previously unselected package python3-texttable.
Preparing to unpack .../8-python3-texttable_1.6.4-1_all.deb ...
Unpacking python3-texttable (1.6.4-1) ...
Selecting previously unselected package docker-compose.
Preparing to unpack .../9-docker-compose_1.29.2-1_all.deb ...
Unpacking docker-compose (1.29.2-1) ...
Setting up python3-dotenv (0.19.2-1) ...
Setting up python3-texttable (1.6.4-1) ...
Setting up python3-docopt (0.6.2-4) ...
Setting up containerd.io (1.6.21-1) ...
Setting up docker-ce-cli (5:24.0.2-1~ubuntu.22.04~jammy) ...
Setting up python3-websocket (1.2.3-1) ...
Setting up python3-dockerpty (0.4.1-2) ...
Setting up python3-docker (5.0.3-1) ...
Setting up docker-ce (5:24.0.2-1~ubuntu.22.04~jammy) ...
Setting up docker-compose (1.29.2-1) ...
Processing triggers for man-db (2.10.2-1) ...
Scanning processes...                                                                                                                                                                                                                                                                                                       
Scanning linux images...                                                                                                                                                                                                                                                                                                    

Running kernel seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.
     docker-compose is installed
     grep is installed
     gawk is installed
-----------------------------------------------------
dependency check complete
-----------------------------------------------------
-----------------------------------------------------
Would you like to use your own domain for netmaker, or an auto-generated domain?
To use your own domain, add a Wildcard DNS record (e.x: *.netmaker.example.com) pointing to [MY EXTERNAL IP]
IMPORTANT: Due to the high volume of requests, the auto-generated domain has been rate-limited by the certificate provider.
For this reason, we STRONGLY RECOMMEND using your own domain. Using the auto-generated domain may lead to a failed installation due to rate limiting.
-----------------------------------------------------
1) Auto Generated (nm.[MY EXTERNAL IP].nip.io)
2) Custom Domain (e.x: netmaker.example.com)
#? nm.organization.tld
invalid option nm.organization.tld
#? 2
Enter Custom Domain (make sure  *.domain points to [MY EXTERNAL IP] first): nm.organization.tld
using nm.organization.tld
. . .
. . .
-----------------------------------------------------
The following subdomains will be used:
          dashboard.nm.organization.tld
                api.nm.organization.tld
             broker.nm.organization.tld
               stun.nm.organization.tld
               turn.nm.organization.tld
            turnapi.nm.organization.tld
-----------------------------------------------------
before continuing, confirm DNS is configured correctly, with records pointing to [MY EXTERNAL IP]
Does everything look right? [y/n]: y
. . .
Email Address for Domain Registration (click 'enter' to use e7b087507b1254f4@email.com): devops@organization.tld
. . .
Enter Credentials For MQ...
MQ Username (click 'enter' to use 'netmaker'): 
using default username for mq
1) Auto Generated / Config Password
2) Input Your Own Password
#? 1
using random password for mq
Enter Credentials For TURN...
TURN Username (click 'enter' to use 'netmaker'): 
using default username for TURN
1) Auto Generated / Config Password
2) Input Your Own Password
#? 1
using random password for turn
. . .
. . .
-----------------------------------------------------------------
                SETUP ARGUMENTS
-----------------------------------------------------------------
        domain: nm.organization.tld
         email: devops@organization.tld
     public ip: [MY EXTERNAL IP]
-----------------------------------------------------------------
Confirm Settings for Installation
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Does everything look right? [y/n]: y
Stopping all containers...
-----------------------------------------------------------------
Beginning installation...
-----------------------------------------------------------------
. . .
. . .
. . .
Pulling config files...
Saving the config to /root/netmaker.env
Setting up SSL certificates...
Unable to find image 'certbot/certbot:latest' locally
latest: Pulling from certbot/certbot
91d30c5bc195: Pull complete 
5d0bb5b9b8b7: Pull complete 
9369f9a4ffc3: Pull complete 
d7106b6bd561: Pull complete 
8c76a60d2b3a: Pull complete 
e08bf95bbd48: Pull complete 
26174528a65c: Pull complete 
41fea484b2e1: Pull complete 
5dd9ac53b45b: Pull complete 
899e8acb0a4c: Pull complete 
dad4a746029b: Pull complete 
b524cdf16a36: Pull complete 
Digest: sha256:92092d214a4eb75d049720d04f7acc50b40ea226d77736bce6a6bf43981b6e86
Status: Downloaded newer image for certbot/certbot:latest
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/community/x86_64/APKINDEX.tar.gz
v3.16.5-130-gb62986e55df [https://dl-cdn.alpinelinux.org/alpine/v3.16/main]
v3.16.5-127-g643d8ee0752 [https://dl-cdn.alpinelinux.org/alpine/v3.16/community]
OK: 17048 distinct packages available
(1/5) Installing bash (5.1.16-r2)
Executing bash-5.1.16-r2.post-install
(2/5) Installing brotli-libs (1.0.9-r6)
(3/5) Installing nghttp2-libs (1.47.0-r0)
(4/5) Installing libcurl (8.1.2-r0)
(5/5) Installing curl (8.1.2-r0)
Executing busybox-1.35.0-r17.trigger
OK: 29 MiB in 46 packages
certonly --standalone --non-interactive --agree-tos -m devops@organization.tld -d stun.nm.organization.tld -d api.nm.organization.tld -d broker.nm.organization.tld -d dashboard.nm.organization.tld -d turn.nm.organization.tld -d turnapi.nm.organization.tld -d netmaker-exporter.nm.organization.tld -d grafana.nm.organization.tld -d prometheus.nm.organization.tld
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Account registered.
Requesting a certificate for stun.nm.organization.tld and 8 more domains

Certbot failed to authenticate some domains (authenticator: standalone). The Certificate Authority reported these problems:
  Domain: api.nm.organization.tld
  Type:   connection
  Detail: [MY EXTERNAL IP]: Fetching http://api.nm.organization.tld/.well-known/acme-challenge/kHuDcO6jXx4WqAkKajkqylXgAXGKC1AEwHwUE1RA4ig: Timeout during connect (likely firewall problem)

  Domain: broker.nm.organization.tld
  Type:   connection
  Detail: [MY EXTERNAL IP]: Fetching http://broker.nm.organization.tld/.well-known/acme-challenge/C4Y90-blxnp_J1PlGalhUyz0Cd9E6Dhkl5VEH_14dWE: Timeout during connect (likely firewall problem)

  Domain: dashboard.nm.organization.tld
  Type:   connection
  Detail: [MY EXTERNAL IP]: Fetching http://dashboard.nm.organization.tld/.well-known/acme-challenge/D_17FSCBZBsNWleUVOvT0lHMUunWSJsaosHBEbwJlKc: Timeout during connect (likely firewall problem)

  Domain: grafana.nm.organization.tld
  Type:   connection
  Detail: [MY EXTERNAL IP]: Fetching http://grafana.nm.organization.tld/.well-known/acme-challenge/khKhDLvx0mVJFN2SGbRySA5iLQR-PpjxhCo12OR5CqM: Timeout during connect (likely firewall problem)

  Domain: netmaker-exporter.nm.organization.tld
  Type:   connection
  Detail: [MY EXTERNAL IP]: Fetching http://netmaker-exporter.nm.organization.tld/.well-known/acme-challenge/sI5r7c_nIee-uYW1ppLSaMTkRx0k73r60tjBvQyx-28: Timeout during connect (likely firewall problem)

  Domain: prometheus.nm.organization.tld
  Type:   connection
  Detail: [MY EXTERNAL IP]: Fetching http://prometheus.nm.organization.tld/.well-known/acme-challenge/HA6DlJJ8B1yj9qtSXK4eyXYePltVq2uOtvbqof4hCLc: Timeout during connect (likely firewall problem)

  Domain: stun.nm.organization.tld
  Type:   connection
  Detail: [MY EXTERNAL IP]: Fetching http://stun.nm.organization.tld/.well-known/acme-challenge/-lBP2rUh04d0LXw_PpUsFJLs1zblA_PmRRGmq0MqezI: Timeout during connect (likely firewall problem)

  Domain: turn.nm.organization.tld
  Type:   connection
  Detail: [MY EXTERNAL IP]: Fetching http://turn.nm.organization.tld/.well-known/acme-challenge/-5nOxI2u3bTTsYr_RsUSuFcZxeSeZNyVlwKmzJyrXts: Timeout during connect (likely firewall problem)

  Domain: turnapi.nm.organization.tld
  Type:   connection
  Detail: [MY EXTERNAL IP]: Fetching http://turnapi.nm.organization.tld/.well-known/acme-challenge/WmbXINx4na7D9h0tUR5fNPUBWO7BR7UkwHdB-VnXtAs: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to download the challenge files from the temporary standalone webserver started by Certbot on port 80. Ensure that the listed domains point to this machine and that it can accept inbound connections from the internet.

Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for stun.nm.organization.tld and 8 more domains

Certbot failed to authenticate some domains (authenticator: standalone). The Certificate Authority reported these problems:
  Domain: api.nm.organization.tld
  Type:   connection
  Detail: [MY EXTERNAL IP]: Fetching http://api.nm.organization.tld/.well-known/acme-challenge/KYX1ZIfPaXmNJ_WWPuhAI5-g4dONGhEChGhReGktfx0: Timeout during connect (likely firewall problem)

  Domain: broker.nm.organization.tld
  Type:   connection
  Detail: [MY EXTERNAL IP]: Fetching http://broker.nm.organization.tld/.well-known/acme-challenge/Gm7Ej1eFaOQSIlQ6E16Jeee3ULmPV_fwEVxfREcc1Xk: Timeout during connect (likely firewall problem)

  Domain: dashboard.nm.organization.tld
  Type:   connection
  Detail: [MY EXTERNAL IP]: Fetching http://dashboard.nm.organization.tld/.well-known/acme-challenge/G37KTTzuu_d42Ck_5nj9Visf4u31dCnZaWRorvrUgpI: Timeout during connect (likely firewall problem)

  Domain: grafana.nm.organization.tld
  Type:   connection
  Detail: [MY EXTERNAL IP]: Fetching http://grafana.nm.organization.tld/.well-known/acme-challenge/miKji3WNpPpug8skpFbpFKYEoHGDymdR7vzO1svTX9E: Timeout during connect (likely firewall problem)

  Domain: netmaker-exporter.nm.organization.tld
  Type:   connection
  Detail: [MY EXTERNAL IP]: Fetching http://netmaker-exporter.nm.organization.tld/.well-known/acme-challenge/xSWTvgmYvdjIwssdoe4R5YaS0qDIXSVmPiL3k3EONRA: Timeout during connect (likely firewall problem)

  Domain: prometheus.nm.organization.tld
  Type:   connection
  Detail: [MY EXTERNAL IP]: Fetching http://prometheus.nm.organization.tld/.well-known/acme-challenge/IAST2tsk2F2bgD7gilC_fnVoMi4QUv8p8pE5S-wTdJc: Timeout during connect (likely firewall problem)

  Domain: stun.nm.organization.tld
  Type:   connection
  Detail: [MY EXTERNAL IP]: Fetching http://stun.nm.organization.tld/.well-known/acme-challenge/YQldE1Hfg8O78uJnPhxYpXZ_cQduWsOcLZQRO1VE3L4: Timeout during connect (likely firewall problem)

  Domain: turn.nm.organization.tld
  Type:   connection
  Detail: [MY EXTERNAL IP]: Fetching http://turn.nm.organization.tld/.well-known/acme-challenge/6KO2Wt029BYX1n0hlEJgiVNaONKbQaszG5CrEkQuuj0: Timeout during connect (likely firewall problem)

  Domain: turnapi.nm.organization.tld
  Type:   connection
  Detail: [MY EXTERNAL IP]: Fetching http://turnapi.nm.organization.tld/.well-known/acme-challenge/pgKL8dWEuLyLjmni24_m1a_e0VxcitI6TbN8DmHA4lU: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to download the challenge files from the temporary standalone webserver started by Certbot on port 80. Ensure that the listed domains point to this machine and that it can accept inbound connections from the internet.

Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
Missing file: /root/letsencrypt/live/stun.nm.organization.tld/fullchain.pem
SSL certificates failed
root@wgdashboard:~# ./nm-quick.sh 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
                                                                                         
 __   __     ______     ______   __    __     ______     __  __     ______     ______    
/\ "-.\ \   /\  ___\   /\__  _\ /\ "-./  \   /\  __ \   /\ \/ /    /\  ___\   /\  == \   
\ \ \-.  \  \ \  __\   \/_/\ \/ \ \ \-./\ \  \ \  __ \  \ \  _"-.  \ \  __\   \ \  __<   
 \ \_\\"\_\  \ \_____\    \ \_\  \ \_\ \ \_\  \ \_\ \_\  \ \_\ \_\  \ \_____\  \ \_\ \_\ 
  \/_/ \/_/   \/_____/     \/_/   \/_/  \/_/   \/_/\/_/   \/_/\/_/   \/_____/   \/_/ /_/ 
                                                                                                                                                                                                 

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Using config: /root/netmaker.env
-----------Build Options-----------------------------
    EE or CE: ce
  Build Type: version
   Build Tag: v0.20.1
   Image Tag: v0.20.1
   Installer: v0.1.0
-----------------------------------------------------
checking dependencies...
Hit:1 http://us-east-2.ec2.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://us-east-2.ec2.archive.ubuntu.com/ubuntu jammy-updates InRelease             
Hit:3 http://us-east-2.ec2.archive.ubuntu.com/ubuntu jammy-backports InRelease           
Hit:4 https://download.docker.com/linux/ubuntu jammy InRelease                                           
Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease                                         
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
101 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: https://download.docker.com/linux/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
     git is installed
     wireguard is installed
     wireguard-tools is installed
     dnsutils is installed
     jq is installed
     docker.io is not installed. Attempting install.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 containerd.io : Conflicts: containerd
                 Conflicts: runc
E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages.
   FAILED TO INSTALL docker.io
   This may break functionality.
     docker-compose is installed
     grep is installed
     gawk is installed
-----------------------------------------------------
dependency check complete
-----------------------------------------------------
-----------------------------------------------------
Would you like to use your own domain for netmaker, or an auto-generated domain?
To use your own domain, add a Wildcard DNS record (e.x: *.netmaker.example.com) pointing to [MY EXTERNAL IP]
IMPORTANT: Due to the high volume of requests, the auto-generated domain has been rate-limited by the certificate provider.
For this reason, we STRONGLY RECOMMEND using your own domain. Using the auto-generated domain may lead to a failed installation due to rate limiting.
-----------------------------------------------------
1) Auto Generated (nm.[MY EXTERNAL IP].nip.io)
2) Custom Domain (e.x: netmaker.example.com)
#? 2^C
root@wgdashboard:~# docker ps -a
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
root@wgdashboard:~# ./nm-quick.sh 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
                                                                                         
 __   __     ______     ______   __    __     ______     __  __     ______     ______    
/\ "-.\ \   /\  ___\   /\__  _\ /\ "-./  \   /\  __ \   /\ \/ /    /\  ___\   /\  == \   
\ \ \-.  \  \ \  __\   \/_/\ \/ \ \ \-./\ \  \ \  __ \  \ \  _"-.  \ \  __\   \ \  __<   
 \ \_\\"\_\  \ \_____\    \ \_\  \ \_\ \ \_\  \ \_\ \_\  \ \_\ \_\  \ \_____\  \ \_\ \_\ 
  \/_/ \/_/   \/_____/     \/_/   \/_/  \/_/   \/_/\/_/   \/_/\/_/   \/_____/   \/_/ /_/ 
                                                                                                                                                                                                 

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Using config: /root/netmaker.env
-----------Build Options-----------------------------
    EE or CE: ce
  Build Type: version
   Build Tag: v0.20.1
   Image Tag: v0.20.1
   Installer: v0.1.0
-----------------------------------------------------
checking dependencies...
Hit:1 http://us-east-2.ec2.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://us-east-2.ec2.archive.ubuntu.com/ubuntu jammy-updates InRelease                                
Hit:3 http://us-east-2.ec2.archive.ubuntu.com/ubuntu jammy-backports InRelease                              
Hit:4 https://download.docker.com/linux/ubuntu jammy InRelease                                              
Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease                                            
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
101 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: https://download.docker.com/linux/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
     git is installed
     wireguard is installed
     wireguard-tools is installed
     dnsutils is installed
     jq is installed
     docker.io is not installed. Attempting install.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 containerd.io : Conflicts: containerd
                 Conflicts: runc
E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages.
   FAILED TO INSTALL docker.io
   This may break functionality.
     docker-compose is installed
     grep is installed
     gawk is installed
-----------------------------------------------------
dependency check complete
-----------------------------------------------------
-----------------------------------------------------
Would you like to use your own domain for netmaker, or an auto-generated domain?
To use your own domain, add a Wildcard DNS record (e.x: *.netmaker.example.com) pointing to [MY EXTERNAL IP]
IMPORTANT: Due to the high volume of requests, the auto-generated domain has been rate-limited by the certificate provider.
For this reason, we STRONGLY RECOMMEND using your own domain. Using the auto-generated domain may lead to a failed installation due to rate limiting.
-----------------------------------------------------
1) Auto Generated (nm.[MY EXTERNAL IP].nip.io)
2) Custom Domain (e.x: netmaker.example.com)
#? 2
Enter Custom Domain (make sure  *.domain points to [MY EXTERNAL IP] first): nm.organization.tld
using nm.organization.tld
. . .
. . .
-----------------------------------------------------
The following subdomains will be used:
          dashboard.nm.organization.tld
                api.nm.organization.tld
             broker.nm.organization.tld
               stun.nm.organization.tld
               turn.nm.organization.tld
            turnapi.nm.organization.tld
-----------------------------------------------------
before continuing, confirm DNS is configured correctly, with records pointing to [MY EXTERNAL IP]
Does everything look right? [y/n]: y
. . .
Email Address for Domain Registration (click 'enter' to use devops@organization.tld): 
using config email
. . .
Enter Credentials For MQ...
MQ Username (click 'enter' to use 'netmaker'): 
using default username for mq
1) Auto Generated / Config Password
2) Input Your Own Password
#? 1
using random password for mq
Enter Credentials For TURN...
TURN Username (click 'enter' to use 'netmaker'):  
using default username for TURN
1) Auto Generated / Config Password
2) Input Your Own Password
#? 1
using random password for turn
. . .
. . .
-----------------------------------------------------------------
                SETUP ARGUMENTS
-----------------------------------------------------------------
        domain: nm.organization.tld
         email: devops@organization.tld
     public ip: [MY EXTERNAL IP]
-----------------------------------------------------------------
Confirm Settings for Installation
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Does everything look right? [y/n]: y
Stopping all containers...
-----------------------------------------------------------------
Beginning installation...
-----------------------------------------------------------------
. . .
. . .
. . .
Pulling config files...
Saving the config to /root/netmaker.env
Setting up SSL certificates...
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/community/x86_64/APKINDEX.tar.gz
v3.16.5-130-gb62986e55df [https://dl-cdn.alpinelinux.org/alpine/v3.16/main]
v3.16.5-127-g643d8ee0752 [https://dl-cdn.alpinelinux.org/alpine/v3.16/community]
OK: 17048 distinct packages available
(1/5) Installing bash (5.1.16-r2)
Executing bash-5.1.16-r2.post-install
(2/5) Installing brotli-libs (1.0.9-r6)
(3/5) Installing nghttp2-libs (1.47.0-r0)
(4/5) Installing libcurl (8.1.2-r0)
(5/5) Installing curl (8.1.2-r0)
Executing busybox-1.35.0-r17.trigger
OK: 29 MiB in 46 packages
certonly --standalone --non-interactive --agree-tos -m devops@organization.tld -d stun.nm.organization.tld -d api.nm.organization.tld -d broker.nm.organization.tld -d dashboard.nm.organization.tld -d turn.nm.organization.tld -d turnapi.nm.organization.tld -d netmaker-exporter.nm.organization.tld -d grafana.nm.organization.tld -d prometheus.nm.organization.tld
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for stun.nm.organization.tld and 8 more domains

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/stun.nm.organization.tld/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/stun.nm.organization.tld/privkey.pem
This certificate expires on 2023-09-06.
These files will be updated when the certificate renews.

NEXT STEPS:
- The certificate will need to be renewed before it expires. Certbot can automatically renew the certificate in the background, but you may need to take steps to enable that functionality. See https://certbot.org/renewal-setup for instructions.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
SSL certificates ready
Starting containers...
Creating network "root_default" with the default driver
Creating volume "root_caddy_data" with default driver
Creating volume "root_caddy_conf" with default driver
Creating volume "root_sqldata" with default driver
Creating volume "root_dnsconfig" with default driver
Creating volume "root_mosquitto_logs" with default driver
Creating volume "root_mosquitto_data" with default driver
Creating volume "root_turn_server" with default driver
Pulling netmaker (gravitl/netmaker:v0.20.1)...
v0.20.1: Pulling from gravitl/netmaker
8a49fdb3b6a5: Pull complete
4f4fb700ef54: Pull complete
a20c10d68daa: Pull complete
a97714e78c97: Pull complete
8b6b0dde7590: Pull complete
Digest: sha256:d521d03524974ba7dc780b1f8ca9c41fb480bdc219a4e959797bb7ad6b0eb8b9
Status: Downloaded newer image for gravitl/netmaker:v0.20.1
Pulling netmaker-ui (gravitl/netmaker-ui:v0.20.1)...
v0.20.1: Pulling from gravitl/netmaker-ui
f56be85fc22e: Pull complete
97c80f11709c: Pull complete
afb503c1f124: Pull complete
f8c948b732dd: Pull complete
d021bba29710: Pull complete
cadcca1af197: Pull complete
4aacde79cec4: Pull complete
8c65be865286: Pull complete
49f1bbe1741e: Pull complete
94ede08527e8: Pull complete
c26d1e43911c: Pull complete
937bb3d5e64b: Pull complete
Digest: sha256:b68fe30b63a007b21f2dd7cc0e909f15dd006d155115f19908375d791018a063
Status: Downloaded newer image for gravitl/netmaker-ui:v0.20.1
Pulling caddy (caddy:2.6.2)...
2.6.2: Pulling from library/caddy
213ec9aee27d: Already exists
5625668cf98f: Pull complete
675d09b34c53: Pull complete
f1747be70658: Pull complete
db8ee6c4c21d: Pull complete
Digest: sha256:16f4d944907ac8adc93e11df3e4d1b8405d42e4f45e6bc1aae4eb1119552ee79
Status: Downloaded newer image for caddy:2.6.2
Pulling mq (eclipse-mosquitto:2.0.15-openssl)...
2.0.15-openssl: Pulling from library/eclipse-mosquitto
8a49fdb3b6a5: Already exists
1396839f52c6: Pull complete
9b8055bd1a92: Pull complete
Digest: sha256:4a4413784ab9cd1b2008e9555043bf60e86b6c7ae3d2333e507dba011f1297c6
Status: Downloaded newer image for eclipse-mosquitto:2.0.15-openssl
Pulling turn (gravitl/turnserver:v1.0.0)...
v1.0.0: Pulling from gravitl/turnserver
3aa4d0bbde19: Already exists
4f4fb700ef54: Pull complete
9fff0440bf87: Pull complete
Digest: sha256:aa9034872afcf191770e1f2ae7cd29cc6354081dd88b6d01069b71b80b22664e
Status: Downloaded newer image for gravitl/turnserver:v1.0.0
Creating netmaker ... done
Creating caddy       ... done
Creating turn     ... done
Creating coredns     ... done
Creating netmaker-ui ... done
Creating mq          ... done
. . .
. . .
Testing Caddy setup (please be patient, this may take 1-2 minutes)
    Certificates ok
Downloading nmctl...
using server api.nm.organization.tld
using master key fRw4FsRBbv7Auxu6le3H7MqtYAymuI
. . .
. . .
. . .
. . .
. . .
Creating netmaker network (10.101.0.0/16)
2023/06/08 11:44:34 Error Status: 400 Response: {"Code":400,"Message":"could not find any records"}
. . .
. . .
. . .
. . .
. . .
Obtaining a netmaker enrollment key...
Enrollment key ready
. . .
. . .
. . .
[netclient] 2023-06-08 11:44:42 uninstalling netclient... 
[netclient] 2023-06-08 11:44:42 removed systemd remnants if any existed 
[netclient] 2023-06-08 11:44:42 uninstalled netclient 
[netclient] 2023-06-08 11:44:43 setting OS 
[netclient] 2023-06-08 11:44:43 setting version 
[netclient] 2023-06-08 11:44:43 setting netclient hostid 
[netclient] 2023-06-08 11:44:43 setting name 
[netclient] 2023-06-08 11:44:43 setting macAddress 
[netclient] 2023-06-08 11:44:43 setting wireguard keys 
[netclient] 2023-06-08 11:44:43 setting wireguard interface 
[netclient] 2023-06-08 11:44:43 setting listenport 
[netclient] 2023-06-08 11:44:43 setting proxyListenPort 
[netclient] 2023-06-08 11:44:43 setting MTU 
[netclient] 2023-06-08 11:44:43 setting traffic keys 
2023/06/08 11:44:43 calling systemctl stop netclient
Register token: [EXPUNGED]
registered with server nm.organization.tld
waiting for netclient to become available
register complete. New node ID: 55d65c91-1902-4647-bb2d-ac55b6dccae4
making host a default
Host ID: 93200fb7-ccba-4309-a9bd-e2cefb34c445
{
  "id": "93200fb7-ccba-4309-a9bd-e2cefb34c445",
  "verbosity": 0,
  "firewallinuse": "",
  "version": "v0.20.1",
  "name": "[EXPUNGED]",
  "os": "linux",
  "debug": false,
  "isstatic": false,
  "listenport": 51821,
  "locallistenport": 0,
  "proxy_listen_port": 51722,
  "public_listen_port": 0,
  "wg_public_listen_port": 0,
  "mtu": 1420,
  "interfaces": [
    {
      "name": "eth0",
      "address": {
        "IP": "172.31.105.5",
        "Mask": "////AA=="
      },
      "addressString": "172.31.105.5/24"
    },
    {
      "name": "eth0",
      "address": {
        "IP": "fe80::845:feff:fed5:16ee",
        "Mask": "//////////8AAAAAAAAAAA=="
      },
      "addressString": "fe80::845:feff:fed5:16ee/64"
    },
    {
      "name": "veth949890c",
      "address": {
        "IP": "fe80::843e:13ff:fe18:d588",
        "Mask": "//////////8AAAAAAAAAAA=="
      },
      "addressString": "fe80::843e:13ff:fe18:d588/64"
    },
    {
      "name": "veth5bac274",
      "address": {
        "IP": "fe80::988a:afff:fe08:55f",
        "Mask": "//////////8AAAAAAAAAAA=="
      },
      "addressString": "fe80::988a:afff:fe08:55f/64"
    },
    {
      "name": "veth214842e",
      "address": {
        "IP": "fe80::6856:1fff:fe8f:6023",
        "Mask": "//////////8AAAAAAAAAAA=="
      },
      "addressString": "fe80::6856:1fff:fe8f:6023/64"
    },
    {
      "name": "veth37963ae",
      "address": {
        "IP": "fe80::fce8:e2ff:fec6:2f02",
        "Mask": "//////////8AAAAAAAAAAA=="
      },
      "addressString": "fe80::fce8:e2ff:fec6:2f02/64"
    },
    {
      "name": "vethbe84196",
      "address": {
        "IP": "fe80::8c9d:76ff:fe5b:40fe",
        "Mask": "//////////8AAAAAAAAAAA=="
      },
      "addressString": "fe80::8c9d:76ff:fe5b:40fe/64"
    }
  ],
  "defaultinterface": "eth0",
  "endpointip": "\u003cnil\u003e",
  "publickey": "Zvgv/QMcc6J4ZBcLLEG/9infsZYCHhErUPlcHwO4y0Y=",
  "macaddress": "0a:45:fe:d5:16:ee",
  "internetgateway": "",
  "nodes": [
    "55d65c91-1902-4647-bb2d-ac55b6dccae4"
  ],
  "proxy_enabled": false,
  "isdefault": true,
  "isrelayed": false,
  "relayed_by": "",
  "isrelay": false,
  "relay_hosts": null
}
{
  "id": "55d65c91-1902-4647-bb2d-ac55b6dccae4",
  "hostid": "93200fb7-ccba-4309-a9bd-e2cefb34c445",
  "address": "10.101.0.1/16",
  "address6": "",
  "localaddress": "",
  "allowedips": null,
  "persistentkeepalive": 20,
  "lastmodified": 1686224731,
  "expdatetime": 2001894703,
  "lastcheckin": 1686224703,
  "lastpeerupdate": -62135596800,
  "network": "netmaker",
  "networkrange": "10.101.0.0/16",
  "networkrange6": "",
  "isrelayed": false,
  "isrelay": false,
  "isegressgateway": false,
  "isingressgateway": true,
  "egressgatewayranges": null,
  "egressgatewaynatenabled": false,
  "relayaddrs": null,
  "failovernode": "00000000-0000-0000-0000-000000000000",
  "dnson": true,
  "ingressdns": "",
  "server": "nm.organization.tld",
  "internetgateway": "",
  "connected": true,
  "pendingdelete": false,
  "defaultacl": "yes",
  "failover": false
}
-----------------------------------------------------------------
-----------------------------------------------------------------
Netmaker setup is now complete. You are ready to begin using Netmaker.
Visit dashboard.nm.organization.tld to log in
-----------------------------------------------------------------
-----------------------------------------------------------------

Contributing guidelines

• Yes, I did.

[afeiszli]

The machine should initially have no endpoint, but it should be present within ~1 minute. Did you try refreshing to see if it came back up?

[ivdok]

No, it wasn't populated even an hour later, but deleting node and network, and then adding it back in helped.