[Bug]: Netdata cloud Installation fails #13498

caroffer12 · 2022-08-08T19:19:50Z

Bug description

Hi Team,

My netdata cloud agent stopped working. I tried to reinstall the agent .

 wget -O /tmp/netdata-kickstart.sh https://my-netdata.io/kickstart.sh && sh /tmp/netdata-kickstart.sh --claim-token xxxxxxxxxxxxxx --claim-url https://app.netdata.cloud
--2022-08-08 19:00:59--  https://my-netdata.io/kickstart.sh
Resolving my-netdata.io (my-netdata.io)... 172.67.156.192, 104.21.13.159, 2606:4700:3031::6815:d9f, ...
Connecting to my-netdata.io (my-netdata.io)|172.67.156.192|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/octet-stream]
Saving to: ‘/tmp/netdata-kickstart.sh’

/tmp/netdata-kickstart.sh                                           [ <=>                                                                                                                                                 ]  75.12K  --.-KB/s    in 0.05s

2022-08-08 19:00:59 (1.55 MB/s) - ‘/tmp/netdata-kickstart.sh’ saved [76921]


 --- Using /tmp/netdata-kickstart-fWHRPD5QI2 as a temporary directory. ---
 --- Checking for existing installations of Netdata... ---
 WARNING  Found an existing netdata install at /, but could not determine the install type.

 WARNING  Usually this means you installed Netdata through your distribution’s regular package repositories or some other unsupported method.

The following non-fatal warnings or errors were encountered:

  - Found an existing netdata install at /, but could not determine the install type.
  - Usually this means you installed Netdata through your distribution’s regular package repositories or some other unsupported method.

 ABORTED  We do not support trying to update or claim installations when we cannot determine the install type. You will need to uninstall the existing install using the same method you used to install it to proceed.

Expected behavior

Netdata needs to run on the server without any error and I should be able to see my server on the cloud url. https://app.netdata.cloud/spaces/xxxxxxxx

Steps to reproduce

wget -O /tmp/netdata-kickstart.sh https://my-netdata.io/kickstart.sh && sh /tmp/netdata-kickstart.sh --claim-token xxxxxxxxxx --claim-url https://app.netdata.cloud

2.Deleted all the existing files

/var/lib/netdata
/var/cache/netdata
/var/log/netdata
/usr/lib/netdata
/usr/libexec/netdata
/usr/share/netdata
/usr/sbin/netdata, /usr/sbin/netdatacli and /usr/sbin/netdata-claim.sh

...

Installation method

kickstart.sh

System info

uname -a; grep -HvE "^#|URL" /etc/*release
Linux NADA-QA1 5.4.0-1029-aws #30-Ubuntu SMP Tue Oct 20 10:06:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
/etc/lsb-release:DISTRIB_ID=Ubuntu
/etc/lsb-release:DISTRIB_RELEASE=20.04
/etc/lsb-release:DISTRIB_CODENAME=focal
/etc/lsb-release:DISTRIB_DESCRIPTION="Ubuntu 20.04.2 LTS"
/etc/os-release:NAME="Ubuntu"
/etc/os-release:VERSION="20.04.2 LTS (Focal Fossa)"
/etc/os-release:ID=ubuntu
/etc/os-release:ID_LIKE=debian
/etc/os-release:PRETTY_NAME="Ubuntu 20.04.2 LTS"
/etc/os-release:VERSION_ID="20.04"
/etc/os-release:VERSION_CODENAME=focal
/etc/os-release:UBUNTU_CODENAME=focal

Netdata build info

$(ps aux | grep -m1 -E -o "[a-zA-Z/]+netdata ") -W buildinfo
-W: command not found

Additional info

No response

The text was updated successfully, but these errors were encountered:

MrZammler · 2022-08-10T07:49:13Z

Hi @caroffer12 !

This is similar to #13430 somehow .. ? Can we check a bit the part My netdata cloud agent stopped working. ? Did something happened that caused it to stop working? Was the installation made through the package manager ?

caroffer12 · 2022-08-10T14:18:23Z

@MrZammler . I am not sure why it stopped working. when i check the curl http://localhost:19999/api/v1/info it show "agent-claimed": false,
"aclk-available": false,
so i deleted all the existing files as per #13430 . Then i run this command wget -O /tmp/netdata-kickstart.sh https://my-netdata.io/kickstart.sh && sh /tmp/netdata-kickstart.sh --claim-token xxxxxxxxxx --claim-url https://app.netdata.cloud i am getting this error

MrZammler · 2022-08-11T08:08:36Z

Hi @caroffer12. I think we need to check why it stops working. Is the only thing affected when this happens the cloud connectivity? If going to the local dashboard, do you see also past metrics missing?

Is there any clue as to what might have caused it? It seems like something deleted files in netdata's installation. Maybe a system update or similar?

Was the old installation made using kickstart or the package manager?

In any case, since you already deleted those files, check the /etc/netdata as well. Most likely it needs to be renamed, it might cause the installer to pick up the previous installation from there.

caroffer12 · 2022-08-11T10:07:00Z

@MrZammler . Not sure what caused it to stop working. previous installation is through kickstart only. i have deleted /etc/netdata how do i proceed further

MrZammler · 2022-08-11T10:07:46Z

Try again the kickstart command please.

caroffer12 · 2022-08-11T10:22:19Z

@MrZammler . Please find the below output when i run kickstart

 wget -O /tmp/netdata-kickstart.sh https://my-netdata.io/kickstart.sh && sh /tmp/netdata-kickstart.sh --claim-token xxxxxx --claim-url https://app.netdata.cloud
--2022-08-11 10:21:21--  https://my-netdata.io/kickstart.sh
Resolving my-netdata.io (my-netdata.io)... 104.21.13.159, 172.67.156.192, 2606:4700:3036::ac43:9cc0, ...
Connecting to my-netdata.io (my-netdata.io)|104.21.13.159|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/octet-stream]
Saving to: ‘/tmp/netdata-kickstart.sh’

/tmp/netdata-kickstart.sh                   [ <=>                                                                         ]  75.12K  --.-KB/s    in 0.002s

2022-08-11 10:21:21 (43.7 MB/s) - ‘/tmp/netdata-kickstart.sh’ saved [76921]


 --- Using /tmp/netdata-kickstart-2wn5EaCZOh as a temporary directory. ---
 --- Checking for existing installations of Netdata... ---
 WARNING  Found an existing netdata install at /, but could not determine the install type.

 WARNING  Usually this means you installed Netdata through your distribution’s regular package repositories or some other unsupported method.

The following non-fatal warnings or errors were encountered:

  - Found an existing netdata install at /, but could not determine the install type.
  - Usually this means you installed Netdata through your distribution’s regular package repositories or some other unsupported method.

 ABORTED  We do not support trying to update or claim installations when we cannot determine the install type. You will need to uninstall the existing install using the same method you used to install it to proceed.

For community support, you can connect with us on:
  - GitHub: https://github.com/netdata/netdata/discussions
  - Discord: https://discord.gg/5ygS846fR6

MrZammler · 2022-08-11T11:52:43Z

Can you try to just install first, and then claim? I.e. first run:

wget -O /tmp/netdata-kickstart.sh https://my-netdata.io/kickstart.sh && sh /tmp/netdata-kickstart.sh

and then re-run it with the token parameters?

caroffer12 · 2022-08-11T12:07:43Z

@MrZammler same error.

 wget -O /tmp/netdata-kickstart.sh https://my-netdata.io/kickstart.sh && sh /tmp/netdata-kickstart.sh
--2022-08-11 12:07:11--  https://my-netdata.io/kickstart.sh
Resolving my-netdata.io (my-netdata.io)... 104.21.13.159, 172.67.156.192, 2606:4700:3031::6815:d9f, ...
Connecting to my-netdata.io (my-netdata.io)|104.21.13.159|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/octet-stream]
Saving to: ‘/tmp/netdata-kickstart.sh’

/tmp/netdata-kickstart.sh                   [ <=>                                                                         ]  75.12K  --.-KB/s    in 0.002s

2022-08-11 12:07:11 (40.5 MB/s) - ‘/tmp/netdata-kickstart.sh’ saved [76921]


 --- Using /tmp/netdata-kickstart-UYOIyxGkzl as a temporary directory. ---
 --- Checking for existing installations of Netdata... ---
 WARNING  Found an existing netdata install at /, but could not determine the install type.

 WARNING  Usually this means you installed Netdata through your distribution’s regular package repositories or some other unsupported method.

The following non-fatal warnings or errors were encountered:

  - Found an existing netdata install at /, but could not determine the install type.
  - Usually this means you installed Netdata through your distribution’s regular package repositories or some other unsupported method.

 ABORTED  We do not support trying to update or claim installations when we cannot determine the install type. You will need to uninstall the existing install using the same method you used to install it to proceed.

For community support, you can connect with us on:
  - GitHub: https://github.com/netdata/netdata/discussions
  - Discord: https://discord.gg/5ygS846fR6
  - Our community forums: https://community.netdata.cloud/

Ferroin · 2022-08-11T12:56:37Z

Most likely it’s seeing a netdata binary left over from a previous install (but not seeing an install type marker, hence can’t determine the install type). Running with --reinstall should get things working again (it should prompt you about whether you really want to reinstall or not).

caroffer12 · 2022-08-11T14:44:05Z

@MrZammler . I tried with reinstall

wget -O /tmp/netdata-kickstart.sh https://my-netdata.io/kickstart.sh && sh /tmp/netdata-kickstart.sh --reinstall
--2022-08-11 14:40:16--  https://my-netdata.io/kickstart.sh
Resolving my-netdata.io (my-netdata.io)... 104.21.13.159, 172.67.156.192, 2606:4700:3031::6815:d9f, ...
Connecting to my-netdata.io (my-netdata.io)|104.21.13.159|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/octet-stream]
Saving to: ‘/tmp/netdata-kickstart.sh’

/tmp/netdata-kickstart.sh                                           [ <=>                                                                                                                                                 ]  75.12K  --.-KB/s    in 0.002s

2022-08-11 14:40:17 (35.8 MB/s) - ‘/tmp/netdata-kickstart.sh’ saved [76921]


 --- Using /tmp/netdata-kickstart-s5b0BuC2Oz as a temporary directory. ---
 --- Checking for existing installations of Netdata... ---
 WARNING  Found an existing netdata install at /, but could not determine the install type.

 WARNING  Usually this means you installed Netdata through your distribution’s regular package repositories or some other unsupported method.

 --- Found an existing netdata install at /, but user requested reinstall, continuing. ---
Reinstalling over top of a unknown installation may be unsafe, do you want to continue? [y/n]
y
 --- OK, continuing. ---
 --- Attempting to install using native packages... ---
 --- Repository configuration is already present, attempting to install netdata. ---
[/tmp/netdata-kickstart-s5b0BuC2Oz]# env apt-get install netdata
Reading package lists... Done
Building dependency tree
Reading state information... Done
netdata is already the newest version (1.35.0-263-nightly).
0 upgraded, 0 newly installed, 0 to remove and 94 not upgraded.
 OK

 WARNING  Could not find netdata-updater.sh. This means that auto-updates cannot (currently) be enabled on this system. See https://learn.netdata.cloud/docs/agent/packaging/installer/update for more information about updating Netdata.

Successfully installed the Netdata Agent.

The following non-fatal warnings or errors were encountered:

  - Found an existing netdata install at /, but could not determine the install type.
  - Usually this means you installed Netdata through your distribution’s regular package repositories or some other unsupported method.
  - Could not find netdata-updater.sh. This means that auto-updates cannot (currently) be enabled on this system. See https://learn.netdata.cloud/docs/agent/packaging/installer/update for more information about updating Netdata.

Official documentation can be found online at https://learn.netdata.cloud/docs/.

Looking to monitor all of your infrastructure with Netdata? Check out Netdata Cloud at https://app.netdata.cloud.

Join our community and connect with us on:
  - GitHub: https://github.com/netdata/netdata/discussions
  - Discord: https://discord.gg/5ygS846fR6
  - Our community forums: https://community.netdata.cloud/

systemctl status netdata
● netdata.service - Real time performance monitoring
     Loaded: loaded (/lib/systemd/system/netdata.service; enabled; vendor preset: enabled)
     Active: failed (Result: timeout) since Mon 2022-08-08 19:00:28 UTC; 2 days ago
   Main PID: 1848769 (code=killed, signal=KILL)

Warning: journal has been rotated since unit was started, output may be incomplete.
root@NADA-QA1:/home/ubuntu# systemctl start netdata
root@NADA-QA1:/home/ubuntu# systemctl status netdata
● netdata.service - Real time performance monitoring
     Loaded: loaded (/lib/systemd/system/netdata.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Thu 2022-08-11 14:42:08 UTC; 1s ago
    Process: 2066895 ExecStart=/usr/sbin/netdata -D $EXTRA_OPTS (code=exited, status=203/EXEC)
   Main PID: 2066895 (code=exited, status=203/EXEC)

Aug 11 14:42:08 NADA-QA1 systemd[1]: netdata.service: Failed with result 'exit-code'.
root@NADA-QA1:/home/ubuntu# curl http://localhost:19999/api/v1/info
curl: (7) Failed to connect to localhost port 19999: Connection refused

I tried to start the netdata service it still fails

caroffer12 · 2022-08-16T10:02:49Z

@MrZammler . Any update on the issue

MrZammler · 2022-08-16T10:32:33Z

Can you please send your /var/log/netdata/error.log to manolis@netdata.cloud ? Thanks!

caroffer12 · 2022-08-16T12:31:37Z

@MrZammler . There is no error.log file

 tail -f /var/log/netdata/error.log
tail: cannot open '/var/log/netdata/error.log' for reading: No such file or directory
tail: no files remaining

root@NADA-QA1:/var/log/netdata# ls -lrt
total 0

caroffer12 · 2022-08-19T10:10:04Z

@MrZammler Any update on this

ilyam8 · 2022-08-19T10:19:44Z

Hi, @caroffer12. Can you show

ps faxu | grep netdata
dpkg -l | grep netdata
cat /etc/passwd | grep netdata

caroffer12 · 2022-08-19T12:27:11Z

@ilyam8

 ps faxu | grep netdata
root     2636557  0.0  0.0   8160   652 pts/0    S+   12:26   0:00                          \_ grep --color=auto netdata
root@NADA-QA1:/home/ubuntu# dpkg -l | grep netdata
ii  netdata                           1.35.0-263-nightly                amd64        real-time charts for system monitoring
ii  netdata-repo-edge                 1-2                               all          Configuration for the official Netdata Edge package repository.
root@NADA-QA1:/home/ubuntu# cat /etc/passwd | grep netdata
netdata:x:996:998:netdata:/var/lib/netdata:/usr/sbin/nologin

MrZammler · 2022-08-19T12:35:25Z

Also please if possilble dpkg -L | grep netdata ?

caroffer12 · 2022-08-19T13:06:42Z

@MrZammler . I am getting error

root@NADA-QA1:/home/ubuntu# dpkg -L | grep netdata
dpkg-query: error: --listfiles needs at least one package name argument

Use --help for help about querying packages.

MrZammler · 2022-08-19T13:31:41Z

Yes, sorry about this. I believe -l shows it's already installed. Can you do a sudo dpkg --remove netdata and then try again to install with kickstart?

ilyam8 · 2022-08-19T13:33:53Z

Can you do a sudo dpkg --remove netdata

Better to do the following

sudo apt-get purge netdata
sudo rm -rf /var/log/netdata/
sudo rm -rf /var/lib/netdata/
sudo rm -rf /var/cache/netdata/

caroffer12 · 2022-08-19T14:20:16Z

@MrZammler @ilyam8 . I did the following

sudo dpkg --remove netdata
sudo apt-get purge netdata
sudo rm -rf /var/log/netdata/
sudo rm -rf /var/lib/netdata/
sudo rm -rf /var/cache/netdata/

Then when i checked dpkg list i am seeing this

 ps faxu | grep netdata
root     2642848  0.0  0.0   8160   716 pts/1    S+   14:17   0:00                          \_ grep --color=auto netdata
root@NADA-QA1:~# dpkg -l | grep netdata
ii  netdata-repo-edge                 1-2                               all          Configuration for the official Netdata Edge package repository.
root@NADA-QA1:~# cat /etc/passwd | grep netdata
netdata:x:996:998:netdata:/var/lib/netdata:/usr/sbin/nologin

Let me know if i can install netdata now.

MrZammler · 2022-08-19T14:35:08Z

Yes, please try now.

caroffer12 · 2022-08-19T15:01:46Z

@MrZammler . Thanks it worked

MrZammler · 2022-08-19T15:26:26Z

Nice to hear! Can you please post the output of netdata -W buildinfo. ?

Is it now claimed to the cloud as well? Do let us know if it somehow loses cloud connection again please.

caroffer12 · 2022-08-19T15:39:36Z

@MrZammler . Please find the output of buildinfo. Sure will let you if something goes wrong.

yes it claimed to Cloud. I am able to the server in cloud console.

netdata -W buildinfo
Version: netdata v1.35.0-263-nightly
Configure options:  '--build=x86_64-linux-gnu' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--libexecdir=${prefix}/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--prefix=/usr' '--sysconfdir=/etc' '--localstatedir=/var' '--libdir=/usr/lib' '--libexecdir=/usr/libexec' '--with-user=netdata' '--with-math' '--with-zlib' '--with-webdir=/var/lib/netdata/www' '--disable-dependency-tracking' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fdebug-prefix-map=/usr/src/netdata=. -fstack-protector-strong -Wformat -Werror=format-security' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fdebug-prefix-map=/usr/src/netdata=. -fstack-protector-strong -Wformat -Werror=format-security'
Install type: binpkg-deb
    Binary architecture: x86_64
    Packaging distro:
Features:
    dbengine:                   YES
    Native HTTPS:               YES
    Netdata Cloud:              YES
    ACLK Next Generation:       YES
    ACLK-NG New Cloud Protocol: YES
    ACLK Legacy:                NO
    TLS Host Verification:      YES
    Machine Learning:           YES
    Stream Compression:         YES
Libraries:
    protobuf:                YES (system)
    jemalloc:                NO
    JSON-C:                  YES
    libcap:                  NO
    libcrypto:               YES
    libm:                    YES
    tcalloc:                 NO
    zlib:                    YES
Plugins:
    apps:                    YES
    cgroup Network Tracking: YES
    CUPS:                    YES
    EBPF:                    YES
    IPMI:                    YES
    NFACCT:                  YES
    perf:                    YES
    slabinfo:                YES
    Xen:                     NO
    Xen VBD Error Tracking:  NO
Exporters:
    AWS Kinesis:             NO
    GCP PubSub:              NO
    MongoDB:                 NO
    Prometheus Remote Write: YES

caroffer12 added bug needs triage Issues which need to be manually labelled labels Aug 8, 2022

ilyam8 added the area/packaging Packaging and operating systems support label Aug 8, 2022

ilyam8 assigned Ferroin Aug 8, 2022

ilyam8 added question and removed bug area/packaging Packaging and operating systems support needs triage Issues which need to be manually labelled labels Aug 19, 2022

ilyam8 closed this as completed Aug 19, 2022

ilyam8 mentioned this issue Aug 19, 2022

[Bug]: Claiming a node fails #13516

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: Netdata cloud Installation fails #13498

[Bug]: Netdata cloud Installation fails #13498

caroffer12 commented Aug 8, 2022 •

edited by ilyam8

Loading

MrZammler commented Aug 10, 2022

caroffer12 commented Aug 10, 2022

MrZammler commented Aug 11, 2022

caroffer12 commented Aug 11, 2022

MrZammler commented Aug 11, 2022

caroffer12 commented Aug 11, 2022 •

edited

Loading

MrZammler commented Aug 11, 2022

caroffer12 commented Aug 11, 2022

Ferroin commented Aug 11, 2022

caroffer12 commented Aug 11, 2022

caroffer12 commented Aug 16, 2022

MrZammler commented Aug 16, 2022 •

edited

Loading

caroffer12 commented Aug 16, 2022

caroffer12 commented Aug 19, 2022

ilyam8 commented Aug 19, 2022

caroffer12 commented Aug 19, 2022

MrZammler commented Aug 19, 2022

caroffer12 commented Aug 19, 2022

MrZammler commented Aug 19, 2022

ilyam8 commented Aug 19, 2022

caroffer12 commented Aug 19, 2022 •

edited

Loading

MrZammler commented Aug 19, 2022

caroffer12 commented Aug 19, 2022

MrZammler commented Aug 19, 2022

caroffer12 commented Aug 19, 2022 •

edited

Loading

[Bug]: Netdata cloud Installation fails #13498

[Bug]: Netdata cloud Installation fails #13498

Comments

caroffer12 commented Aug 8, 2022 • edited by ilyam8 Loading

Bug description

Expected behavior

Steps to reproduce

Installation method

System info

Netdata build info

Additional info

MrZammler commented Aug 10, 2022

caroffer12 commented Aug 10, 2022

MrZammler commented Aug 11, 2022

caroffer12 commented Aug 11, 2022

MrZammler commented Aug 11, 2022

caroffer12 commented Aug 11, 2022 • edited Loading

MrZammler commented Aug 11, 2022

caroffer12 commented Aug 11, 2022

Ferroin commented Aug 11, 2022

caroffer12 commented Aug 11, 2022

caroffer12 commented Aug 16, 2022

MrZammler commented Aug 16, 2022 • edited Loading

caroffer12 commented Aug 16, 2022

caroffer12 commented Aug 19, 2022

ilyam8 commented Aug 19, 2022

caroffer12 commented Aug 19, 2022

MrZammler commented Aug 19, 2022

caroffer12 commented Aug 19, 2022

MrZammler commented Aug 19, 2022

ilyam8 commented Aug 19, 2022

caroffer12 commented Aug 19, 2022 • edited Loading

MrZammler commented Aug 19, 2022

caroffer12 commented Aug 19, 2022

MrZammler commented Aug 19, 2022

caroffer12 commented Aug 19, 2022 • edited Loading

caroffer12 commented Aug 8, 2022 •

edited by ilyam8

Loading

caroffer12 commented Aug 11, 2022 •

edited

Loading

MrZammler commented Aug 16, 2022 •

edited

Loading

caroffer12 commented Aug 19, 2022 •

edited

Loading

caroffer12 commented Aug 19, 2022 •

edited

Loading