[Bug]: Instructions to install netdata from source, are incomplete and confusing.

[ktsaou]

Bug description

There are quite a few documentation pages on installing netdata from source:

• https://learn.netdata.cloud/docs/agent/packaging/installer/methods/source
• https://learn.netdata.cloud/docs/agent/packaging/installer/methods/manual

Still the steps are quite simple and straight forward.

1. sudo su or su - root
   To become user root. You may use any other method to become user root and run the following commands.

2. Install git to your operating system
   Depending on the operating system, a different command may be required.

3. cd /usr/src
   Enter the directory we are going to put the source code of netdata.

4. git clone https://github.com/netdata/netdata.git netdata.git
   Clone the netdata source from github. This will create the directory netdata.git.

5. cd netdata.git
   Enter the source directory just downloaded.

6. git submodule update --init --recursive
   Download the git submodules required by netdata.
   This command is missing from all documentation pages.

7. ./packaging/installer/install-required-packages.sh netdata
   Install the required packages (compilers, libraries, etc) to build netdata on this system.
   Documentation pages are instructing users to run a network version of it, but it is already in the source tree downloaded.

8. ./netdata-installer.sh
   Compile and install netdata. It will also start it.

To claim this netdata and connect it to netdata cloud, the following are also needed:

1. Go to your netdata cloud, select "Connect Nodes" or "Add Nodes" and find the kickstart command.
2. Find the 2 parameters required:
   • the TOKEN
   • the ROOMS
3. Run the command: netdata-claim.sh -token=TOKEN -rooms=ROOMS -url=https://app.netdata.cloud
   Replace TOKEN and ROOMS with your keys.

Expected behavior

Have precise instructions on how to install netdata from source.

Steps to reproduce

Try to install netdata from source, using any of the instructions provided by our documentation.

Installation method

from source

System info

Tested on FreeBSD

# uname -a; uname -K
FreeBSD freebsd 12.3-RELEASE FreeBSD 12.3-RELEASE r371126 GENERIC  amd64
1203000

Netdata build info

The compiled netdata on FreeBSD:

# netdata -W buildinfo
Version: netdata v1.33.1-177-gd14059673
Configure options:  '--prefix=/usr' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=/usr/libexec' '--libdir=/usr/lib' '--with-zlib' '--with-math' '--with-user=netdata' '--disable-dependency-tracking' '--with-bundled-protobuf' '--with-bundled-libJudy' 'CFLAGS=-O2' 'LDFLAGS='
Install type: custom
Features:
    dbengine:                   YES
    Native HTTPS:               YES
    Netdata Cloud:              YES 
    ACLK Next Generation:       YES
    ACLK-NG New Cloud Protocol: YES
    ACLK Legacy:                NO
    TLS Host Verification:      YES
    Machine Learning:           YES
    Stream Compression:         YES
Libraries:
    protobuf:                YES (bundled)
    jemalloc:                NO
    JSON-C:                  YES
    libcap:                  NO
    libcrypto:               YES
    libm:                    YES
    tcalloc:                 NO
    zlib:                    YES
Plugins:
    apps:                    YES
    cgroup Network Tracking: NO
    CUPS:                    NO
    EBPF:                    NO
    IPMI:                    NO
    NFACCT:                  NO
    perf:                    NO
    slabinfo:                NO
    Xen:                     NO
    Xen VBD Error Tracking:  NO
Exporters:
    AWS Kinesis:             NO
    GCP PubSub:              NO
    MongoDB:                 NO
    Prometheus Remote Write: NO

Additional info

No response

[Ferroin]

@ktsaou Just making sure I understand this: You want us to make it easier for people who do not have experience dealing with building things by hand to build and install Netdata in a way we officially do not provide support for?

To clarify a bit further: The only reason we have these documents at all is to make life easier for people who are packaging Netdata (for example, the people who maintain Netdata in the FreeBSD Ports Tree, which is what you should be using on FreeBSD). IOW, this is technical documentation, not user documentation, so while it is out of date, it does not need to be ‘easy’ for a normal user to follow.

[ktsaou]

@ktsaou Just making sure I understand this: You want us to make it easier for people who do not have experience dealing with building things by hand to build and install Netdata in a way we officially do not provide support for?

I want our documentation to be accurate and work for our users.

To clarify a bit further: The only reason we have these documents at all is to make life easier for people who are packaging Netdata (for example, the people who maintain Netdata in the FreeBSD Ports Tree, which is what you should be using on FreeBSD).

I don't agree this is the only reason. There are many reasons. Top of my mind:

1. Our packaging does not provide packages for them (eg FreeBSD, WRT, exotic Linux distros, etc)
2. Our kickstart.sh fails to compile it (it failed for me [Bug]: Cannot install on FreeBSD with kickstart.sh - Tarball checksum validation failed #12426 )
3. They are contributors and they need to build it on their own to test their contributions before submitting them to us
4. They are packaging netdata for their own needs (install their version on all their systems)

I am sure we can find more use cases.

IOW, this is technical documentation, not user documentation, so while it is out of date, it does not need to be ‘easy’ for a normal user to follow.

All our documentation is technical. Our users are technical by nature and they need technical documents.
But, we shouldn't overcomplicate things.

Both of the pages posted above, say a lot, but they are overcomplicated, time consuming to follow and incomplete.
We need to fix them.

[Ferroin]

@ktsaou Just making sure I understand this: You want us to make it easier for people who do not have experience dealing with building things by hand to build and install Netdata in a way we officially do not provide support for?

I want our documentation to be accurate and work for our users.

To clarify a bit further: The only reason we have these documents at all is to make life easier for people who are packaging Netdata (for example, the people who maintain Netdata in the FreeBSD Ports Tree, which is what you should be using on FreeBSD).

I don't agree this is the only reason. There are many reasons. Top of my mind:

1. Our packaging does not provide packages for them (eg FreeBSD, WRT, exotic Linux distros, etc)
2. Our kickstart.sh fails to compile it (it failed for me [Bug]: Cannot install on FreeBSD with kickstart.sh - Tarball checksum validation failed #12426 )
3. They are contributors and they need to build it on their own to test their contributions before submitting them to us
4. They are packaging netdata for their own needs (install their version on all their systems)

I am sure we can find more use cases.

WE DO NOT OFFICIALLY SUPPORT BUILDING NETDATA BY HAND. We cannot support this sanely because we’ve backed ourselves into a corner with our complicated mess of split repositories and secondary components, and there is no ‘simple’ way to install by hand. Anybody who meets both of the first two categories should be reporting a bug, and probably can’t use the netdata-installer.sh script either because that is (usually) where the bug will actually be.

And anybody who falls into the third or fourth categories can be safely assumed to be knowledgeable enough that we do not need to sugarcoat things for them. Our audience here is not normal users, it’s the type of people who are already comfortable with building things from source themselves.

IOW, this is technical documentation, not user documentation, so while it is out of date, it does not need to be ‘easy’ for a normal user to follow.

All our documentation is technical. Our users are technical by nature and they need technical documents. But, we shouldn't overcomplicate things.

I should probably better clarify what I mean here:

This is documentation for package maintainers and Netdata developers, not end-users. We can and should assume a minimum level of actual competence here.

Both of the pages posted above, say a lot, but they are overcomplicated, time consuming to follow and incomplete. We need to fix them.

Building Netdata is by it’s very nature complex and time consuming. No matter how well we write our docs, the processes of building and installing by hand will be complex and time consuming.

I agree that they are out of date and that does need to be fixed though.

[ktsaou]

I agree that they are out of date and that does need to be fixed though.

Thank you. This is what I said in the first place.

[kickoke]

Okay, I just caught up with the thread. While I can update the docs, I can tell you directly:
I probably can't build Netdata from source: I don't have the skillset of a package maintainer and I won't know if I did it right (assuming I won't run into any errors). If I do get errors along the way, I will probably not know how to resolve them.

@Ferroin I just saw that you opened a PR for this. Thank you very much!

[Ferroin]

@kickoke Actually, I havne’t opened one for this yet, the one you probably saw is adding documentation for how to build the packages using our own package build infrastructure for testing purposes, which is unrelated to this.

I should have a PR up for this later today.

[ktsaou]

• git clone https://github.com/netdata/netdata.git netdata.git
  Clone the netdata source from github. This will create the directory netdata.git.
• cd netdata.git
  Enter the source directory just downloaded.
• git submodule update --init --recursive
  Download the git submodules required by netdata.
  This command is missing from all documentation pages.

The above can be simplified as:

• git clone --recursive https://github.com/netdata/netdata.git netdata.git --depth=1
  Clone the netdata source from github. This will create the directory netdata.git.

• cd netdata.git
  Enter the source directory just downloaded.

I am not sure about the implications of --depth=1 into future updates of the repo (or the tags).

[ktsaou]

Another issue with these instructions, is that the source is compiled as root, which is probably not that good.

[Ferroin]

Another issue with these instructions, is that the source is compiled as root, which is probably not that good.

We can’t avoid that for the netdata-installer.sh script. It blindly assumes that it’s being run as root and we cannot really change that without an extremely invasive overhaul.

[Dim-P]

@DShreve2 This one has gone stale so assigning it to you! If the instructions have been updated recently we can close it, otherwise a PR can be created to reflect the requested changes?