Can't configure Netdata to export via prometheus_remote_write to Victoria Metrics

[agrimal]

As I'm having trouble configuring Netdata with Victoria Metrics backend via prometheus_remote_write, I opened an issue on Victoria Metrics github.
Here's the link : VictoriaMetrics/VictoriaMetrics/issues/867

Please tell me if I have to copy/paste the full content of the issue here, or if you need more information.

[vlvkobal]

Could you try #9911? There were changes to the Prometheus remote write connector and the issue might be already solved.

[agrimal]

I can confirm it's working with Netdata v1.24.0 but not since 1.25.0.
I'm only using debian packages for Netdata installation, will this PR be included in a nightly package soon ?

[vlvkobal]

It's almost ready for a review. It might be merged the next week.

[vlvkobal]

Well, then #9775 can be the cause of the issue.

What's weird is that I've tested #9911 with the following configuration

[prometheus_remote_write:instance6]
enabled = yes
destination = localhost:8428
remote write URL path = /api/v1/write
data source = as collected
update every = 2
send charts matching = example.random
buffer on failures = 10

and it works with

docker run -it --rm -v /tmp/victoria-metrics-data:/victoria-metrics-data -p 8428:8428 victoriametrics/victoria-metrics

[ilyam8]

I'm only using debian packages for Netdata installation, will this PR be included in a nightly package soon ?

if you use docker you/we can create a custom image and you test it. What do you think?

[agrimal]

I'm trying to test your PR from a git clone install of Netdata, but I can't get the prometheus remote write backend enabled, because missing libraries :

EDIT : FIXED => was missing some packages

root@testdeb10 [/opt/netdata-git] (test_pr9911) # ./configure 
checking whether to enable maintainer-specific portions of Makefiles... no
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking for gawk... no
checking for mawk... mawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking how to create a pax tar archive... gnutar
checking whether make supports nested variables... (cached) yes
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking whether make supports the include directive... yes (GNU style)
checking dependency style of gcc... gcc3
checking for gcc option to accept ISO C99... none needed
checking for g++... no
checking for c++... no
checking for gpp... no
checking for aCC... no
checking for CC... no
checking for cxx... no
checking for cc++... no
checking for cl.exe... no
checking for FCC... no
checking for KCC... no
checking for RCC... no
checking for xlC_r... no
checking for xlC... no
checking whether we are using the GNU C++ compiler... no
checking whether g++ accepts -g... no
checking dependency style of g++... none
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking minix/config.h usability... no
checking minix/config.h presence... no
checking for minix/config.h... no
checking whether it is safe to define __EXTENSIONS__... yes
checking for __attribute__((returns_nonnull))... yes
checking for __attribute__((malloc))... yes
checking for __attribute__((noreturn))... yes
checking for __attribute__((noinline))... yes
checking for __attribute__((format))... yes
checking for __attribute__((warn_unused_result))... yes
checking for struct timespec... yes
checking for clockid_t... yes
checking for library containing clock_gettime... none required
checking for clock_gettime... yes
checking for sched_setscheduler... yes
checking for sched_getscheduler... yes
checking for sched_getparam... yes
checking for sched_get_priority_min... yes
checking for sched_get_priority_max... yes
checking for getpriority... yes
checking for setpriority... yes
checking for nice... yes
checking for recvmmsg... yes
checking for int8_t... yes
checking for int16_t... yes
checking for int32_t... yes
checking for int64_t... yes
checking for uint8_t... yes
checking for uint16_t... yes
checking for uint32_t... yes
checking for uint64_t... yes
checking for inline... inline
checking whether strerror_r is declared... yes
checking for strerror_r... yes
checking whether strerror_r returns char *... yes
checking for _Generic... yes
checking for __atomic... yes
checking size of void *... 8
checking whether sys/types.h defines makedev... no
checking sys/mkdev.h usability... no
checking sys/mkdev.h presence... no
checking for sys/mkdev.h... no
checking sys/sysmacros.h usability... yes
checking sys/sysmacros.h presence... yes
checking for sys/sysmacros.h... yes
checking for sys/types.h... (cached) yes
checking for netinet/in.h... yes
checking for arpa/nameser.h... yes
checking for netdb.h... yes
checking for resolv.h... yes
checking for sys/prctl.h... yes
checking for sys/vfs.h... yes
checking for sys/statfs.h... yes
checking for sys/statvfs.h... yes
checking for sys/mount.h... yes
checking for accept4... yes
checking operating system... linux with id 1
checking if compiler needs -Werror to reject unknown flags... no
checking for the pthreads library -lpthreads... no
checking whether pthreads work without any flags... no
checking whether pthreads work with -Kthread... no
checking whether pthreads work with -kthread... no
checking for the pthreads library -llthread... no
checking whether pthreads work with -pthread... yes
checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE
checking if more special flags are required for pthreads... no
checking for PTHREAD_PRIO_INHERIT... yes
checking for pthread_getname_np in -lpthread... yes
checking for sin in -lm... yes
checking if libm should be used... yes
checking for uv_fs_scandir_next in -luv... yes
checking for LZ4_compress_default in -llz4... yes
checking for ZLIB... yes
checking if zlib should be used... yes
checking for UUID... yes
checking for SHA256_Init in -lcrypto... yes
checking for X509_VERIFY_PARAM_set1_host in -lcrypto... yes
checking for JSON... no
checking for json_object_get_type in -ljson... no
checking for JudyLIns in -lJudy... yes
checking if netdata dbengine should be used... yes
checking if netdata https should be used... yes
checking if statically built json-c is present... yes
checking if json-c should be used... yes
checking for memory allocator... system
checking for mallopt... yes
checking for mallinfo... yes
checking for LIBCAP... no
checking if libcap should be used... no
checking if cloud functionality should be enabled... detect
checking if libmosquitto static lib is present (and builds)... yes
checking if libwebsockets static lib is present... yes
checking if json-c available for ACLK... yes
checking if netdata agent-cloud-link can be enabled... yes
checking if netdata agent-cloud-link should/will be enabled... yes
checking if apps.plugin should be enabled... yes
checking for IPMIMONITORING... no
checking if freeipmi.plugin should be enabled... no
checking for httpConnect2 in -lcups... no
checking for cups-config... no
checking if cups.plugin should be enabled... no
checking linux/netfilter/nfnetlink_conntrack.h usability... yes
checking linux/netfilter/nfnetlink_conntrack.h presence... yes
checking for linux/netfilter/nfnetlink_conntrack.h... yes
checking whether CTA_STATS_MAX is declared... yes
checking for NFACCT... no
checking for LIBMNL... yes
checking for mnl_socket_open in -lmnl... yes
checking if nfacct.plugin should be enabled... no
checking for YAJL... no
checking for xenstat_init in -lxenstat... no
checking for XENLIGHT... no
checking if xenstat.plugin should be enabled... no
checking linux/perf_event.h usability... yes
checking linux/perf_event.h presence... yes
checking for linux/perf_event.h... yes
checking whether PERF_COUNT_HW_REF_CPU_CYCLES is declared... yes
checking if perf.plugin should be enabled... yes
checking for LIBELF... yes
checking for struct bpf_prog_info... yes
checking for externaldeps/libbpf/libbpf.a... yes
checking if ebpf.plugin should be enabled... yes
checking if slabinfo.plugin should be enabled... yes
checking for LIBCRYPTO... yes
checking for CRYPTO_new_ex_data in -lcrypto... yes
checking for LIBSSL... yes
checking for SSL_connect in -lssl... yes
checking for LIBCURL... no
checking for cJSON_free in -laws-cpp-sdk-core... no
checking for Aws::Kinesis::Model::PutRecordRequest in -laws-cpp-sdk-kinesis... no
checking if kinesis backend should be enabled... no
checking for GRPC... no
checking for PUBSUB... no
checking for g++... no
checking if pubsub exporting connector should be enabled... detect
checking for PROTOBUF... no
checking for snappy::RawCompress in -lsnappy... no
checking for protoc... no
checking for g++... (cached) no
checking if prometheus remote write backend should be enabled... no
checking for LIBMONGOC... no
checking if mongodb backend should be enabled... no
checking for setns... yes
checking if cgroup-network can be enabled... yes
checking whether C compiler accepts -flto... yes
checking if -flto builds executables... yes
checking if LTO should be enabled... yes
checking for CMOCKA... no
configure: CMocka not found on the system. Unit tests disabled
checking that generated files are newer than configure... done
configure: creating ./config.status
[truncated]

[agrimal]

So I think this is still not working with your PR, althought I'm not 100% sure i did things right.

Steps :

cd /opt
git clone https://github.com/netdata/netdata.git
mv netdata netdata-git
cd netdata-git
git fetch origin pull/9911/head:test_pr9911
git checkout test_pr9911

And then :

./netdata-installer.sh --disable-telemetry

After setting configuration for exporting and restarting i still get these logs :

root@testdeb10 [~] # grep victoria /var/log/netdata/*
/var/log/netdata/error.log:2020-10-29 13:58:42: netdata INFO  : EXPORTING : Processing connector instance (victoria_metrics)
/var/log/netdata/error.log:2020-10-29 13:58:42: netdata INFO  : EXPORTING : Instance (victoria_metrics) on connector (prometheus_remote_write) is enabled and scheduled for activation
/var/log/netdata/error.log:2020-10-29 13:58:42: netdata INFO  : EXPORTING : Instance victoria_metrics on prometheus_remote_write
/var/log/netdata/error.log:2020-10-29 13:58:44: netdata INFO  : EXPORTING : enabled exporting of host 'localhost' for instance 'victoria_metrics'
/var/log/netdata/error.log:2020-10-29 13:58:54: netdata ERROR : EXPORTING : EXPORTING: cannot format metric for victoria_metrics
/var/log/netdata/error.log:2020-10-29 13:58:54: netdata ERROR : EXPORTING : EXPORTING: Instance victoria_metrics disabled
/var/log/netdata/error.log:2020-10-29 13:59:15: netdata ERROR : MAIN : EXPORTING: failed to update 'victoria-metrics1:8480 victoria-metrics2:8480 victoria-metrics3:8480'
/var/log/netdata/error.log:2020-10-29 13:59:45: netdata ERROR : MAIN : EXPORTING: failed to update 'victoria-metrics1:8480 victoria-metrics2:8480 victoria-metrics3:8480'

Conf :

root@testdeb10 [~] # cat /etc/netdata/exporting.conf 

[prometheus_remote_write:victoria_metrics]
enabled = yes
destination = victoria-metrics1:8480 victoria-metrics2:8480 victoria-metrics3:8480
remote write URL path = /insert/1/prometheus/api/v1/write
data source = as collected
prefix = netdata
update every = 5
timeout ms = 10000
send charts matching = *
send hosts matching = *
send names instead of ids = yes

[ilyam8]

@agrimal what is your VictoriaMetrics version?

[agrimal]

root@victoria-metrics1 [~] # vminsert -version
vminsert-20200923-004417-tags-v1.41.1-cluster-0-g90d254942

[vlvkobal]

Quite a strange endpoint remote write URL path = /insert/1/prometheus/api/v1/write 🤔.

[agrimal]

Look Cluster-VictoriaMetrics, section URL Format
Also it doesn't work with /insert/1/prometheus endpoint.
But all of this work with Netdata v1.24.0.

[vlvkobal]

Ah, I see, both suffixes are allowed. Then we have a problem with reproduction ☹.

[vlvkobal]

The only change I see between 1.24.0 and 1.25.0 is

diff --git a/exporting/prometheus/remote_write/remote_write.c b/exporting/prometheus/remote_write/remote_write.c
index 28ddc7e1..61061461 100644
--- a/exporting/prometheus/remote_write/remote_write.c
+++ b/exporting/prometheus/remote_write/remote_write.c
@@ -39,7 +39,7 @@ int prometheus_remote_write_send_header(int *sock, struct instance *instance)
         "Content-Length: %zu\r\n"
         "Content-Type: application/x-www-form-urlencoded\r\n\r\n",
         connector_specific_config->remote_write_path,
-        instance->engine->config.hostname,
+        instance->config.destination,
         buffer_strlen((BUFFER *)instance->buffer));

[agrimal]

Hmm okay, what else could I do to help you ?

[vlvkobal]

Please check if exporting is working with data source = average.