Unable to write influx data when using cluster (installed with helm) insert endpoint

[ylebon]

Describe the bug
I am not able to insert influx points when using the cluster deployed with helm

To Reproduce
influxdb client endpoint:
http://victoria-metrics-cluster-vminsert.victoria-metrics-cluster.svc.cluster.local:8480/insert/0/influx/write

vminsert logs:
error VictoriaMetrics@/lib/httpserver/httpserver.go:368 cannot parse path "/write": cannot find {prefix}
2019-12-12T18:54:04.039+0000 error VictoriaMetrics@/lib/httpserver/httpserver.go:368 cannot parse path "/write": cannot find {prefix}

Expected behavior
A clear and concise description of what you expected to happen.
I have no issues when running it from a standalone mode using the following endpoint and the same client:
http://victoria:8428/feeds

Screenshots
If applicable, add screenshots to help explain your problem.

Version
The line returned when passing --version command line flag to binary. For example:

$ ./victoria-metrics-prod --version
victoria-metrics-20190730-121249-heads-single-node-0-g671d9e55

Additional context
Add any other context about the problem here such as error logs, /metrics output, screenshots from the official Grafana dashboard for VictoriaMetrics.

[valyala]

Which Influx client do you use? It looks like it doesn't support enpoints with paths after the domain and port like http://victoria-metrics-cluster-vminsert.victoria-metrics-cluster.svc.cluster.local:8480/insert/0/influx/write. It substitutes /insert/0/influx/write with /write in the url according to the error message.
You can verify that the endpoint works by issuing the following command in terminal:

curl -d 'test_measurement value=123' -X POST http://victoria-metrics-cluster-vminsert.victoria-metrics-cluster.svc.cluster.local:8480/insert/0/influx/write

This command sends test_measurement_value metric with 123 value to VictoriaMetrics. Later you can query it via vmselect according to the docs:

curl -G 'http://victoria-metrics-cluster-vmselect.victoria-metrics-cluster.svc.cluster.local:8481/select/0/prometheus/api/v1/export' -d 'match={__name__=~"test_measurement.*"}'

[ylebon]

It got a different error with the command
curl -d 'test_measurement value=123' -X POST http://victoria-metrics-cluster-vminsert.victoria-metrics-cluster.svc.cluster.local:8480/insert/0/influx/write
Error:
2019-12-12T22:28:39.133+0000 error VictoriaMetrics@/app/vminsert/netstorage/netstorage.go:100 cannot send data to vmstorage victoria-metrics-cluster-vmstorage-0.victoria-metrics-cluster-vmstorage.victoria-metrics-cluster.svc.cluster.local:8400: cannot dial "victoria-metrics-cluster-vmstorage-0.victoria-metrics-cluster-vmstorage.victoria-metrics-cluster.svc.cluster.local:8400": dial tcp4 172.17.0.5:8400: i/o timeout; re-routing data to healthy vmstorage nodes

[valyala]

This error means that vminsert cannot establish connection to vmstorage at victoria-metrics-cluster-vmstorage-0.victoria-metrics-cluster-vmstorage.victoria-metrics-cluster.svc.cluster.local:8400. Make sure the vmstorage is available at this address. It looks like the address contains superfluous victoria-metrics-cluster-vmstorage string. Probably it should look like the following, i.e. similar to the address for vminsert:

victoria-metrics-cluster-vmstorage-0.victoria-metrics-cluster.svc.cluster.local:8400

cc'ing @tenmozes , since this could be related to incorrect Helm chart configuration.

[tenmozes]

On fresh environment which is deployed via helm

helm install -n victoria-metrics-cluster vm/victoria-metrics-cluster

➜  scripts git:(master) ✗ helm list
NAME                    	REVISION	UPDATED                 	STATUS  	CHART                         	APP VERSION	NAMESPACE
victoria-metrics-cluster	1       	Sat Dec 14 10:20:38 2019	DEPLOYED	victoria-metrics-cluster-0.3.0	1.30.5     	default

i was able to write data to using influx protocol

/ # curl -d 'test_measurement value=123' -X POST http://victoria-metrics-cluster-vminsert.default.svc.cluster.local:8480/insert/0/influx/write
/ # curl -d 'test_measurement value=123' -X POST http://victoria-metrics-cluster-vminsert.default.svc.cluster.local:8480/insert/0/influx/write

[tenmozes]

@madazone the second message means that you have no healty storage nodes.
could you please make sure that all storage nodes up and running
run helm status <name> or kubectl get po

[ylebon]

The issue is now solved, it was related to a connection issue in my cluster nodes. I have few more questions:

• Is it possible expose (public) the select endpoint in the helm?
• Can I setup authentication in this helm chart?
• In a cluster mode, as I have many vmstorage pods, how do I know on which pod I should run the vmbackup?

[valyala]

Is it possible expose (public) the select endpoint in the helm?

I don't know . cc'ing @tenmozes , who could have more information about this.

Can I setup authentication in this helm chart?

As I know the official helm chart for VictoriaMetrics cluster doesn't support authentication. It must be performed by a separate authentication proxy sitting in front of vminsert and vmselect services. Such a proxy isn't included in VictoriaMetrics, since it is hard to create generic authentication solution, which would fit the majority of our users. We are happy to consult regarding such a proxy - just drop us email at info@victoriametrics.com with your requirements.

In a cluster mode, as I have many vmstorage pods, how do I know on which pod I should run the vmbackup?

Data between vmstorage pods isn't shared, so vmbackup must run on each vmstorage pod. Data from each vmstorage pod must be backed up into distinct backup folder such as s3://victoriametrics-backups/vmstorage-XX/...

[tenmozes]

There are two supported options for vmselect and insert to expose endpoints

• on service level (change type of service to LoadBalancer) and get public ip of it
• on ingress level

You may support of basic auth on ingress level

• https://kubernetes.github.io/ingress-nginx/examples/auth/basic/ here is an example

[ylebon]

Thanks for the answers

I thought victoriametrics has milliseconds precision, but the following request returns second precision

http://server/api/v1/query_range?query=metric_1&start=2019-12-22T00:00:00.000Z&end=2019-12-22T00:00:00.999Z

{'status': 'success', 'data': {'resultType': 'matrix', 'result': [{'metric': {'name': 'metric_1', 'db': 'write'}, 'values': [[1576972800, '479']]}]}}

[valyala]

VictoriaMetrics stores timestamps with millisecond precision. These timestamps could be obtained via /api/v1/export API.

As for /api/v1/query_range and /api/v1/query APIs, they return calculated timestamps at regular interval set via step query arg as Prometheus does. See Prometheus querying API docs and the following section from Prometheus data model docs for details:

When queries are run, timestamps at which to sample data are selected independently of the actual present time series data. This is mainly to support cases like aggregation (sum, avg, and so on), where multiple aggregated time series do not exactly align in time. Because of their independence, Prometheus needs to assign a value at those timestamps for each relevant time series. It does so by simply taking the newest sample before this timestamp.

[valyala]

Closing the issue as resolved. @madazone , feel free re-opening it if you have additional questions regarding this issue.