Build Instructions

[lmangani]

Hello @faceair - Great project here! I run cLoki, similiar initiative in NodeJS for clickhouse.

Could you share or add the intended basic build instructions to generate and test a local binary?

[faceair]

Hi, I updated the readme, you can try it out!

[lmangani]

Thanks! will dockerize and test - is the URL/PATH for the /loki/api/v1/push API the same as the datasource?

[faceair]

I wrote in the documentation to configure the datasource address to be http://127.0.0.1:8481/select/0/, be sure to add /select/0/ in front of it. If you call it manually, the full URL is http://127.0.0.1:8481/select/0/loki/api/v1/query_range.
If you want to call push api to insert data, use http://127.0.0.1:8480/insert/0/loki/api/v1/push, but I haven't tested this api yet, maybe it's broken.

[lmangani]

The service seems to start fine:

....
2020-10-16T20:18:53.605Z        info    /tmp/VictoriaLogs/lib/mergeset/table.go:169     opening table "/root/vmstorage-data/indexdb/163E927168F483C0"...
2020-10-16T20:18:53.844Z        info    /tmp/VictoriaLogs/lib/mergeset/table.go:203     table "/root/vmstorage-data/indexdb/163E927168F483C0" has been opened in 0.240 seconds; partsCount: 0; blocksCount: 0, itemsCount: 0; sizeBytes: 0
2020-10-16T20:18:54.066Z        info    /tmp/VictoriaLogs/app/vmstorage/main.go:66      successfully opened storage "vmstorage-data" in 0.885 seconds; partsCount: 0; blocksCount: 0; rowsCount: 0; sizeBytes: 0
2020-10-16T20:18:54.067Z        info    /tmp/VictoriaLogs/app/vmstorage/transport/server.go:188 accepting vmselect conns at 0.0.0.0:8401
2020-10-16T20:18:54.067Z        info    /tmp/VictoriaLogs/lib/httpserver/httpserver.go:78       starting http server at http://:8482/
2020-10-16T20:18:54.067Z        info    /tmp/VictoriaLogs/app/vmstorage/transport/server.go:121 accepting vminsert conns at 0.0.0.0:8400
2020-10-16T20:18:54.067Z        info    /tmp/VictoriaLogs/lib/httpserver/httpserver.go:79       pprof handlers are exposed at http://:8482/debug/pprof/

The pprof endpoint works, but unfortunately the http server on 8481 is rejecting all connections without any logging event.

# curl http://127.0.0.1:8481/select/0/
curl: (56) Recv failure: Connection reset by peer
# curl http://127.0.0.1:8481/select/0/loki/api/v1/query_range
curl: (56) Recv failure: Connection reset by peer

Suggestions?

[faceair]

Are you running in a container? You may need to use the container's IP to access or expose the port.
Or show me the output of the netstat -nltp command.

[lmangani]

It fails equally from inside/outside the container. 8482 in comparison works fine.

# netstat -nl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 127.0.0.11:34639        0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:8400            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:8401            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:8482            0.0.0.0:*               LISTEN
udp        0      0 127.0.0.11:60502        0.0.0.0:*

Should the logs show any service listening on 8481? I don't see any at all

[faceair]

https://github.com/faceair/VictoriaLogs/blob/master/app/vmselect/main.go#L32
vmselect should be listening on port 8481. Is there any error when it starts?

[lmangani]

Should -storageNode be different between vmselect and vminsert? settings them both to 8401 seems to do the trick

[faceair]

Yes, the ports are different, you can just use the ports mentioned in the readme.

$ bin/vmselect -storageNode 127.0.0.1:8401
$ bin/vminsert -storageNode 127.0.0.1:8400 -importerListenAddr 127.0.0.1:2003

[lmangani]

Up and running now, throwing it some data! Thanks for the assistance, will PR the container and compose with Grafana if any useful.

[lmangani]

Netcat inserts seem to work, the http push api as you mentioned not really. If you can use a tester once you patch it, happy to do so.

[faceair]

Right now I'm just verifying the feasibility of the project. I'm not sure yet how it will be maintained in the future, and I need to discuss it more with the VictoriaMetrics Team. You can keep an eye on it.

[lmangani]

Absolutely, I was mostly curious to compare a few things against our Loki over Clickhouse clone. Your time is much appreciated!

[faceair]

I can fix the push api issue first, which should help you better evaluate performance.

[faceair]

Seems like a simple problem that I've fixed. If there are other issues that affect your testing, open a new issue!