Content length 0 for second route.rule backend

[matejzero]

Hello,

I'm trying to setup influxdb-srelay to send metrics to influxdb and victoriametrics storage backends.

Instead of HA, I want to configure 2 single instances and put up some filters to slowly transition to VM, since I had some problems with HA setup.

It turns out, metrics are only sent to the first route.rule defined backend, while the request is sent to the second, but the content-lenght is 0.

My config:

# InfluxDB Backend InfluxDB01
[[influxdb]]
  name = "influxdb"
  location = "http://127.0.0.1:8086/"
  timeout = "15s"

# InfluxDB Backend InfluxDB02
[[influxdb]]
  name = "victoriametrics"
  location = "http://127.0.0.1:10000/"
  timeout = "15s"

# Cluster for linux Metrics
[[influxcluster]]
  name  = "influxdb_cluster"
  members = ["influxdb"]
  log-file = "cluster_influxdb.log"
  log-level = "debug"
  type = "Single"
  rate-limit = 1000
  burst-limit =  1000
  default-ping-response-code = 200

[[influxcluster]]
  name  = "victoriametrics_cluster"
  members = ["victoriametrics"]
  log-file = "cluster_victoriametrics.log"
  log-level = "debug"
  type = "Single"
  rate-limit = 1000
  burst-limit =  1000
  default-ping-response-code = 200

# HTTP Server
[[http]]
  name = "influxdb-ingestor"
  bind-addr = "0.0.0.0:9096"
  log-file = "http_relay_9096.log"
  log-level = "debug"
  access-log = "access.log"
  rate-limit = 1000000
  burst-limit = 2000000

  #
  # IQL /query Endpoint
  #
  [[http.endpoint]]
    uri=["/query"]
    type="RD"
    source_format="IQL"

  #
  #
  # IQL /write Endpoint

  [[http.endpoint]]
    uri=["/write"]
    source_format="ILP"
    type = "WR"

    ## WRITE request - windows
    [[http.endpoint.route]]
      name="any_write"
      level="http"

      [[http.endpoint.route.filter]]
        name="pass_all"
        key="db"
        match=".*"

      ## Send to influxdb instance
      [[http.endpoint.route.rule]]
        name="route_all_influxdb"
        action="route"
        key="db"
        match=".*"
        to_cluster="influxdb_cluster"


      ## Send to victoriametrics cluster
      [[http.endpoint.route.rule]]
        name="route_all_victoriametrics"
        action="route"
        key="db"
        match=".*"
        to_cluster="victoriametrics_cluster"

http relay log:

2019-11-22 07:40:51 DBG Processing [0][RD] endpoint [/query]
2019-11-22 07:40:51 DBG Discarding Input  Endpoint  /write for endpoints [/query]
2019-11-22 07:40:51 DBG Processing [1][WR] endpoint [/write]
2019-11-22 07:40:51 INF Init Processing Endpoint /write
2019-11-22 07:40:51 DBG Processing WRITE route 0 , any_write
2019-11-22 07:40:51 DBG Processing WRITE route 0 , &{cfg:0xc000192980 DecodeFmt:ILP Type:WR log:0xc00016c8a0 filters:[0xc0001d29e0] rules:[0xc0001770c0 0xc000177100] DecodeData:<nil>}
2019-11-22 07:40:51 DBG ----------------------END bodyMiddleWare------------------------
2019-11-22 07:40:51 DBG ----------------------END queryMiddleWare------------------------
2019-11-22 07:40:51 DBG ----------------------END logMiddleWare------------------------
2019-11-22 07:40:51 DBG ----------------------END rateMiddleware-----------------------

In access log, I can see both rules match: 2019-11-22 07:39:51 INF bk_duration_ms=0.480224 duration_ms=6.147961 latency_ms=5.671812 method=POST referer= returnsize=0 source=[2001:1470:8000:82d::173]:48682 status=204 trace-route="http:influxdb-ingestor> rt:any_write> rule:route_all_influxdb> rule:route_all_victoriametrics> " url=/write?consistency=&db=hw&precision=s&rp=autogen user=test user-agent=snmpcollector write-points=-1 write-size=0

In http_route_any_write I can see both rules match:

2019-11-22 07:41:41 INF ROUTE RULE (ACTION: route_all_influxdb): Key db | Match .* | Value  | Matched !!!!
2019-11-22 07:41:41 INF ROUTE RULE (ACTION: route_all_victoriametrics): Key db | Match .* | Value  | Matched !!!!

Metrics are correctly sent to the first configured server:

2019-11-22 07:42:10 DBG Content Length BODYBUF: 13082
2019-11-22 07:42:11 DBG Content Length BODYBUF: 36458
2019-11-22 07:42:20 DBG Content Length BODYBUF: 1871

But the second server only receives empty content lenght:

2019-11-22 07:42:50 DBG Content Length BODYBUF: 0
2019-11-22 07:42:50 DBG Content Length BODYBUF: 0
2019-11-22 07:42:51 DBG Content Length BODYBUF: 0
2019-11-22 07:42:51 DBG Content Length BODYBUF: 0

srelay version: influxdb-srelay version 0.6.0

I hope I read example configs correctly and to my understanding this should work.

[PeterSzegedi]

Hi @toni-moreno, @matejzero,

First of all ,thanks for the great implementation, it is very useful for us @toni-moreno.

This issue is stale for quite a while, but I wanted to leave an update since we have ran into the same issue.

The issue is most likely occurs due to a problem with the handleWriteSingle function.
https://github.com/toni-moreno/influxdb-srelay/blob/master/pkg/cluster/influxcluster.go#L168-L178

The function will be called in a loop from a parent function that loops through on all matching routes.

The issue is that:
https://github.com/toni-moreno/influxdb-srelay/blob/master/pkg/cluster/influxcluster.go#L178 will be called twice.

The getBuf() function is doing the following:
c.bufPool.Get().(*bytes.Buffer)

bufPool is:
bufPool sync.Pool

sync.Pool.Get() will result in this:
https://golang.org/pkg/sync/#Pool.Get

As you can see the issue lies here:
Get selects an arbitrary item from the Pool, removes it from the Pool, and returns it to the caller.

The first call will remove the data and the second call will only get an empty result.

The one big HACluster with multiple boxes is working, because it is using the handleWriteHA function, which takes the buffer the same way, but loops through on all endpoints with the same variable inside the same function.
https://github.com/toni-moreno/influxdb-srelay/blob/master/pkg/cluster/influxcluster.go#L221-L254

Could you take a look if I am right?

Many thanks,
Peter