Duplicate information of cluster #9

zhangxin511 · 2019-05-22T18:46:54Z

The influxcluster in rwha-sample.influxdb-srelay.cong seems duplicated info with query-router-endpoint-api?

For the SyncFlux, we can run as a HA Cluster, with 1 master 1 slave.

My question is:

Do we still need to define the cluster information again in srelay?
Current implementation of SyncFlux, the HA cluster can have two DB Backend one master one slave. can it have more slaves?
I am not quite understand the current example rwha-sample.influxdb-srelay.conf, say I have two DB Backends running, one on localhost:8086 one on localhost:8087. We can define two [[influxdb]], but the example show we can also define two endpoints in query-router-endpoint-api, which is the cluster endpoint of SyncFlux. My understanding is we can only have one SyncFlux cluster by the existing two db.
Also I am not quite understand how hamonitor of SyncFlux works. I tested to write data when one node is down (either master or slave), but I don't see data was synced.

The text was updated successfully, but these errors were encountered:

toni-moreno · 2019-05-23T06:58:09Z

Hi @zhangxin511 , you should keep in mind the proposed architecture layout.

As you could see , there are:

2 influxdb-srelay instances
2 syncflux instances

Working on both nodes

In this layout with the dbbackend names in influxdb-srelay.conf and the influxdb name in the syncflux.conf shoud be the same.

Supose the rwha-sample. influxdb-srelay.conf

influxdb-srelay.conf ( in myinfluxdb01_server )

...
...
[[influxdb]]
  name = "myinfluxdb01"
  location = "http://myinfluxdb01_server:8086/"
  timeout = "10s"


[[influxdb]]
  name = "myinfluxdb02"
  location = "http://myinfluxdb02_server:8086/"
  timeout = "10s"

[[influxcluster]]
  # name = cluster id for route configs and logs
  name  = "ha_cluster"
  # members = array of influxdb backends
  members = ["myinfluxdb01","myinfluxdb02"]
  log-file = "ha_cluster.log"
  log-level = "info"
  type = "HA"
  query-router-endpoint-api = ["http://myinfluxdb01_server:4090/api/queryactive","http://myinfluxdb02_server:4090/api/queryactive"]
..
...

influxdb-srelay.conf ( in myinfluxdb02_server )

...
...
[[influxdb]]
  name = "myinfluxdb01"
  location = "http://myinfluxdb01_server:8086/"
  timeout = "10s"


[[influxdb]]
  name = "myinfluxdb02"
  location = "http://myinfluxdb02_server:8086/"
  timeout = "10s"

[[influxcluster]]
  # name = cluster id for route configs and logs
  name  = "ha_cluster"
  # members = array of influxdb backends
  members = ["myinfluxdb02","myinfluxdb01"] 
  log-file = "ha_cluster.log"
  log-level = "info"
  type = "HA"
  query-router-endpoint-api = ["http://myinfluxdb02_server:4090/api/queryactive","http://myinfluxdb01_server:4090/api/queryactive"]
...
...

Only changes the members and query-router-endpoint-api order to query first its own syncflux

syncflux.conf (on myinfluxdb01_server )

 master-db = "myinfluxdb01"
 slave-db = "myinfluxdb02"

[[influxdb]]
 release = "1x"
 name = "myinfluxdb01"
 location = "http://myinfluxdb01_server:8086/"
 admin-user = "admin"
 admin-passwd = "admin"
 timeout = "10s"

[[influxdb]]
 release = "1x"
 name = "myinfluxdb02"
 location = "http://myinfluxdb02_server:8086/"
 admin-user = "admin"
 admin-passwd = "admin"
 timeout = "10s"

syncflux.conf (on myinfluxdb02_server )

Only swaps mater and slave values

 master-db = "myinfluxdb02"
 slave-db = "myinfluxdb01"

[[influxdb]]
 release = "1x"
 name = "myinfluxdb01"
 location = "http://myinfluxdb01_server:8086/"
 admin-user = "admin"
 admin-passwd = "admin"
 timeout = "10s"

[[influxdb]]
 release = "1x"
 name = "myinfluxdb02"
 location = "http://myinfluxdb02_server:8086/"
 admin-user = "admin"
 admin-passwd = "admin"
 timeout = "10s"

About your questions:

As in the previous example ( let me know if more doubts on that issue)
Right now , there is no way to have more than one (as this one is also a very young project) , but if needed won't be difficult to add this feature.
3,4) Config as in the layout and example. If you have more questions about how to config and possible errors please open and specific issue in the https://github.com/toni-moreno/syncflux issue manager.

I hope you can understand how smart-relay and syncflux can work together to build a better HA solution when we can not run a InfluxDB Enterprise cluster.

Any other question?

zhangxin511 · 2019-05-23T15:36:28Z

Appreciated for you detailed info. I tried your approach, since I am not sure where the HA load Balance coming from I made only one srelay instance but keeps the others as you suggested, but still can't make data in sync when one node is down.
This is what I have done:

docker-compose up
curl -XPOST http://localhost:8086/query --data-urlencode "q=CREATE DATABASE mydb"
curl -XPOST http://localhost:8087/query --data-urlencode "q=CREATE DATABASE mydb" (haven't try to turn on the admin on influx to use your admin endpoint)
Baseline, curl -i -XPOST "http://127.0.0.1:9096/write?db=mydb" --data-binary "cpu_load_short,host=server01,region=us-west value=0.64 1434055561000000000", both database backend got the data 2015-06-11T20:46:01Z server01 us-west 0.64
stop influx-a(running on 8086), docker-compose stop influx-a
Try insert data while a is down, curl -i -XPOST "http://127.0.0.1:9096/write?db=mydb" --data-binary "cpu_load_short,host=server01,region=us-west value=0.64 1434055562000000000", which is 2015-06-11T20:46:02Z server01 us-west 0.64
Start influx-a again docker-compose start influx-a
Wait, then check database, the 2015-06-11T20:46:02Z server01 us-west 0.64 never synced back to a, while b have the two data entries.

Here is my setup:

docker-compose.yml

version: '3.7'
services:
  influx-a:
    image: influxdb:1.7
    ports:
      - 8086:8086
    volumes:
      - C:/Docker/InfluxHA/Influxdb/a:/var/lib/influxdb
  influx-b:
    image: influxdb:1.7
    ports: 
      - 8087:8086
    volumes:
      - C:/Docker/InfluxHA/Influxdb/b:/var/lib/influxdb
  influx-relay:
    image: tonimoreno/influxdb-srelay:latest
    ports:
      - 9096:9096
    links:
      - influx-a
      - influx-b
      - sync-flux-a
      - sync-flux-b
    volumes:
      - C:/Docker/InfluxHA/Influx-srelay/conf/influxdb-srelay.conf:/etc/influxdb-srelay/influxdb-srelay.conf
      - C:/Docker/InfluxHA/Influx-srelay/log/:/var/log/
  sync-flux-a:
    image: tonimoreno/syncflux
    ports:
      - 4090:4090
    links:
      - influx-a
      - influx-b
    volumes:
      - C:/Docker/InfluxHA/Sync-flux/a/conf/:/opt/syncflux/conf/
      - C:/Docker/InfluxHA/Sync-flux/a/log/:/opt/syncflux/log/
  sync-flux-b:
    image: tonimoreno/syncflux
    ports:
      - 4091:4090
    links:
      - influx-a
      - influx-b
    volumes:
      - C:/Docker/InfluxHA/Sync-flux/b/conf/:/opt/syncflux/conf/
      - C:/Docker/InfluxHA/Sync-flux/b/log/:/opt/syncflux/log/

The configuration file and folder structure are attached. I am sorry to bother you like this, but could you take a look and let me know what went wrong? Much appreciated!
InfluxHA.zip

toni-moreno · 2019-05-24T09:59:18Z

Hi @zhangxin511 I will check your config ASAP

toni-moreno · 2019-05-27T07:06:44Z

Hi @zhangxin511 first thing I've detected is in your syncflux.toml config.

db names should be the same in both engines srelay and syncflux config

influxdb-srelay.conf

  [[influxdb]]
    name = "myinfluxdb01"
    location = "http://influx-a:8086/"
    timeout = "10s"
  
  [[influxdb]]
    name = "myinfluxdb02"
    location = "http://influx-b:8086/"
    timeout = "10s"

syncflux-a.toml

master-db = "myinfluxdb01"
slave-db = "myinfluxdb02"

[[influxdb]]
 release = "1x"
 name = "myinfluxdb01"
 location = "http://influx-a:8086/"
 admin-user = "admin"
 admin-passwd = "admin"
 timeout = "10s"

[[influxdb]]
 release = "1x"
 name = "myinfluxdb02"
 location = "http://influx-b:8086/"
 admin-user = "admin"
 admin-passwd = "admin"
 timeout = "10s"

syncflux-b.toml

master-db = "myinfluxdb02"
slave-db = "myinfluxdb01"

[[influxdb]]
 release = "1x"
 name = "myinfluxdb01"
 location = "http://influx-a:8086/"
 admin-user = "admin"
 admin-passwd = "admin"
 timeout = "10s"

[[influxdb]]
 release = "1x"
 name = "myinfluxdb02"
 location = "http://influx-b:8086/"
 admin-user = "admin"
 admin-passwd = "admin"
 timeout = "10s"

Could you fix these config files and test again please?

zhangxin511 · 2019-05-29T18:30:18Z

@toni-moreno I was not able to make recover works by using your suggestions only. But after change the docker-compose file from using tonimoreno/syncflux to tonimoreno/syncflux:latest data got synced! looks like I was using some old version of syncflux which might not have recover feature(I was able to see partial of the recovery logs but not all)?. Anyway it is finally working for me now. I will do more performance testing and keep you posted.

I understand this is still in the early development phase, a suggestion based on my issue: it looks like srelay and syncflux are tightly related, maybe consider to merge these two repos to one repo, or break the naming retraction?

zhangxin511 · 2019-05-29T18:57:17Z

Sorry, too early, it looks like the data recovery is not ALWAYS working for my case , I actually get only one good replication so far and others all failed.

I do see the syncflux was trying to recover data based on the log. When the recovery not recovering data shows:

time="2019-05-29 18:45:42" level=info msg="HACluster check...."
time="2019-05-29 18:45:42" level=info msg="HACLuster: detected UP Last(2019-05-29 18:45:32.1542501 +0000 UTC m=+540.024756701) Duratio OK (9.9980293s) RECOVERING"
time="2019-05-29 18:45:42" level=info msg="HACLUSTER: INIT RECOVERY : FROM [ 2019-05-29 18:44:32.1545497 +0000 UTC m=+480.024959901 ] TO [ 2019-05-29 18:45:32.1542501 +0000 UTC m=+540.024756701 ]"
time="2019-05-29 18:45:42" level=info msg="Replicating Data from DB mydb RP autogen..."
time="2019-05-29 18:45:42" level=debug msg="SYNC-DB-RP[mydb|&{autogen 0s 168h0m0s %!s(int64=1) %!s(bool=true) map[]}] From:2019-05-29 18:44:32.1545497 +0000 UTC m=+480.024959901 To:2019-05-29 18:45:32.1542501 +0000 UTC m=+540.024756701 | Duration: 59.9997968s || #chunks: 1  | chunk Duration 1h0m0s "
time="2019-05-29 18:45:42" level=info msg="InfluxMonitor: InfluxDB : myinfluxdb01  OK (Version  1.7.6 : Duration 1.454ms )"
time="2019-05-29 18:45:42" level=info msg="Processed Chunk [1/1](100%) from [1559151932][2019-05-29 17:45:32 +0000 UTC] to [1559155532][2019-05-29 18:45:32 +0000 UTC] (0) Points Took [9.7µs]"
time="2019-05-29 18:45:42" level=info msg="Processed DB data from myinfluxdb02[mydb|autogen] to myinfluxdb01[mydb|autogen] has done  #Points (0)  Took [798.4µs] !\n"
time="2019-05-29 18:45:42" level=info msg="HACLUSTER: DATA SYNCRONIZATION Took 1.9629ms"

While only one time there was a good recover, which give this output:

time="2019-05-29 18:23:34" level=info msg="HACluster check...."
time="2019-05-29 18:23:34" level=info msg="HACLuster: detected UP Last(2019-05-29 18:23:24.5806122 +0000 UTC m=+280.038978101) Duratio OK (9.9964807s) RECOVERING"
time="2019-05-29 18:23:34" level=info msg="HACLUSTER: INIT RECOVERY : FROM [ 2019-05-29 18:22:34.5799461 +0000 UTC m=+230.038399201 ] TO [ 2019-05-29 18:23:24.5806122 +0000 UTC m=+280.038978101 ]"
time="2019-05-29 18:23:34" level=info msg="Replicating Data from DB mydb RP autogen..."
time="2019-05-29 18:23:34" level=debug msg="SYNC-DB-RP[mydb|&{autogen 0s 168h0m0s %!s(int64=1) %!s(bool=true) map[cpu_load_short:%!s(*agent.MeasurementSch=&{cpu_load_short map[value:0xc000276600]})]}] From:2019-05-29 18:22:34.5799461 +0000 UTC m=+230.038399201 To:2019-05-29 18:23:24.5806122 +0000 UTC m=+280.038978101 | Duration: 50.0005789s || #chunks: 1  | chunk Duration 1h0m0s "
time="2019-05-29 18:23:34" level=debug msg="processing Database mydb Measurement cpu_load_short from 1559150604 to 1559154204"
time="2019-05-29 18:23:34" level=info msg="InfluxMonitor: InfluxDB : myinfluxdb01  OK (Version  1.7.6 : Duration 1.6828ms )"
time="2019-05-29 18:23:34" level=info msg="InfluxMonitor: InfluxDB : myinfluxdb02  OK (Version  1.7.6 : Duration 2.414ms )"
time="2019-05-29 18:23:34" level=debug msg="Query [select * from  \"cpu_load_short\" where time  > 1559150604s and time < 1559154204s group by *] took 1.5465ms "
time="2019-05-29 18:23:34" level=debug msg="processed 4 points"
time="2019-05-29 18:23:34" level=debug msg="Write attempt [1] took 3.3763ms "
time="2019-05-29 18:23:34" level=info msg="Processed Chunk [1/1](100%) from [1559150604][2019-05-29 17:23:24 +0000 UTC] to [1559154204][2019-05-29 18:23:24 +0000 UTC] (4) Points Took [6.3572ms]"
time="2019-05-29 18:23:34" level=info msg="Processed DB data from myinfluxdb02[mydb|autogen] to myinfluxdb01[mydb|autogen] has done  #Points (4)  Took [6.6578ms] !\n"
time="2019-05-29 18:23:34" level=info msg="HACLUSTER: DATA SYNCRONIZATION Took 8.6714ms"

It looks like this block of code is not always executed, which I have no idea how: https://github.com/toni-moreno/syncflux/blob/6627a8281cd93305f9315b6b6be325f4cdbd0dbb/pkg/agent/client.go#L594-L615

toni-moreno · 2019-05-29T19:09:01Z

Hi @zhangxin511 my workmate @sbengo will review your case ASAP

zhangxin511 · 2019-05-29T19:43:57Z

Thank you @toni-moreno fpr your continuously helps! Let me know if you need anything else @sbengo

zhangxin511 · 2019-05-30T22:29:51Z

@sbengo @toni-moreno I figured out partially why my data was not recovered:

Whenever the log processing Database mydb Measurement cpu_load_short appears, it means it is a "good" recover states for me. But even in the good state, there are two issues for the logic of syncflux getvalues := fmt.Sprintf("select * from \"%v\" where time > %vs and time < %vs group by *", m, startsec, endsec) [here] (https://github.com/toni-moreno/syncflux/blob/6627a8281cd93305f9315b6b6be325f4cdbd0dbb/pkg/agent/client.go#L602):

If I insert data without specify a time, influx will use its current utc time to add to the time tag key. The logic above will find the missing data during the node down and backfill when the node alive again. However, due to the time difference the influx server may have, and when the syncflux detects when the cluster went down, it may add duplicate records. For example, the value 2 got inserted before node went down, and after node recovered, syncflux think it should add this value again and got duplicated
For us, most data we inserted has an previous timestamp due to our delayed/scheduled/batch job. Say a node was down at time 8, and at time 10 we insert an entry happened at time 3, then the broken node recovered at time 11. The entry will not be synced at all due to above logic.

I still don't know why some times the processing Database mydb Measurement cpu_load_short log never appear and put the recovery in a "bad" state.

sbengo · 2019-06-03T06:57:03Z

Hi @zhangxin511 , thanks for the info and sorry for the late response!

When the Syncflux is initiated it gets info about available databases, rps and measurements attached to them (a.k.a schema), and currently it never refresh it (only on init).

As I can see on your logs, on the failing case the schema seems to be empty so it won't iterate over the measurements (on linked function)

Bad case:

...
time="2019-05-29 18:45:42" level=debug msg="SYNC-DB-RP[mydb|&{autogen 0s 168h0m0s %!s(int64=1) %!s(bool=true) map[]}] From:2019-05-29 18:44:32.1545497 +0000 UTC m=+480.024959901 To:2019-05-29 18:45:32.1542501 +0000 UTC m=+540.024756701 | Duration: 59.9997968s || #chunks: 1  | chunk Duration 1h0m0s "
...

Working case:

time="2019-05-29 18:23:34" level=debug msg="SYNC-DB-RP[mydb|&{autogen 0s 168h0m0s %!s(int64=1) %!s(bool=true) map[cpu_load_short:%!s(*agent.MeasurementSch=&{cpu_load_short map[value:0xc000276600]})]}] From:2019-05-29 18:22:34.5799461 +0000 UTC m=+230.038399201 To:2019-05-29 18:23:24.5806122 +0000 UTC m=+280.038978101 | Duration: 50.0005789s || #chunks: 1  | chunk Duration 1h0m0s "

Review

I think its related with schema creation (if there were no data, the schema would be empty: only db, rp was stored). So:

There were data on your DB when you init (up) srelay + syncflux stack?

@toni-moreno opened an issue (I think it was before your comment!) asking for a reload schema toni-moreno/syncflux#16 . We have discussed it and we think we will add this feature on next days, so the schema will be always reloaded before the sync data process.

About timing issues/feature, we will keep discussing , but we currently doesn't support those cases

Thanks,
Regards!

zhangxin511 · 2019-06-03T14:51:43Z

@sbengo Thank you for your detailed response.
Yes I created DB AFTER started the syncflux, therefore syncflux won't know the DB. The change you mentioned makes sense.

It would be great to backfill data based on when the data was inserted instead of based on pure the time tag, because a lot of influxDB data are inserted by scheduled JOBs instead of real-time insertion.

Lastly, I think the syncflux takes time to start, there is a noticeable delay. Hope you can take a look at.

With these being said, I have a full srelay setup and working as your specific, I will close this issue now. Appreciate all your helps @toni-moreno and @sbengo

zhangxin511 closed this as completed Jun 3, 2019

akashgautam24 mentioned this issue Sep 13, 2019

[Bug] queryRouterEndpointAPI not working, does not seem to be populated #22

Closed

mkruliv mentioned this issue Aug 11, 2020

Data don't sync between master and slave toni-moreno/syncflux#40

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Duplicate information of cluster #9

Duplicate information of cluster #9

zhangxin511 commented May 22, 2019 •

edited

toni-moreno commented May 23, 2019

zhangxin511 commented May 23, 2019

toni-moreno commented May 24, 2019

toni-moreno commented May 27, 2019

zhangxin511 commented May 29, 2019

zhangxin511 commented May 29, 2019

toni-moreno commented May 29, 2019

zhangxin511 commented May 29, 2019

zhangxin511 commented May 30, 2019

sbengo commented Jun 3, 2019

zhangxin511 commented Jun 3, 2019

Duplicate information of cluster #9

Duplicate information of cluster #9

Comments

zhangxin511 commented May 22, 2019 • edited

toni-moreno commented May 23, 2019

influxdb-srelay.conf ( in myinfluxdb01_server )

influxdb-srelay.conf ( in myinfluxdb02_server )

syncflux.conf (on myinfluxdb01_server )

syncflux.conf (on myinfluxdb02_server )

zhangxin511 commented May 23, 2019

toni-moreno commented May 24, 2019

toni-moreno commented May 27, 2019

influxdb-srelay.conf

syncflux-a.toml

syncflux-b.toml

zhangxin511 commented May 29, 2019

zhangxin511 commented May 29, 2019

toni-moreno commented May 29, 2019

zhangxin511 commented May 29, 2019

zhangxin511 commented May 30, 2019

sbengo commented Jun 3, 2019

Bad case:

Working case:

Review

zhangxin511 commented Jun 3, 2019

zhangxin511 commented May 22, 2019 •

edited