Skip to content

Commit

Permalink
Influxdb Pusher, let InfluxDB create the timestamp
Browse files Browse the repository at this point in the history
  • Loading branch information
@gizmocuz
gizmocuz committed Feb 16, 2017
1 parent 424b31f commit 00b7111
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion push/InfluxPush.cpp
View file
Expand Up @@ -195,7 +195,7 @@ void CInfluxPush::Do_Work()
if (m_bInfluxDebugActive) {
_log.Log(LOG_NORM, "InfluxLink: value %s", sziData.str().c_str());
}
sziData << " " << (itt->stimestamp * 1000000000);
//sziData << " " << (itt->stimestamp * 1000000000);
if (!sSendData.empty())
sSendData += "\n";
sSendData += sziData.str();
Expand Down

13 comments on commit 00b7111

@oohlaf
Copy link
Contributor

@oohlaf oohlaf commented on 00b7111 Feb 18, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would you let influxdb determine the timestamp? It would deviate from the Domoticz timestamp. Suppose influxdb is down because of an upgrade and you want to determine the missed pushes to import manually via a script that reads sqlite. But if the timestamps are not shared it becomes difficult to pinpoint exactly.

@gizmocuz
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It did not work out for some users...
Worked on mine setup
But if it's down you got bigger issues then a timestamp of a temperature sensor
Happy to accept a working patch

@oohlaf
Copy link
Contributor

@oohlaf oohlaf commented on 00b7111 Feb 18, 2017 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, just read his report and that is weird. He seemed to have issues with some data being in second precision, while other data in nanosecond. That makes no sense.

But you don't actually have to convert the timestamp to nanoseconds, and the InfluxDB team recommend to send in a lower precision if the source is not nanoseconds.

** We recommend using the least precise precision possible as this can result in significant improvements in compression.
influxdb write documentation

This circumvents the issue of multiplying by 1**9, allows InfluxDB too compress better, and you only need to add &precision=s to the influxdb URL (behind ?db=XXX)

@gizmocuz
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well I tested before I committed the code....and without the conversion the timestamps where completely wrong in my influxdb (latest version)
I would have expect I could indeed just use the output from time_t
That's why I needed to convert them

But from his report, the temperature timestamps looked fine
Maybe because we send (could send) multiple values in one go, that the timestamps for the second and up lines are not correctly being received or handled?

Bdw, at the moment when.your influx machine can not be received, the package is also dropped

@grubstake
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am experience invalid timestamps in a new configuration of influxdb data push as well. My Domoticz is v3.6708, InfluxDB 1.2.2. My sensor data has accumulated over several days, all with invalid "1970" dates but slowly accumulating time in the lower digits. Tacking on various precision alternatives from ns to ms to s had no effect other than to "speed up the clock" on the subsequent timestamps, without providing the correct date. Details on http://www.domoticz.com/forum/viewtopic.php?f=28&t=16108&p=127806#p127806

@gizmocuz
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems you use the http push ? and not the influxdb push?

@grubstake
Copy link

@grubstake grubstake commented on 00b7111 Apr 9, 2017 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, my comments above are based on using InfluxDB push, and the post on domoticz forum were based on influxdb push problem.

You probably saw in my forum post where user sincze suggested I try http push using: http://influxmachine:8086/write?db=domoticz&u=root&p=root&precision=ms as an experiment, that post has been corrected.

I found an interesting discussion of this kind of trouble with timestamps in influxDB github issue: influxdata/influxdb#6041

More data: from database created yesterday, configured as posted in forum link above. Data push (influxdb), queries now after ~ 12-18 hours of data:

> use domoinflux
Using database domoinflux
> precision ns
> select "value" from "Temperature" 
name: Temperature
-----------------
time		value
1276416		63.3
1276416		54.1
…
2142177792	57
2142991872	54.1
2143905792	55.9

> precision ms
> select "value" from "Temperature"
name: Temperature
-----------------
time	 value
1	 54.1
1	 63.3
2	 55.8
…
2142	 57
2142	 54.1
2143	 55.9

You can see that timestamp is not correct for either ns or ms precision.

@gizmocuz
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see no straight answer in the Influxdb issue either...
I can remove the timestamp completely as in the first commit... this was, influxdb will generate the timestamp, shall we try this ?

@grubstake
Copy link

@grubstake grubstake commented on 00b7111 Apr 10, 2017 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excuse my unqualified comment, but is this perhaps simply an error in declaring the precision to be the same as the timestamp data when writing?. So much timestamp confusion in many github influxdata issues. An interesting one: influxdata/influxdb#4226.

Edited: more detailed and current issue influxdata/influxdb#8053.

As of now, writing to InfluxDB with the precision flag in the query string is inconsistent. When writing without a timestamp, the precision flag tells Influx to truncate the write time to the given unit of precision. When writing with a timestamp, the precision flag is telling Influx the units of the write.

From my user perspective there is no difference in timestamp provided with domoticz data or by influxdata import time, but perhaps it could be a checkbox option.

Thank you to everyone involved in writing and maintaining this project.

@gizmocuz
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to remove the time stamping in Domoticz... but quess what... it was already removed
So InfluxDb is timestamping.... so this is definitely not a Domoticz issue...
We only send the value

Currently my VPN connection is broke, so i am unable to test

@grubstake
Copy link

@grubstake grubstake commented on 00b7111 Apr 10, 2017 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gizmocuz
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have multiple sensors that can be received at the same time ? This because the influxdb pusher is able to push multiple values
Maybe something goes wrong here, when it actually sends multiple values ?

Have a look here, it is a very small file:
https://github.com/domoticz/domoticz/blob/master/push/InfluxPush.cpp

@grubstake
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have the skill to understand the source code, sorry. I have many years experience working with unix and (old school) databases, but I'm an amateur coder. I have been pushing data from 4+ different sensors with both Influx push first and trying HTTP push later. Both are running now.

InfluxDB push is correctly loading all my sensor data, consistently. But not the timestamp.

My simultaneous HTTP push to InfluxDB server is also loading the same data (into a different InfluxDB database) consistently. But only the correct timestamp (ms significant plus zero padding to ns digits) and value, no luck getting any device name, idx, etc.

I was curious if the difference between the timestamp data in the Domoticz log (correct time) and in the InfluxDB log (incorrect time) would help understand the issue. Of course the big problem is that the timestamps in InfluxDB are starting at the Epoch date beginning 01-Jan-1970. But they continue to increment at a slow rate, so I measured it.

I don't know if it helps, but the the influxDB timestamps continue to increment from 1970/01/01 at real elapsed time*.0035. The epoch NS time stamps are not at all incorrect simply by the decimal place.

It is hard to understand how the trouble could be with the InfluxDB engine as opposed to the push code.

I am asking on the Domoticz forum if any users are getting successful results with Domoticz version V3.6708, like I am using. I wonder if reports of success are really from earlier versions.

Thank you for your work on this project.

Please sign in to comment.