Should we drop trailing zeros on timestamps? #2977

Closed
beckettsean opened this Issue Jun 12, 2015 · 1 comment

Comments

Projects
None yet
1 participant
@beckettsean
Contributor

beckettsean commented Jun 12, 2015

Let me preface by saying this is less of an actual problem with the way the data is returned and more of a problem of perception. I know I was confused by this behavior, but not necessarily inconvenienced by it. I don't really have a strong opinion how we address it, but I would like to have a well-reasoned answer for those, like me, who find it initially jarring.

When returning the timestamp, we drop any trailing zeros. That obscures the precision with which the points were written. For example:

> INSERT cpu,host=serverB,region=us-west value=20.1
> INSERT cpu,host=serverB,region=us-west value=10.2
> INSERT cpu,host=serverB,region=us-west value=-10.2
> INSERT cpu,host=serverB,region=us-west value=-10.0
> select * from cpu
name: cpu
tags: host=serverB, region=us-west
time                value
----                -----
2015-06-12T22:26:55.443124385Z  20.1
2015-06-12T22:27:00.595680237Z  10.2
2015-06-12T22:27:04.02801Z  -10.2
2015-06-12T22:27:10.676721429Z  -10.0

(I verified that JSON and CSV output formats also drop the trailing zeros.)

Note the third returned point has a timestamp of 2015-06-12T22:27:04.02801Z, which is accurate to the tens of microseconds. I was writing points without a timestamp and without a precision, so I know they were stored at nanosecond precision. That is also demonstrated by the other timestamps, which are precise to the nanosecond.

When querying my data I cannot tell from the return whether I wrote that point at nanosecond precision 2015-06-12T22:27:04.028010000Z or 10 microsecond precision 2015-06-12T22:27:04.02801Z.

In discussion with @gunnaraasen he proposed a possible fix, although it would perhaps break the RFC3339 convention.

By default, return full nanosecond precision, e.g. 2015-06-12T22:27:04.028010000Z.

For those users that store at s, ms, or u precision that will mean a lot of trailing zeros, and meaningless ones at that. Although we store all timestamps to nanosecond precision, if the write accuracy is only in milliseconds the last six zeros have no significance.

To combat that we allow the user to specify the precision of the returned timestamps, and then note that precision by truncating the timestamp with the character. For example:

Precision in nanoseconds:
2015-06-12T22:27:04.028010000Z
Precision in microseconds:
2015-06-12T22:27:04.028010u
Precision in milliseconds:
2015-06-12T22:27:04.0280m
Precision in seconds:
2015-06-12T22:27:04s

Alternately we could provide the precision in the column headers, e.g.

> precision ns
> select * from cpu
precision: ns
name: cpu
tags: host=serverB, region=us-west
time                value
----                -----
2015-06-12T22:26:55.443124385Z  20.1
2015-06-12T22:27:00.595680237Z  10.2
2015-06-12T22:27:04.028010000Z  -10.2
2015-06-12T22:27:10.676721429Z  -10.0
> precision ms
> select * from cpu
precision: ms
name: cpu
tags: host=serverB, region=us-west
time                value
----                -----
2015-06-12T22:26:55.443Z    20.1
2015-06-12T22:27:00.595Z    10.2
2015-06-12T22:27:04.028Z    -10.2
2015-06-12T22:27:10.676Z    -10.0

@beckettsean beckettsean added the RFC label Jun 16, 2015

@beckettsean

This comment has been minimized.

Show comment
Hide comment
@beckettsean

beckettsean Jun 16, 2015

Contributor

All points are stored in nanos but we do not store the write precision separately. In order to prevent lots of trailing zeros for points written in microseconds, milliseconds, or seconds it's good to drop the trailing zeros.

This is something to document, but no development action to take at this time.

Contributor

beckettsean commented Jun 16, 2015

All points are stored in nanos but we do not store the write precision separately. In order to prevent lots of trailing zeros for points written in microseconds, milliseconds, or seconds it's good to drop the trailing zeros.

This is something to document, but no development action to take at this time.

@beckettsean beckettsean referenced this issue in influxdata/docs.influxdata.com May 27, 2016

Closed

check docs for resolved issues #468

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment