Added Downsample normalize on week month and year #548

Closed
wants to merge 2 commits into
from

Projects

None yet

4 participants

@moarcaccio

downsample normalize on week, month and year on non-UTC times using Java Calendar.
It needs additional refinements, but it works, the resulting timestamp will be rounded off at the beginning of the week (7days), month (31days) or year (365days).

moarcaccio added some commits Aug 17, 2015
@moarcaccio moarcaccio Added Downsample normalize on week month and year
downsample normalize on week, month and year on non-UTC times using Java Calendar.
It needs additional refinements, but it works, the resulting timestamp will be rounded off at the beginning of the week (7days), month (31days) or year (365days).
7c80ce4
@moarcaccio moarcaccio Fix missing import java.util.Calendar; 29775fb
@manolama
Member

Cool, thanks for taking a stab at this! Could you do a couple of things:

  • Point it at the Next branch please and work off that?
  • Add a configuration flag like "tsd.query.downsample.use_calendar" that would enable this feature or use the current default for backwards compatibility? Some folks may depend on the current behavior.
  • Add a per-query flag that would override the above
  • Add daily support (some timezones are off by 30 minutes, etc [including North Korea now])
  • Sign the contributors agreement
@manolama manolama commented on the diff Aug 27, 2015
src/core/Downsampler.java
+ // 1 year (365days), align the timestamp at the first day of year.
+ if (interval_ms == 31536000000L)
+ return toFirstDayOfYear(timestamp);
+ // 1 month (31days), align the timestamp at the first day of month.
+ else if (interval_ms == 2678400000L)
+ return toFirstDayOfMonth(timestamp);
+ // 1 week (7days), align the timestamp at the first day of week.
+ else if (interval_ms == 604800000L)
+ return toFirstDayOfWeek(timestamp);
+ // default timestamp normalization (tsdb v2.1.0)
+ else
+ return timestamp - (timestamp % interval_ms);
+ }
+
+ /** Returns timestamp aligned at the first day of month. */
+ private long toFirstDayOfMonth(long timestamp) {
@manolama
manolama Aug 27, 2015 Member

Move these to the DateTime class as we may want to share them

@manolama manolama commented on the diff Aug 27, 2015
src/core/Downsampler.java
+ if (interval_ms == 31536000000L)
+ return toFirstDayOfYear(timestamp);
+ // 1 month (31days), align the timestamp at the first day of month.
+ else if (interval_ms == 2678400000L)
+ return toFirstDayOfMonth(timestamp);
+ // 1 week (7days), align the timestamp at the first day of week.
+ else if (interval_ms == 604800000L)
+ return toFirstDayOfWeek(timestamp);
+ // default timestamp normalization (tsdb v2.1.0)
+ else
+ return timestamp - (timestamp % interval_ms);
+ }
+
+ /** Returns timestamp aligned at the first day of month. */
+ private long toFirstDayOfMonth(long timestamp) {
+ Calendar c = Calendar.getInstance();
@manolama
manolama Aug 27, 2015 Member

Be sure to pass in the time zone if set.

@manolama manolama commented on the diff Aug 27, 2015
src/core/Downsampler.java
@@ -192,8 +193,55 @@ private long getIntervalTimestamp() {
}
/** Returns timestamp aligned by interval. */
- private long alignTimestamp(long timestamp) {
- return timestamp - (timestamp % interval_ms);
+ private long alignTimestamp(long timestamp) {
+ // 1 year (365days), align the timestamp at the first day of year.
+ if (interval_ms == 31536000000L)
@manolama
manolama Aug 27, 2015 Member

It should handle multiples as well, e.g. 2y or 4w
And please follow TSDB's coding style: http://opentsdb.net/docs/build/html/development/index.html#guidelines

@mxk1235
Contributor
mxk1235 commented Dec 12, 2015

this is super useful. is this going to make it into 2.2?

@cpdevoto
Contributor

This pull request is incomplete. I have issued a new pull request (#657) which builds on the work initiated by @moarcaccio, but which also addresses all of the enhancements requested by @manolama, and resolves a few defects with the original pull request.

@manolama
Member

Merged @cpdevoto's in 5012f7d

@manolama manolama closed this Mar 13, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment