Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Lens] Formula: Time scaling #94605

Closed
flash1293 opened this issue Mar 15, 2021 · 9 comments · Fixed by #128303
Closed

[Lens] Formula: Time scaling #94605

flash1293 opened this issue Mar 15, 2021 · 9 comments · Fixed by #128303
Assignees
Labels
enhancement New value added to drive a business result Feature:Lens Team:Visualizations Visualization editors, elastic-charts and infrastructure

Comments

@flash1293
Copy link
Contributor

It's easy to allow the user to do time scaling of the full formula using the UI outside of the formula syntax itself.

However in some cases, it's easier to understand or even necessary to do time scaling of just a part of the formula which requires a way to call it from within the formula.

Implementation

Add a function time_scale(<metric or math>, unit=<unit>) (unit is s/m/h/d, metric is just another AST node) which compiles in the same way as a calculation operation like cumulative sum (nested math is calculated as a separate operation, then the time scaling, then the root math context as a separate operation) and uses the existing time scale expression function.

Use case

Adding a counter metric and a gauge metric: time_scale(counter_rate(max(in_bytes)), unit=s) + last_value(internal_traffic_speed)

@flash1293 flash1293 added discuss enhancement New value added to drive a business result Team:Visualizations Visualization editors, elastic-charts and infrastructure Feature:Lens labels Mar 15, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-app (Team:KibanaApp)

@wylieconlon
Copy link
Contributor

I was expecting that we'd do time scaling as a named parameter, not a separate function. But no strong opinion about how we do this.

@flash1293
Copy link
Contributor Author

If we do it as a named parameter the user can't use it on a math node anymore:
time_scale(sum(bytes_in) + sum(bytes_out), unit=s)

That's not a big issue because time scaling is distributive and I can't think of a good reason to do anything else besides adding / subtracting time scaled metrics, so
sum(bytes_in, time_scale=s) + sum(bytes_out, time_scale=s)

would work just as well.

It still seemed more consistent to me to have it as a separate function because it's very similar in nature to calculations like cumulative_sum , differences etc.

@wylieconlon
Copy link
Contributor

wylieconlon commented Mar 23, 2021

Brainstorming some alternatives here, just for completeness:

  • Time scaling could be allowed only at the top level of the formula, which would let us apply a time suffix formatter to the entire column.
  • Similar to TSVB, we could inject named variables into the math context, like interval_ms and time_range.from. This would let users construct a time scaling expression manually, by doing (sum(bytes_in) / interval_ms) * 1000
  • We could expose a time scaling function which has different options for how the time interval is calculated. This would let users choose a date histogram or the global time picker for scaling info

I think the /s formatter is pretty important, so I would propose that we allow time scaling of the entire formula using the dropdown options we have today. Since this won't cover every use case, I would also propose that we introduce some named variables so that users can create their own time-based math.

@flash1293
Copy link
Contributor Author

flash1293 commented Mar 23, 2021

I think the /s formatter is pretty important, so I would propose that we allow time scaling of the entire formula using the dropdown options we have today

Agreed, this should be the go-to solution. This issue was meant as an extension of this for the cases a global setting won't work

Similar to TSVB, we could inject named variables into the math context, like interval_ms and time_range.from

I'm not sure whether I'm a big fan of this - as there are edge cases with uneven intervals (like months, leap years, DST switches and so on), I think a function would be a cleaner approach because we handle these cases already.

@ojdo
Copy link

ojdo commented Mar 11, 2022

Just to be sure: as of now (8.1.0), I cannot accomplish any "normalize by time unit" output in lens for data coming from a custom formula?

My use case is to display an index against its sum(..., shift={1,2,3,4}w)-ed counterparts, but preferably formatted as a (1/s) rate, not the raw sum.

@flash1293
Copy link
Contributor Author

@ojdo That's correct, but by using the "customize interval" option for the underlying date histogram, the bucket size will stay the same for most situations, so you could hardcode the value in there:
Screenshot 2022-03-15 at 17 07 33
Screenshot 2022-03-15 at 17 08 44

Caveat: If the user selects a long time range (e.g for the hour interval more than a week), Lens will change the used interval and the calculation won't be accurate anymore. You can push out that limit by increasing the value in advanced settings here:
Screenshot 2022-03-15 at 17 10 26

But the general point still applies, it will just happen later.

If that works for your use case it might be a good work around until we will get to this feature. It's planned for the near term future though (probably 8.3)

@ojdo
Copy link

ojdo commented Mar 16, 2022

First, thanks for confirming and fueling my anticipation for future 8.x releases.
Second: I have this very work-around, i.e. baking the conversion into the data, so to speak, currently in another widget (aggregation-based line chart - great for multiple breakdowns, but scaling...), so I'd rather not expand its use.

@flash1293 flash1293 removed the discuss label Mar 21, 2022
@flash1293
Copy link
Contributor Author

This can be built super similar to the existing overall_* functions:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New value added to drive a business result Feature:Lens Team:Visualizations Visualization editors, elastic-charts and infrastructure
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants