Skip to content

Feat/time offsets #109

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Nov 15, 2022
Merged

Feat/time offsets #109

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
7dbe2b2
Added offsets. Work In Progress
dbuezas Oct 31, 2022
60c2e4a
Added offset yaml to readme
dbuezas Oct 31, 2022
d8ac252
Don't clear out-of-view data from the cache
dbuezas Nov 12, 2022
89528b1
Add extend_to_present option to entities
dbuezas Nov 12, 2022
923076f
Reintroduced range out-of-screen data cleaning, but out of the cache
dbuezas Nov 12, 2022
7fd272b
Clear cache when exiting browsing mode (e.g scroll, zoom)
dbuezas Nov 13, 2022
0417972
refactor: move images in a separate folder
Nov 14, 2022
f021a6a
docs: add offset to the README
Nov 14, 2022
8a40c4f
Fix inverted `entity.extend_to_present`default
dbuezas Nov 14, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
0 demo.gif → docs/resources/demo.gif
View file
Open in desktop
File renamed without changes
0 demo2.gif → docs/resources/demo2.gif
View file
Open in desktop
File renamed without changes
0 example1.png → docs/resources/example1.png
View file
Open in desktop
File renamed without changes
Binary file added docs/resources/offset-nowline.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/resources/offset-temperature.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
0 rangeselector.apng → docs/resources/rangeselector.apng
View file
Open in desktop
File renamed without changes.
112 changes: 108 additions & 4 deletions readme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@

# Plotly Graph Card

<img src="demo.gif" width="300" align="left">
<img src="demo2.gif" width="300" align="right">
<img src="docs/resources/demo.gif" width="300" align="left">
<img src="docs/resources/demo2.gif" width="300" align="right">

<br clear="both"/>
<br clear="both"/>
Expand Down Expand Up @@ -54,7 +54,7 @@ refresh_interval: 10

### Filling, line width, color

![](example1.png)
![](docs/resources/example1.png)

```yaml
type: custom:plotly-graph
Expand All @@ -81,7 +81,7 @@ refresh_interval: 10 # in seconds

### Range Selector buttons

![](rangeselector.apng)
![](docs/resources/rangeselector.apng)

```yaml
type: custom:plotly-graph
Expand Down Expand Up @@ -242,6 +242,92 @@ entities:

Note that `5minute` period statistics are limited in time as normal recorder history is, contrary to other periods which keep data for years.

## Offsets
Offsets are useful to shift data in the temporal axis. For example, if you have a sensor that reports the forecasted temperature 3 hours from now, it means that the current value should be plotted in the future. With the `offset` attribute you can shift the data so it is placed in the correct position.
Another possible use is to compare past data with the current one. For example, you can plot yesterday's temperature and the current one on top of each other.

The `offset` flag can be specified in two places.
**1)** When used at the top level of the configuration, it specifies how much "future" the graph shows by default. For example, if `hours_to_show` is 16 and `offset` is 3h, the graph shows the past 13 hours (16-3) plus the next 3 hours.
**2)** When used at the trace level, it offsets the trace by the specified amount.


```yaml
type: custom:plotly-graph
hours_to_show: 16
offset: 3h
entities:
- entity: sensor.current_temperature
line:
width: 3
color: orange
- entity: sensor.current_temperature
name: Temperature yesterday
offset: 1d
line:
width: 1
dash: dot
color: orange
- entity: sensor.temperature_12h_forecast
offset: 12h
name: Forecast temperature
line:
width: 1
dash: dot
color: grey
```

![Graph with offsets](docs/resources/offset-temperature.png)

### Now line
When using offsets, it is useful to have a line that indicates the current time. This can be done by using a lambda function that returns a line with the current time as x value and 0 and 1 as y values. The line is then hidden from the legend.

```yaml
type: custom:plotly-graph
hours_to_show: 6
offset: 3h
entities:
- entity: sensor.forecast_temperature
yaxis: y1
offset: 3h
- entity: sensor.nothing_now
name: Now
yaxis: y9
showlegend: false
line:
width: 1
dash: dot
color: deepskyblue
lambda: |-
() => {
return {x:[Date.now(),Date.now()], y:[0,1]}
}
layout:
yaxis9:
visible: false
fixedrange: true
```

![Graph with offsets and now-line](docs/resources/offset-nowline.png)

## Duration
Whenever a time duration can be specified, this is the notation to use:

| Unit | Suffix | Notes |
|--------------|--------|----------|
| Milliseconds | `ms` | |
| Seconds | `s` | |
| Minutes | `m` | |
| Hours | `h` | |
| Days | `d` | |
| Weeks | `w` | |
| Months | `M` | 30 days |
| Years | `y` | 365 days |

Example:
```yaml
offset: 3h
```

## Extra entity attributes:

```yaml
Expand All @@ -266,6 +352,24 @@ entities:
<extra></extra>
```

### Extend_to_present
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@FrnchFrgg
Here's the toggle for extend_to_present with the defaults you suggested


The boolean `extend_to_present` will take the last known datapoint and "expand" it to the present by creating a duplicate and setting its date to `now`.
This is useful to make the plot look fuller.
It's recommended to turn it off when using `offset`s, or when setting the mode of the trace to `markers`.
Defaults to `true` for state history, and `false` for statistics.

```yaml
type: custom:plotly-graph
entities:
- entity: sensor.weather_24h_forecast
mode: "markers"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Side note: is mode documented somewhere?

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not within this repo's readme, it's a plotlyjs feature, so it's in their docs

extend_to_present: false # true by default for state history
- entity: sensor.actual_temperature
statistics: mean
extend_to_present: true # false by default for statistics
```

### `lambda:` transforms

`lambda` takes a js function (as a string) to pre process the data before plotting it. Here you can do things like normalisation, integration. For example:
Expand Down
47 changes: 14 additions & 33 deletions src/cache/Cache.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,11 +124,14 @@ export default class Cache {
}
getHistory(entity: EntityConfig) {
let key = getEntityKey(entity);
return this.histories[key] || [];
const history = this.histories[key] || [];
return history.map((datum) => ({
...datum,
timestamp: datum.timestamp + entity.offset,
}));
}
async update(
range: TimestampRange,
removeOutsideRange: boolean,
entities: EntityConfig[],
hass: HomeAssistant,
significant_changes_only: boolean,
Expand All @@ -138,13 +141,17 @@ export default class Cache {
return (this.busy = this.busy
.catch(() => {})
.then(async () => {
if (removeOutsideRange) {
this.removeOutsideRange(range);
}
const promises = entities.flatMap(async (entity) => {
const promises = entities.map(async (entity) => {
const entityKey = getEntityKey(entity);
this.ranges[entityKey] ??= [];
const rangesToFetch = subtractRanges([range], this.ranges[entityKey]);
const offsetRange = [
range[0] - entity.offset,
range[1] - entity.offset,
];
const rangesToFetch = subtractRanges(
[offsetRange],
this.ranges[entityKey]
);
for (const aRange of rangesToFetch) {
const fetchedHistory = await fetchSingleRange(
hass,
Expand All @@ -161,30 +168,4 @@ export default class Cache {
await Promise.all(promises);
}));
}

removeOutsideRange(range: TimestampRange) {
this.ranges = mapValues(this.ranges, (ranges) =>
subtractRanges(ranges, [
[Number.NEGATIVE_INFINITY, range[0] - 1],
[range[1] + 1, Number.POSITIVE_INFINITY],
])
);
this.histories = mapValues(this.histories, (history) => {
let first: EntityState | undefined;
let last: EntityState | undefined;
const newHistory = history.filter((datum) => {
if (datum.timestamp <= range[0]) first = datum;
else if (!last && datum.timestamp >= range[1]) last = datum;
else return true;
return false;
});
if (first) {
newHistory.unshift(first);
}
if (last) {
newHistory.push(last);
}
return newHistory;
});
}
}
68 changes: 51 additions & 17 deletions src/plotly-graph-card.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,12 @@ import Plotly from "./plotly";
import {
Config,
EntityConfig,
EntityState,
InputConfig,
isEntityIdAttrConfig,
isEntityIdStateConfig,
isEntityIdStatisticsConfig,
TimestampRange,
} from "./types";
import Cache from "./cache/Cache";
import getThemedLayout from "./themed-layout";
Expand Down Expand Up @@ -49,12 +51,33 @@ function patchLonelyDatapoints(xs: Datum[], ys: Datum[]) {
}
}

function extendLastDatapointToPresent(xs: Datum[], ys: Datum[]) {
function extendLastDatapointToPresent(
xs: Datum[],
ys: Datum[],
offset: number
) {
if (xs.length === 0) return;
const last = JSON.parse(JSON.stringify(ys[ys.length - 1]));
xs.push(new Date());
xs.push(new Date(Date.now() + offset));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extendLastDatapointToPresent(...) should be disabled when an offset is set. Those traces reflects values that are not related to the present.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's discuss this.
An extreme case is a monthly statistic, without extending, it will look like the plot just stops at end of last month.
IMO, of the behavior is to extend to the present, then it should be consistent or it will surprise the user.
I'll put some example screenshots later

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, it depends. 😂
When using offsets, we are talking about things that occurred in the past and is now being shifted.
If we talk about forecast it doesn't make sense to extend, because the forecast is referred to a specific point in time, extending it is nonsensical.
However, when talking a tank level, if the level stays at X liters, and it doesn't change, it would make sense to extend it.

Neither is perfect, but maybe the second scenario is more common so we should stick with your vision. Opinions?

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True. Give it a try when you test it. I think when you see how a part of the plot disappears as soon as the offset is added you may get the same "this is wrong" feeling.

It is true that it can be seen as a (poor) forecast, but it can also be interpreted as "the last known state at that point".

Let me know what you find out from experimenting. We can always add an option to deactivate the extension

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For temperatures and statistics (which are currently timed at the start of their period) extending is surprising, because it extends horizontally between one and two hours but the real temperature didn't stay constant. It looks wierd. If you extended by adding the current state instead of the last fetched one it would look better, but that's IMHO too smart, non generalizable and out of scope.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "extendLastDatapointToPresent" is not a problem if the offset is moving the trace later.

If one offsets right (positive) by a month and the visible range is a week, the only the section [now - 1month -1 week, now - 1month] will he fetched] . Extending that to now will just add a huge 1 month straight line (visible when you zoom).
If one offsets left (negative), then it is less problematic

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if having data past now is not a problem

X axis scaling is now done explicitly, so this is less of an issue now

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that my proposition makes having an offset (say a small enough offset that you cannot see, but non-zero nonetheless) totally transparent and not changing suddenly the way the trace looks.

I think i got you now. Whatever we do, it should be equivalent to expanding the true last state first and the offseting. (Not offsetting and then expanding the last fetched one, which may not be the latest state of the sensor)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think i got you now. Whatever we do, it should be equivalent to expanding the true last state first and the offseting.

If there is an existing state right of the visible range, it should probably be fetched for continuity purposes. Extending the last visible state to the right only works when plotting the values piecewise constant. With other interpolation methods the result is wrong.

If there is no such later state, you may want to assume the state did not change by extending to the right, but never past now+offset. I believe this should default to true for normal history and false for statistics, but should be configurable by the user.

Sadly, I don't think HA has a suitable API to fetch « whatever is the next point after this time » so the first paragraph may be infeasible at least for sensors updating very irregularly.

Copy link
Owner Author

@dbuezas dbuezas Nov 12, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

never past now+offset

absolutly

I believe this should default to true for normal history

Agree

and false for statistics

It took me a second to realise why, but I agree now too

but should be configurable by the user.

Also agree. Luckily very easy to implement with the new better structured code.

I don't think HA has a suitable API to fetch « whatever is the next point after this time »

Yeah, I didn't find such an API either.

I'll give this a try now and finally merge this PR

ys.push(last);
}
function removeOutOfRange(xs: Datum[], ys: Datum[], range: TimestampRange) {
let first = -1;
let last = -1;

for (let i = 0; i < xs.length; i++) {
if (xs[i]! < range[0]) first = i;
if (xs[i]! > range[1]) last = i;
}
if (last > -1) {
xs = xs.splice(last);
ys = ys.splice(last);
}
if (first > -1) {
xs = xs.splice(0, first);
ys = ys.splice(0, first);
}
}

console.info(
`%c ${componentName.toUpperCase()} %c ${version} ${process.env.NODE_ENV}`,
Expand Down Expand Up @@ -199,8 +222,6 @@ export class PlotlyGraph extends HTMLElement {
this.fetch();
}
if (shouldPlot) {
if (!this.isBrowsing)
this.cache.removeOutsideRange(this.getAutoFetchRange());
this.plot();
}
}
Expand Down Expand Up @@ -252,7 +273,10 @@ export class PlotlyGraph extends HTMLElement {
}
getAutoFetchRange() {
const ms = this.parsed_config.hours_to_show * 60 * 60 * 1000;
return [+new Date() - ms, +new Date()] as [number, number];
return [
+new Date() - ms + this.parsed_config.offset,
+new Date() + this.parsed_config.offset,
] as [number, number];
}
getAutoFetchRangeWithValueMargins() {
const [start, end] = this.getAutoFetchRange();
Expand Down Expand Up @@ -290,20 +314,18 @@ export class PlotlyGraph extends HTMLElement {
return +parseISO(date);
});
}
async enterBrowsingMode() {
enterBrowsingMode = () => {
this.isBrowsing = true;
this.resetButtonEl.classList.remove("hidden");
}
};
exitBrowsingMode = async () => {
this.isBrowsing = false;
this.resetButtonEl.classList.add("hidden");
this.withoutRelayout(async () => {
await Plotly.relayout(this.contentEl, {
uirevision: Math.random(), // to trigger the autoranges in all y-yaxes
xaxis: { range: this.getAutoFetchRangeWithValueMargins() }, // to reset xaxis to hours_to_show quickly, before refetching
});
await this.plot(); // to reset xaxis to hours_to_show quickly, before refetching
this.cache.clearCache(); // so that when the user zooms out and autoranges, not more that what's visible will be autoranged
await this.fetch();
});
await this.fetch();
};
onRestyle = async () => {
// trace visibility changed, fetch missing traces
Expand Down Expand Up @@ -368,6 +390,7 @@ export class PlotlyGraph extends HTMLElement {
title: config.title,
hours_to_show: config.hours_to_show ?? 1,
refresh_interval: config.refresh_interval ?? "auto",
offset: parseTimeDuration(config.offset ?? "0s"),
entities: config.entities.map((entityIn, entityIdx) => {
if (typeof entityIn === "string") entityIn = { entity: entityIn };

Expand All @@ -386,6 +409,7 @@ export class PlotlyGraph extends HTMLElement {
config.defaults?.entity,
entityIn
);
entity.offset = parseTimeDuration(entityIn.offset ?? "0s");
if (entity.lambda) {
entity.lambda = window.eval(entity.lambda);
}
Expand Down Expand Up @@ -432,6 +456,7 @@ export class PlotlyGraph extends HTMLElement {
throw new Error(
`period: "${entity.period}" is not valid. Use ${STATISTIC_PERIODS}`
);
entity.extend_to_present ??= !entity.statistic;
}
const [oldAPI_entity, oldAPI_attribute] = entity.entity.split("::");
if (oldAPI_attribute) {
Expand Down Expand Up @@ -487,8 +512,7 @@ export class PlotlyGraph extends HTMLElement {
const was = this.parsed_config;
this.parsed_config = newConfig;
const is = this.parsed_config;
if (!this.contentEl) return;
if (is.hours_to_show !== was?.hours_to_show) {
if (is.hours_to_show !== was?.hours_to_show || is.offset !== was?.offset) {
this.exitBrowsingMode();
}
await this.fetch();
Expand Down Expand Up @@ -530,7 +554,6 @@ export class PlotlyGraph extends HTMLElement {
try {
await this.cache.update(
range,
!this.isBrowsing,
visibleEntities,
this.hass,
this.parsed_config.minimal_response,
Expand Down Expand Up @@ -593,7 +616,14 @@ export class PlotlyGraph extends HTMLElement {

let xs: Datum[] = xsIn;
let ys = ysIn;
extendLastDatapointToPresent(xs, ys);
if (trace.extend_to_present) {
extendLastDatapointToPresent(xs, ys, trace.offset);
}
if (!this.isBrowsing) {
// to ensure the y axis autorange containst the yaxis
removeOutOfRange(xs, ys, this.getAutoFetchRangeWithValueMargins());
}

if (trace.lambda) {
try {
const r = trace.lambda(ysIn, xsIn, history);
Expand Down Expand Up @@ -662,7 +692,11 @@ export class PlotlyGraph extends HTMLElement {
units.map((unit, i) => ["yaxis" + (i == 0 ? "" : i + 1), { title: unit }])
);
const layout = merge(
{ uirevision: true },
{
uirevision: this.isBrowsing
? this.contentEl.layout.uirevision
: Math.random(), // to trigger the autoranges in all y-yaxes
},
{
xaxis: {
range: this.isBrowsing
Expand Down
Loading