now().timestamp() not returning correct value

[tomlut]

Home Assistant release with the issue:

0.91.4

Last working Home Assistant release (if known):
Unknown

Operating environment (Hass.io/Docker/Windows/etc.):

Hassio NUC image

Component/platform:

https://www.home-assistant.io/docs/configuration/templating/

Description of problem:
The following two templates return the same value:

{{ utcnow().timestamp() }}
{{ now().timestamp() }}

1556243287.686556
1556243287.686613

These however do not return the same value:

{{ utcnow() }}
{{ now() }}

2019-04-26 01:48:07.686663+00:00
2019-04-26 11:48:07.686674+10:00

This has rendered my previously functional time templates incorrect and I now have to add my timezone offset in seconds to the now() timestamp.

Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):

Traceback (if applicable):

Additional information:

[SukramJ]

Hi,
A timestamp is the number of seconds between a particular date and January 1, 1970 at UTC.
As a result: It doesn't matter if you use {{ utcnow().timestamp() }} or {{ now().timestamp() }}
I think all outputs are correct.

BR

[tomlut]

Well something has changed because this used to work:

      value_template: >-
        {% set d = now().strftime("%Y-%m-%d ") %}
        {% set t = now().timestamp() %} ### NOTE THIS
        {% set am_start = strptime(d + states('input_datetime.lounge_ac_am_on_time'), '%Y-%m-%d %H:%M:%S').timestamp() %}
        {% set am_end = strptime(d + states('input_datetime.lounge_ac_am_off_time'), '%Y-%m-%d %H:%M:%S').timestamp() %}
        {{ (am_start <= t <= am_end) and is_state('input_boolean.lounge_ac_am_automation', 'on') }}

Now I have to use:

      value_template: >-
        {% set d = now().strftime("%Y-%m-%d ") %}
        {% set t = now().timestamp() + 36000 %} #### THIS IS WHAT I HAVE TO USE NOW
        {% set am_start = strptime(d + states('input_datetime.lounge_ac_am_on_time'), '%Y-%m-%d %H:%M:%S').timestamp() %}
        {% set am_end = strptime(d + states('input_datetime.lounge_ac_am_off_time'), '%Y-%m-%d %H:%M:%S').timestamp() %}
        {{ (am_start <= t <= am_end) and is_state('input_boolean.lounge_ac_am_automation', 'on') }}

The timestamp for now() used to be the number of seconds from the epoch to the current local time. Now it has become equivalent to the timestamp for utcnow().

[SukramJ]

Looks like, but i think now it is correct.
I am not an expert for templates, but i think you should add the timezone offset (+10)or timezone name to strptime instead of adding 36000 to the timestamp.
BR

[SukramJ]

Mmmh this one works on my 0.92.0. I am in timezone +0200
This is your old version, but I replaced the input_datetime by a fixed value for testing.
I tested this with the development tools for templates.

value_template: >-
        {% set d = now().strftime("%Y-%m-%d ") %}
        {% print d %} 
        {% set t = now().timestamp() %} 
        {% print t %} 
        {% set am_start = strptime(d+'08:30:00', '%Y-%m-%d %H:%M:%S').timestamp() %}
        {% print am_start  %} 
        {% set am_end = strptime(d+'08:45:00', '%Y-%m-%d %H:%M:%S').timestamp() %}
        {% print am_end %} 
        {{ (am_start <= t <= am_end) }}

[tomlut]

In the template editor what do you get for:

{{ utcnow().timestamp() }}
{{ now().timestamp() }}

Edit:I'm a bit reluctant to update to 0.92.0 considering the flood of bug reports.

[SukramJ]

1556261835.273354
1556261835.273411
~ identical

[tomlut]

That's the problem. Until recently they weren't identical. The timestamp for now() used to include the local offset.

[SukramJ]

{{ utcnow() }}
{{ now() }}
->
2019-04-26 07:10:22.099300+00:00
2019-04-26 09:10:22.099335+02:00

And that is definitly correct now.

[tomlut]

Yep. Mine too.

now() includes the offset but now().timestamp() does not.

And I'm 100% sure it used to because I have templates that depend on the offset being included in the timestamp that were working perfectly. Unfortunately I did not notice when they stopped working as they run heaters and recently it has been too warm for them to come on automatically.

[SukramJ]

That is correct.
A timestamp is the number of seconds between a particular date and January 1, 1970 at UTC.
That's why {{ utcnow().timestamp() }} and {{ now().timestamp() }} are equal.

Maybe an issue with strptime in 0.91.4 that local timezone is not considered?

If you agree that my short template (based on yours) is nearly identically. Than is issue seems to be fixed with 0.92.0

[tomlut]

No you're missing the point.

The timestamp was not the difference between the epoch and UTC for now().timestamp() it was the difference between the local time and the epoch. It has only changed to the epoch to UTC difference recently. It has nothing to do with strptime .

These two templates used to evaluate to different timestamps (differing by the local offset)

{{ utcnow().timestamp() }}
{{ now().timestamp() }}

Now they dont for some reason.

[SukramJ]

Ok, that the source i referenced: docs.python.org

• The epoch is the point where the time starts, and is platform dependent. For Unix, the epoch is January 1, 1970, 00:00:00 (UTC). To find out what the epoch is on a given platform, look at time.gmtime(0).
  

• The term seconds since the epoch refers to the total number of elapsed seconds since the epoch, typically excluding leap seconds. Leap seconds are excluded from this total on all POSIX-compliant platforms.
  

[tomlut]

Definitions are not the issue. I can only say it so many times:

These were different:

{{ utcnow().timestamp() }}
{{ now().timestamp() }}

Now they are not.

Can we please have the local offset added back to now().timestamp().

[SukramJ]

@awarecan, @MartinHjelmare
Is it a bug or desired behaviour?

[awarecan]

That is complex. You have system timezone, the timezone defined in your configuration.yaml, and the timezone pass in now() method.

Bottom line, the now() method used in the template is not vanilla python datetime.now(), it is now() in homeassistant/util/dt.py

Python native datetime object don't have tzinfo, e.g. it is local time, no timezone information. However, the way HA now() written make sure we always have tzinfo in our now() result. So you cannot use python document as reference to discuss HA's datetime object.

[SukramJ]

@awarecan thanks for clarification
But what does it mean for the question if {{ utcnow().timestamp() }}
and {{ now().timestamp() }} are expected to be equal or not?

[tomlut]

However, the way HA now() written make sure we always have tzinfo in our now() result.

I like the sound of that. The only problem is that {{ now().timestamp() }} no longer seems to be including the local offset. It is identical to {{ utcnow().timestamp() }} and I'm pretty sure it shouldn't be.

[SukramJ]

Latest change of this file was 9 Oct 2018

[pnbruckner]

@tomlut, you said:

The timestamp was not the difference between the epoch and UTC for now().timestamp() it was the difference between the local time and the epoch. It has only changed to the epoch to UTC difference recently. It has nothing to do with strptime .

It has everything to do with strptime, and the time zone settings in HA and the environment in which HA runs.

As I explained here, one issue is naive datetime objects vs aware datetime objects. As @awarecan, said, in HA, both now() and utcnow() return aware datetime objects. But in your templates you are creating naive datetime objects. When you use a naive datetime object's timestamp method, it gets the time zone information from the environment in which HA runs (typically the OS, but possibly something different in a docker I understand.) If that time zone setting is incorrect, then the timestamp will be incorrect.

So, in your old setup, when you got different results with now().timestamp() vs utcnow().timestamp(), a time zone setting somewhere (HA or OS) was wrong, and/or the clock was set wrong. If everything was set correctly, then these two should agree.

In your new setup they do agree, so that's good. (Well, it's good if they agree because the clock and the time zone settings in both HA and the underlying environment in which HA runs are correct.)

I think the bottom line is, your templates use both naive and aware datetime objects, which makes them vulnerable to incorrect time zone settings, even more so than usual.

[tomlut]

Phil is correct. This was a system (HassOS, not home assistant) time zone setting problem and the issue that caused with naive datetime objects. An update to 0.92.0 corrected this time zone setting and my original templates are working again.

Thank you all for your assistance.