Local time and timezone support for MicroPython (meta-discussion)

[projectgus]

Summarising this common problem and the current best solutions and possibility for a long-term fix.

Problem

Some embedded applications care about the local time. MicroPython always uses UTC timezone and its time module is generally not timezone or localtime aware.

This seems to usually impact folks using ntptime module from micropython-lib. This returns UTC (NTP protocol only uses UTC). How to display a local time somewhere?

Current Simplest Solutions ✔️

• mip install datetime from micropython-lib and convert the timestamp as shown here (thanks @jimmo).
• Make your own one-liner localtime() function, as demonstrated by @MrEbbinghaus (thanks!).

Other Proposed Solutions

• @bellingeri has a pure Python POSIX timezone parser (impressive!). See also the discussion about it.

The following solutions add timezone offset support to the ntptime module:

• @peterhinch has an alternative ntptime.py in his micropython-samples.
• @MrTinkerman has another ntptime.py here.
• Multiple PRs have proposed adding local time into micropython-lib ntptime:
  • Update ntptime.py micropython-lib#715
  • added timezone support in ntptime library micropython-lib#597
  • Feature/ntp timezone support micropython-lib#635
  • Added timezone support for NTP micropython-lib#587
  • ntptime: Allows manual timezone setting. micropython-lib#695
  • ntptime.py: added simple timezone support. #7075

This is clearly something people run into a lot!

Long term solutions

As others, including maintainers, have already said in the linked discussions, the official micropython-lib ntptime module should keep working with UTC. However, it would clearly be useful to have an easy way to convert timestamps to a local timezone in MicroPython.

There are a range of possible solutions here, from a simple "local time offset" variable to a full POSIX timezone format compliant implementation that automatically switches for DST, etc.

Current Status

• Except for the Unix port, the time module in MicroPython has no local time. time.gmtime() and time.localtime() are literally the same function.
• The result of time.gmtime() and time.localtime() in MicroPython is different to CPython. MicroPython returns a tuple with 8 of the 9 indexed fields found in CPython struct_time. CPython's result has a ninth tm_isdst field, plus named access to those 9 fields and the extra fields tm_zone and tm_gmtoff.
• None of CPython's time.timezone, time.tzname, time.tzset(), etc. are implemented in MicroPython due to (reasonable) size constraints.
• micropython-lib datetime is "micro" in size but very comprehensive in function (thanks @lorcap, see Add new module python-stdlib/datetime micropython-lib#449). There is a .patch already in the source tree to re-enable naive datetime support on platforms with meaningful time.localtime() ❤️ .

(An aside, CPython's time module behaviour on Linux was a little different to my initial naive expectation. time.timezone, time.alttime, time.dst variables are writable but changing them doesn't change the time.localtime() result. To do this requires updating TZ (either locally or globally) and calling time.tzset(), which then updates those three variables to match the result of TZ. This makes sense (otherwise what happens when DST transitions and a CPython process is running?) but I still had naively expected changing the variables to have some effect.)

Existing port support

• As mentioned, the Unix port already has an implementation of time.localtime() that calls the POSIX localtime() function. It could easily add more timezone support in the same way (i.e. tzset()), if there was a use for that.
• On some other platforms (Windows, ESP32) the platform has support for POSIX timezone specifications but MicroPython doesn't use them. (There may be tradeoffs on some platforms, for example ESP-IDF has full support for TZ environment variable and setting it but I don't know how much binary size this adds.) Some work has been done on these already, for example @tve has this PR for ESP32.
• On most other platforms there is zero existing support for any of this.

Way forward

This is just me spitballing at the moment:

• For esp32 port, it's technically possible to add a fully CPython compatible implementation. The existing PR is almost exactly this, except it adds a (helpful but incompatible) timezone argument to time.tzset() rather than requiring the TZ environment variable to be set. (Currently only Unix port has a Python interface for setting environment variables, but we could also cheat and expose something like esp32.TZ.)
• For bare metal ports, it seems like it'd be viable and lightweight to have something like machine.RTC.local_tz where a timezone offset in seconds can be written (maybe even micropython.local_tz?). If the application sets this, time.localtime() applies this offset. (The obvious choice here would be to implement time.timezone, etc. and use those, but that isn't how CPython uses them. 😿)
• There is also an idea proposed by @MrEbbinghaus (see here) that ports could choose between supporting simple timezone strings with an offset (i.e. CET-1), or supporting the full POSIX timezone strings (CET-1CEST,M3.5.0,M10.5.0/3).
• Micropython 2.0 could make a breaking change and return either a 9-tuple or struct_time from gmtime() and localtime(), making the results compatible to CPython.
• (Getting amibitious) For the true timezone enthusiast, it would be awesome to mip install micropython-timezone (or similar) that adds a way to set a POSIX timezone specifier string which is then parsed in Python and automatically applied to the system. This would require adding some kind of hook to time.localtime or having the micropython-lib package monkey patch that function, though, and I don't know if that would be acceptable.

[projectgus]

This would require adding some kind of hook to time.localtime or having the micropython-lib package monkey patch that function, though, and I don't know if that would be acceptable.

Maybe this isn't true actually, if the local_tz variable is already implemented then all this might need is a way to install a callback from machine.RTC to rewrite local_tz when the timezone transitions...

[ThinkTransit]

I personally use the datetime module because it supports creating tz objects with daylight savings support etc.... I like the concept of using the datetime module because it is compatible with cpython however I feel it is a little bit slow, when compared to doing a simple add/subtract of seconds.

You mention that None of CPython's [time.timezone, time.tzname, time.tzset(), etc.](https://docs.python.org/3.5/library/time.html#time.timezone) are implemented in MicroPython due to (reasonable) size constraints.
Would it be a feasible to implement these but have them as optional module/component?

[Wind-stormger]

Adding the least amount of code #695 ntptime.py#L45-L46 is my idea, which is also derived from peterhinch/ntptime.py#L36C19.

It seems like there are some drawbacks to this?

I mean, will this cause problems in practical application?

During my test, after setting the time zone, the time.gmtime() return was consistent with my local time.

[projectgus]

During my test, after setting the time zone, the time.gmtime() return was consistent with my local time.

You mean time.gmtime() returned the local time and not UTC time, is that right? I agree this probably works fine for a lot of applications that only need local time, but this is also the problem.

Specifically, if you include any code that relies on getting a valid Epoch timestamp from time.time(), or a UTC time value from time.gmtime(), then that code will no longer be reading the correct time.

[Wind-stormger]

So it's best to have a time.localtime() .

[peterhinch]

There are two application cases. [EDITED]

1. Where the application will always run in one location. It is easy to write region-specific code which checks the current date against the rules for DST and adjusts the time accordingly.
2. Where an application is to run in global locations then full POSIX parsing is the best solution.

As an example of 1. see this module which is a minimal Date class for doing date calculations. The time_offset method is written for UK DST but could readily be adapted for other locations.

[projectgus]

I agree in general @peterhinch and @bellingeri, although I note there's a special case of "1" where a significant proportion of people are in timezones with no DST - so that option is really easy for them, and less easy for the rest of us.

Suggest we can break down the use cases further as:

1. Application doesn't care about local time. (Also, may prefer that timezone handling doesn't take up any extra resources.)
2. Application has a single known timezone. The timezone has no DST, or the application doesn't care about DST. They can set an offset to UTC one time, job done! I think micropython should probably support this out of the box, or with a very small micropython-lib package.
3. Application has a single known timezone. The timezone has DST. The application wants a minimal resource footprint. They can save resources and write custom code for their particular DST transitions. If they want localtime() to keep working over time then this can modify the UTC offset (or set a dst flag) when it changes.
4. The application doesn't know what timezone it will run in, or does know and wants it to "just work". They install "full" timezone support (via micropython-lib) and provide a POSIX timezone string (or allow it to be configured somewhere in the application). localtime() now works the same as on a "big" computer.

For long running programs I was just thinking as to how often it should get a new time from a time server to keep the microcontroller at least accurate to the minute

This is an interesting question, but please start another discussion to ask it as the answer is a "whole thing" that can depend on a few factors (all unrelated to timezones). 😄

[MrEbbinghaus]

You can do 2 with the POSIX string as well. The timezone string isn't a timezone by itself.
The user would have to get the TZ string for the location and call the tz_set function regularly.

TZ_OFFSET_s = 0
FORMAT1 = re.compile(r"^([A-Z]+)([+-]?)(\d+)(?::(\d+))?(?::(\d+))?")

def offset_from_format1(tz_str) -> int:
  tzname, sign, d_hour, d_min, d_sec = re.match(FORMAT1, tz_str).groups(default=0)
  offset = (int(d_hour) * 60 + int(d_min)) * 60 + int(d_sec)
  timezone = offset * (1 if sign == "-" else -1)
  return timezone
  
def dst_data(tz_str: str, year: int) -> tuple[int, int, int]:
  pass # TODO

def offset_from_format2(tz_str):
  non_dst_offset = offset_from_format1(tz_str)
  current_time = time.time() + non_dst_offset
  current_year = time.gmtime(current_time)[0]
  dst_start, dst_end, dst_offset = get_dst_data(tz_str, current_year)
  
  return current_time + (dst_offset if dst_start < current_time < dst_end 0)
  

def tz_set(tz_str = None):
  global TZ_OFFSET_s
  if tz_str:
    TZ_OFFSET_s = offset_from_format2(tz_str)

---

"full just works TZ support" is a little bit harder, I'm afraid.

You would need to regularly (at least twice a year) update your timezone database, as DST rules could change any moment.

[projectgus]

"full just works TZ support" is a little bit harder, I'm afraid.
You would need to regularly (at least twice a year) update your timezone database, as DST rules could change any moment.

Good point, I shouldn't have oversimplified. At best, "just works" really only extends as far as "an application can take the POSIX timezone string that applies for their current time zone and pass it to some API in order to get the corresponding local time behaviour."

I don't think there's a possible outcome where MicroPython starts including an actual timezone database.

[bellingeri]

Did you find problems with my solution (localPTZtime)?

[projectgus]

@bellingeri I don't know if you're asking me or @MrEbbinghaus but speaking for myself I don't see any technical problem with it at all (although i haven't looked very closely).

As to the question of implementing an official MicroPython solution then there are two other factors to using it:

1. Where possible MicroPython aims to provide the same APIs as CPython.
2. localPTZtime is licensed GPLv3, which is a perfectly valid choice but this can't be included (or used as a basis for new work) in MicroPython (which is MIT licensed).

[cwalther]

One detail I haven’t seen mentioned yet: Since currently time.gmtime and time.localtime are the same thing, I can misuse time.mktime as a substitute for calendar.timegm. If time.localtime and its inverse time.mktime were changed to actually use local time, we would also need an implementation of calendar.timegm to have an inverse of time.gmtime.

[projectgus]

Huh... I actually didn't know calendar.timegm was a thing, but it makes sense that this function exists! I guess the other way around this is to construct a datetime object in UTC and read its timestamp, but that's fiddlier. Definitely something to keep in mind.

One good thing about all the proposed changes, as I understand them, is that anyone should have the option to keep localtime in UTC and continue with whatever techniques work for them now.

[scruss]

1. Where the application will always run in one country. It is easy to write country-specific code which checks the current date against the rules for DST and adjusts the time accordingly.

Please don't generalize for simple countries. Consider Canada:

• we have at least six time zones, from Pacific (currently GMT-7) to Atlantic (currently GMT-3) — and also Newfoundland (currently GMT-2:30);
• not every province/territory observes DST: most of Saskatchewan doesn't, so during the summer it has the same time as Alberta to the west of it, but shares the winter time with Manitoba to the east;
• some provinces have more than one time zone. Ontario, for example, is mostly in Eastern time, but the far NW is Central. Other provinces have splits not based strictly on longitude but on local cultural preference.

It's very complicated and very confusing: Time zones and daylight saving time - National Research Council Canada.

[peterhinch]

Good point. I have corrected my post.

[kjm1102]

If the shakers & movers are going to make local time easily available in micropython it might be worth fixing some of the other time related annoyances? In particular the inconsistency between datetime (which uses the year, month, day-of-month, day-of-week, hours, mins, secs, day-of year format) and mktime or gmtime (which use the year, month, day-of-month, hour, mins, secs, day-of-week, day-of-year format).

Interestingly, to me at least, if your project uses cellular connectivity most LTE modems will supply local time via the 4G tower. Even then you are not out of the woods when your application blunders into the twilight zone around 3am twice a year when a whole hour of local time either vanishes or pops into existence like some magical sub atomic particle.

[projectgus]

In particular the inconsistency between datetime (which uses the year, month, day-of-month, day-of-week, hours, mins, secs, day-of year format) and mktime or gmtime (which use the year, month, day-of-month, hour, mins, secs, day-of-week, day-of-year format).

Interesting, do you have a specific example or a link to where this is discussed in more detail? AFAIK all of the micropython-lib datetime APIs match CPython (aside from the gmtime() different mentioned in the first post). However those APIs don't quite match this description.

If there's somewhere that the behaviour either doesn't match CPython, or is a MicroPython-only thing, then there might be room to change it.

Interestingly, to me at least, if your project uses cellular connectivity most LTE modems will supply local time via the 4G tower. Even then you are not out of the woods when your application blunders into the twilight zone around 3am twice a year when a whole hour of local time either vanishes or pops into existence like some magical sub atomic particle.

That is indeed interesting! Do the LTE modems give you the option of UTC, at least?

It must also get weird if you live in an area close to a timezone border, and are able to reach towers in more than one timezone...

[ThinkTransit]

One example here is that datetime.now() isn't fully implemented and returns "not implemented" unless passed a tz object. You could argue that now() is one of the most common datetime functions and not having a working now() doesn't instil confidence in the datetime module. Hoping that this can be merged soon to rectify this.

micropython/micropython-lib#570

[bixb922]

For esp32 port, it's technically possible to add a fully CPython compatible implementation.

This is indeed interesting and would be a good optimization, making the ESP32 implementation smaller. However, it's probably more compatible and less maintenance effort to have the same implementation for all architectures.

For bare metal ports, it seems like it'd be viable and lightweight to have something like machine.RTC.local_tz where a timezone offset in seconds can be written

Having three variables (time zone name, offset and DST indicator) and setting these variables sounds like a simple solution. To use that, the programmer will have to write a task that wakes up when a transition between DST and non-DST occurs, and set the new variables. However, there is no guarantee that this task, be it asyncio, timer or thread, will wake up exactly at the specified time. This task to set the time zone may be triggered a bit late, and other code can call time.localtime() in between. So there will be an interval of time where the offset should have changed, but hasn't yet. This is not desirable. One way to avoid this little design problem is to provide a callback that is called each time time.localtime() is invoked, and that returns (zone name, time offset, DST indicator). I believe the overhead of this call is minimal.

In summary: it might be better to process the time zone rule each time time.localtime() gets called instead of setting time zone variables.

The same reasoning applies to an RTC. I believe the RTC should be set to UTC (never to the local time) and the time zone rules should be applied by software each time the RTC is read. There are several advantages: it makes it possible to work with several time zones, and there is no need for a time critical task to update the RTC on time zone transitions. In fact, if the RTC is set to a a certain local time, it's very tricky to move it back to UTC or to another time zone.

or supporting the full POSIX timezone strings (CET-1CEST,M3.5.0,M10.5.0/3).

This indeed would be very useful. Due to space restraints, this functionality most likely might be mip installable as an optional module. To be compatible with big Python, os.environ[] might be used to set the TZ environment variable and time.tzset() should be required to have support for one time zone at a time.

I think having a MicroPython specific time zone data base may be a lot of work. The database has to be updated weekly or even more frequently. It's probably better to have everyone lookup or request the needed POSIX string. Updating time zone rules via mip install Continent/City) may be interesting, but need not be part of MicroPython for now.

Just to illustate: In some countries, time zone rules have been changed on short notice (a week before the transition). This must be handled as OTA emergency patch. There are many ways available today to support OTA updates, no need to do something time zone specific.

[projectgus]

One way to avoid this little design problem is to provide a callback that is called each time time.localtime() is invoked, and that returns (zone name, time offset, DST indicator). I believe the overhead of this call is minimal.

In summary: it might be better to process the time zone rule each time time.localtime() gets called instead of setting time zone variables.

Yes, this is more or less how I imagined it would work as well. If the UTC timestamp of the next timezone offset change is stored somewhere then comparing the current time against it is a lightweight check for whether or not the offset needs to be re-calculated. (localtime always resolves in the current timezone of "now", regardless of what timestamp is passed in the argument.) (EDIT: not the case at least in CPython, see two replies down.)

I think having a MicroPython specific time zone data base may be a lot of work.

Fully agree, that is a bridge too far.

[cwalther]

To use that, the programmer will have to write a task that wakes up when a transition between DST and non-DST occurs, and set the new variables.

Apart from the other points you mention, this is not correct to begin with. What DST offset to use on a particularlocaltime(t) call does not depend on whether DST is in effect now, but on whether it was/is/will be in effect at time t.

[bixb922]

@cwalther, you are completely right. What I tried to explain in my previous post is that writing a task to change the time offset by setting a variable is not desirable. I'm sorry if could not explain this well. You added yet another good argument against that design.

I had understood (perhaps wrongly) that the proposal of the OP was about timestamps, in the sense of converting "now" to local time. I think time.localtime() (i.e. getting "now" in local time) is by far the most common and needed use case.

You are also right that time.localtime(t)needs more work. Since the time zone rules change in time, this requires to have a history of past and future time zone rules. In some countries time zone rules have changed every 2 to 3 years. So this could mean having, for example, 5 POSIX time zone strings for the last 10 years, each rule with the time range it was valid, plus a rule for a time zone rule change in the near future. This means additional complexity that has to be weighed against usefulness.

[projectgus]

What DST offset to use on a particular localtime(t) call does not depend on whether DST is in effect now, but on whether it was/is/will be in effect at time t.

I'd convinced myself this was otherwise (I found the POSIX description ambiguous), but of course you're right @cwalther. In CPython, the tm_gmtoff, tm_isdst, tm_zone fields reflect the local timezone at time t.

I had understood (perhaps wrongly) that the proposal of the OP was about timestamps, in the sense of converting "now" to local time. I think time.localtime() (i.e. getting "now" in local time) is by far the most common and needed use case.

You're totally correct that this was the motivation for looking into this, as it's by far the most common use case. However for any change in MicroPython, especially if it's an API that also exists in CPython, then we have to think through the other implications as well.

[scruss]

Talking of other implications: in today's Melbourne MicroPython Meetup, Damien noted that with the recent addition of SSL/TLS support to asyncio, it's important to have your MicroPython board set to the correct time. An SSL connection will fail if the system's clock is wrong. It was suggested to use mpremote rtc --set to correct this, which sets the RTC to local time.

[viji2ks]

guys here first most impordent is to connect board to wifi n internet using network commands then this is simple micropython code to see and sync ur current boad to network time Njoy
plz save below as getNow.py and rest how to excute see the code below last line

import time
import ntptime
try:
ntptime.settime()
except:
raise Exception("npttime.settime() failed. No network connection.")

def getNow(UTC_OFFSET=+5.5):
local_time = int(UTC_OFFSET * 3600)
return time.gmtime(time.time() + local_time)

if name == "main":
print(getNow())

"""
getNow.py
viji2ks@gmail.com
2022-12-12

Usage (assuming a it is connected to wifi with internet network connection):

from getNow import getNow
print(getNow(UTC_OFFSET=-5))
(2022, 12, 12, 10, 34, 34, 0, 346)

"""

[createdbyjurand]

If the problem is that there is no marker for the time zone, you could parse datetime() to ISO UTC marking it with "zulu" GMT timezone and all other systems should be able to convert it to local time.

time.time() is unfortunately less precise because it lacks milliseconds count.

ntptime.settime()

rtc = machine.RTC()

def dateTimeToISO(dateTime):
    return "{:04d}-{:02d}-{:02d}T{:02d}:{:02d}:{:02d}.{:03d}Z".format(
        dateTime[0],
        dateTime[1],
        dateTime[2],
        dateTime[4],
        dateTime[5],
        dateTime[6],
        dateTime[7] // 1000,
    )

dateTimeToISO(rtc.datetime())