List Timezones for user selection

[YarekTyshchenko]

I want to use the zonedb as a source of selecting a timezone, during wireless configuration. The ZoneDB list is so complete, and so very well packed, it seems a waste to create a reduced list to select from.

I'm currently just plugging various libraries together, AutoConnect's Simple.ino example, which already includes a timezone selection list, trying out the idea of streaming stored timezone names out to the browser:

https://github.com/YarekTyshchenko/KJ-8/blob/master/timezone-test/timezone-test.ino

The idea of this project is to create a generic base for various clocks that I'm building, providing them with fully automated operation, without any hardware buttons or anything like that.

[bxparks]

(Just to give more context to anyone else who happens to come along to read this, we are moving the discussion in #51 to here.)

Comments about your timezone-test.ino Sample Code

Your sample code that you posted above is generally on the right path. I have a few comments about your usage of some of the AceTime classes:

• I recommend using the BasicZoneManager or the ExtendedZoneManager instead of reaching directly into the basic::ZoneRegistrar or extended::ZoneRegistrar. The ZoneRegistrar classes are not intended for public consumption, they are internal classes which I don't even document anymore in the Date Time TimeZone User Guide.
• The ZoneManager classes have cleaner API, and they manage the ZoneProcessor cache for you. If you need just a single ZoneProcessor as you have in your current code, use a BasicZoneManager<1>.
• You are probably just playing around with the zonedb::kZoneEtc_UTC, but just as an FYI, the simplest way to get a UTC timezone is to use TimeZone::forUtc() or just the default constructor TimeZone(). This will be far cheaper and faster than using the IANA TZ "Etc/UTC".
• Overall, you seem to be using the low-level implementation class. These classes are not guaranteed to be stable. I recommend using the TimeZone classes instead. Use TimeZone.getZoneId() instead ofBasicZone.getZoneId(). I consider ZoneInfo to be an opaque identifier, the internals are implementation details, which I do not guarantee will remain stable.

General Comments about TimeZones Menus

As I mentioned in #51, the problem of how to display the full list of timezones to the end-user is a difficult problem.

• zonedb::kZoneRegistry has 266 zones,
• zonedb::kZoneAndLinkRegistry has 449 zones,
• zonedbx::kZoneRegistry has 386 zones,
• zonedbx::kZoneAndLinkRegistry has 593 zones.

I think the appropriate solution depends on the type of UI that is presented to the user. For example, the solution on a small OLED display will need to be different than on an HTML web page. My own personal projects have not needed to display the entire timezone collection, so my thoughts on this are preliminary and unpolished, but here they are:

It seems like there are roughly 2 ways to sort such a large collection:

1. We can sort timezones alphabetically. This is probably the easiest way of presenting the timezones to the user, but it may be the least useful. Because timezones next to each other alphabetically are not related to each other. A menu based on this ordering assumes that the user knows the IANA TZ name of a particular zone. Most normal users will not know such things. Fortunately, most IANA TZ zone names are named after cities, so people can probably figure things out eventually. But sometimes people living in a certain city won't know the exact IANA TZ identifier that they are supposed to use.

2. We can sort the timezones by the "standard UTC offset". For example, (-08:00 America/Los_Angeles, -07:00 America/Denver, -06:00 America/Chicago, -05:00 America/New_York).

   a) It seems like within that sorting, we also want to group the timezones by regions, so that they appear together. Not sure how easy that will be, since in the IANA TZ DB, the "regions" are quite large, e.g. "America", and "Europe", and "Asia".
   b) The second problem that I can see is that the "standard UTC offset" of a particular timezone is not a constant of time. Some time zones have changed their standard UTC offset in the past, so the value that probably makes sense is the most recent "standard UTC offset".

In terms of implementation, a list of timezones ordered in some way for the purposes of UI will require another array:

• 4 bytes for the zoneId of each timezone
• the "standard UTC offset" for all modern timezones can be captured in 1 byte (using 15-minute increments)

So that's about 1300 bytes for the 266 zones in the zonedb dataset, and about 2000 bytes for the 386 zones in the zonedbx dataset. For a 32-bit processor with at least 128kB flash, these numbers are not a big deal. But for 8-bit processor, the additional flash memory consumption will be significant.

In terms of code structure, I think information related to how the time zones are presented to the user should live somewhat outside of the core AceTime library. Maybe in another C++ namespace. Maybe even in another library that is layered on top of AceTime.

Anyway, these are some of the ideas bouncing around in my mind for a while. Hope this helps.

[YarekTyshchenko]

Thanks! That helps immensely.

On my use of the underlying internal classes, I think at first I didn't really understand the purpose of zone manager, and its cache parameter. It doesn't make sense to me to allocate that memory for something that I can compute again from the ZoneId. I'll take a look at the docs again to see if I can think of a better way to phrase this.

the simplest way to get a UTC timezone is to use TimeZone::forUtc()
But I need the UTC's ZoneId so a zone is preselected in the UI.
I could handle it as a special case, but I find that always setting a valid zone is cleaner.

On sorting, its an interesting problem. I can get away with sorting the list in Javascript, and don't mind sorting alphabetically, though I see your point. I feel like users have enough exposure to timezone by now to know their closest one, but your points are still valid, this isn't optimal way. Anyway storing the order takes up too much space, but it might be possible to trade speed for storage space by brute forcing the order, looping over the registry n*n style. It would allow you to ship various ordering options at no extra flash cost, even addressing the problem of computing the order based on the current time.

[bxparks]

Hi,
Just following up with some observations:

I scanned your code at https://github.com/YarekTyshchenko/KJ-8/blob/master/timezone-test/timezone-test.ino. The AceTime related code looks much better.

• Using a BasicZoneManager<1> to get a TimeZone through createForZoneId() looks right.
• The zonedb::kZoneEtc_UTC.zoneId can be replaced with zonedb::kZoneIdEtc_UTC which is a compile time constant.
• Lines 132-135: You don't have to write the ZonedDateTime zdt to a PrintStr<64>, then print out its getCStr() to Serial. You can print directly to Serial like this: zdt.printTo(Serial).
• Lines 178-190: The for-loop to extract the zoneName through a BasicZone(zoneInfo) looks great. If you use a BasicZone transiently like you do here, you can construct the object on-the-fly, like this BasicZone(zonedb::kZoneRegistry[n]).printNameTo(printer). The C++ compiler will completely optimize away the BasicZone instance since it's just a thin-wrapper around a ZoneInfo* pointer.
• Lines 212-214: Storing the zoneId directly into EEPROM is fine. I created the TimeZone::toTimeZoneData() method to extract a TimeZoneData object which is meant to be a more general way of doing persistence. You then call ZoneManager::createForTimeZoneData().
• One issue with storing the zoneId directly into EEPROM is, how do you know if the 32-bit data that you get out was actually a valid zoneId that was stored by this program. Any random program could have written those 4 bytes. To handle this problem, I created the CrcEeprom class (https://github.com/bxparks/AceUtils/blob/develop/src/crc_eeprom/README.md). It stores the CRC32 checksum of the data, along with a 32-bit contextId that quasi-uniquely identifies the application that wrote the data. Take a look if you run into this data integrity problem.
  • (I've been thinking about extracting CrcEeprom into a separate Arduino library, as part of a larger set of EEPROM related stuff, so the location of this code might be somewhat unstable.)
• Lines 85-99: Here you create this huge HTML string, then replace the {{DateTime}} with the current ZonedDateTime string. This probably works fine. However, I always worry about heap fragmentation when manipulating large strings using Arduino String class. (I think the ESP8266 String class has better optimizations, but it's still a worry.)
  • If you want to avoid this problem, take a look at my ace_common::PrintStrN(uint16_t) class.
  • It is a variant of the PrintStr<N> template class that you are already using.
  • Instead of allocating the buffer on the stack, PrintStrN allocates the buffer on the heap (Your string seems to be about 500 bytes, so PrintStrN printStrN(1024) should do it).
  • You can print the first part of your long string (up to the {{DateTime}}) into the printStrN. Then do a ZonedDateTime.printTo(printStrN). Then print the second part of your long string into the printStrN. Then do a Server.send(200, printStrN.getCstr()).
  • When printStrN goes out of scope at the end of the function, the destructor gets called, freeing up the buffer allocated on the heap. There will be no heap fragmentation at all, as long as the program makes no other heap calls between the creation of PrintStrN() and the end of the function.
  • (Ok, this is probably way too much detail than you wanted...)

Some thoughts on the sorting problem:

• I was thinking that I could allow the library user to define a custom sorting key. Then the end-user of the library could sort the ZoneRegistries in anyway they want. The index into the registry is a uint16_t, 2 bytes. So the library user would need an array of 266 items of 2 bytes each, to sort the zonedb database used by BasicZoneManager. That's only about 500 bytes of RAM.
• I would be embarrassed to provide a O(N^2) sorting algorithm like bubble sort. But I think something like a Shell sort or Quick sort could probably be implemented without too much flash or static memory consumption.

[YarekTyshchenko]

Thank you again for looking. I've implemented the changes you suggested. I'll be shifting the date string out with SPI, so all the printing will be removed. Same with the massive HTML page, just keeping it around for debugging purposes.

EEPROM issue is interesting. A bad read will reset the timezone to UTC, which isn't too bad.

On the issue of sorting, N^2 sort is the only one I can think of that costs no extra flash space, or RAM to sort (The result is being written out directly to network). Now I'm wondering whether the sort order can be pre-processed at compile time, so that the zone registry comes out in your desired sorted order.
On the other hand, I don't think this is a serious enough issue for me, the hardware this is going on actually has quite a lot more RAM than I'm using.

[bxparks]

Just want to mention that I released v1.9.0 yesterday. This version provides the ability to sort zones. Two implementations are provided, implementing sorting in 2 different ways:

1. by zone name (ZoneSorterByName)
2. by standard UTC offset, then by zone name (ZoneSorterByOffsetAndName)

These classes rely on the new AceSorting library. I chose to use the Shell Sort algorithm for these classes. It has a complexity of about O(N^1.5), which seems slower than the O(N log(N)) of QuickSort. But for N <~ 600, where 600 is the maximum number of zones in the TZDB, the ShellSort algorithm is faster than QuickSort, and consumes no additional stack space, whereas QuickSort is a recursive algorithm so consumes additional stack space.

The ZoneSorter classes seem to consume between 200-600 bytes of program flash, depending on the microcontroller architecture.

Please note that I made several breaking changes from v1.7.0 (when this discussion started) and the current v1.9.0 version. See the CHANGES.md file, the README.md, and the MIGRATING.md guide. The AceTime library is now split into 2 user-facing libraries: AceTime and AceTimeClock.

[bxparks]

hi.. i also got here trying to find a way to list all the zones for a user to select from because i'm also making clock that various calculations, and it requires both the zone offset and possibly dst. i'm trying to use tcmenu framework to create menu of this.

i was wondering if it's possible to place all of them in some sort of Map list collection , where the key is the continent and the value is a list of the time zones of tha continenet similar to what you chose in some linux installations

Hi, thanks for the posting the photo. It's good to know that someone other than me is using AceTime. :-) Which OLED display is that? I recognize the second one in the background is the usual SSD1306 128x64.

The problem of timezone selection is still an open problem in my mind. Adding the ability to sort time zones was a step forward, but did not solve the problem to my satisfaction.

In my mind, there are at least 2 different scopes to this problem:

1. User selection from a small subset of time zones.
2. User selection of time zones among all ~600 time zones.

If the application supports only a small number of time zones, I have found that manually listing those zones, and supplying my own list of display name for each time zone is often the fastest solution.

But finding a reasonable UI to allow the user to select from the entire list of ~600 time zones is more difficult. Part of the problem is that the organization and presentation of the time zones depend on the user-interface of the device. For example, if the user was interacting with a web page generated on an ESP32 or ESP8266 controller, with access to a keyboard and browser, then some sort of type-ahead filtering in a drop-down list would probably work. But if the user input consisted of a small OLED display and push buttons, then a hierarchical menu might be the only reasonable solution. But if the display was too narrow, and could only display 10-15 characters, then the hierarchical menu would probably need to be multiple pages or screens. There are also varying memory constraints which are difficult to account for: 8-bit microcontrollers have far smaller flash memory than 32-bit microcontrollers. Ideally, AceTime would be able to support both in an efficient way.

In additional to the UI/UX issues mentioned above, I think there is a fundamental usability problem with the way the TZDB time zones are organized. The time zones in the IANA TZDB files are organized in the form of {continent}/{city}, where the country is explicitly excluded for many reasons:

1. city names are more stable than country names,
2. cities often change countries due to redrawn borders after wars and such,
3. the country of a city can be in dispute due to disputed borders.

But in reality, most people still think of cities in a hierarchical manner, with cities belonging to countries, and countries belong to continents. The TZDB provides a file named zone1970.tab which seems to organize the time zones into countries. And I believe some Unix distributions (BSD?) use this file to organize their time zone picker. But I find this file to be problematic, because it places some time zones in certain countries, without regard to their political boundaries that most people are familiar with. For example, it says that Bahamas (code BS) uses America/Toronto, simply because the Bahamas timezone happened to have the same rules as Toronto after 1970. I believe it would be more correct to list the Bahamas as using America/Nassau, which is a link to America/Toronto.

Two projects that people on various email lists mention often are the CLDR and ICU. The problem I have with those projects is that they are so large, I cannot understand how these projects are organized. And their client libraries and resources seem to assume desktop-class environments with lots of CPU and memory, instead of constrained microcontroller environments. (E.g. I'm never going to load XML documents into microcontrollers. I will probably avoid Unicode characters as long as I can on microcontrollers.)

So... to summarize this long reply, I have been thinking creating my own version of zone1970.tab, organized as a table of iso_country and timezone, like this:

CA America/Toronto
BS America/Nassau # not America/Toronto as listed zone1970.tab
US America/Los_Angeles
US America/Denver

I can reuse the ISO 3166 countries provided by the TZDB database.

I think I can create a reasonable mapping of continent to ISO country.

I probably wouldn't use a C++ map<> since 8-bit microcontrollers don't have enough memory, but a sorted array of these things can be easily accessed using binary search functions. Even a linear search might be reasonable for some of the smaller lookup tables.

Anyway, this is a kind of a brain dump of where I am headed in the near future.

I notice I had a previous, somewhat related, discussion here with @rvdbreemen here: #80

[emaayan]

oh i gave up on arduino (8 - bit controller ) for this type of application,
i mean i wrote a basic clock that only needed dates, but just displaying the dates took around 1500 bytes
so i switched to esp32 for the thing

the timezone file on geonames seems to be listing Nassau under BS
geonames_zones

i was thinking of using geonames files to compile my own list as well

[bxparks]

@emaayan : Thanks for the reference to geonames_zones. It very close that what I wanted. I tried to figure out how they generate this file, how it is maintained, and what happens when the IANA TZDB is updated with new timezones. But I wasn't able to find that information (maybe I didn't look hard enough).

I didn't want to depend on a project that I didn't understand. So I spent some time over the past 2 weeks on a new repo called github.com/bxparks/tzplus. The end product is a file named country_timezones.txt which organizes the timezones into (region, country, timezone) triplets. This should allow the creation of hierarchical timezone selectors for the end users.

I defined 12 regions (continents and oceans) for the world, as listed in this file: regions.txt. I use the ISO 3166 country codes defined by iso3166_long.txt and iso3166_short.txt. I manually processed each of the ~600 timezones in the TZDB, categorizing each Zone and Link entry, so that I can filter out duplicates and obsolete entries, bringing the total down to about ~450.

This dataset is hot off the press, I just finished releasing the first version. I plan to create user interfaces on a few microcontrollers in the coming days and weeks to consume this data, to verify that the timezones are organized a useful way. If you have any comments or question, you can use the tzplus Discussions page.

[emaayan]

that sounds interesting, i myself am creating something less complex, but in json, and splitting the timzone files into counteries, so i won't have a big issue into loading them into memory .i may look into that.
from what i understand geonames it's maintained by users https://www.geonames.org/manual.html wiki style

[bxparks]

Ah, thanks for the link to https://www.geonames.org/manual.html. One thing that was important to me was the ability to programmatically detect data consistency issues when new versions of the TZDB are released. I wrote my own Python code to do that.

Another feature that I plan to research in the future for tzplus: mapping the IATA airport codes to each timezone. This would allow the microcontroller to display or accept LAX instead of America/Los_Angeles. Sometimes, even just the city name Los_Angeles is too long when you have just a few digits of a seven-segment LED module.

Not to dissuade from your own work, because it will still be useful for you, I plan to embedded my tzplus data (the region and ISO country information) directly into the AceTime database in the future, so that user-interfaces can be built on top of that.