Accessing Windows internal leap second information

[MattStephanson]

My original plan for system_clock/utc_clock conversions was to use a static table of leap seconds. I've since discovered that leap seconds appear to be stored in the registry key HKLM\SYSTEM\CurrentControlSet\Control\LeapSecondInformation, under the value LeapSeconds. After playing with this a little, I have a few questions.

1. There appears to be an additional source or store of data, in addition to the registry value. Consider the following commands, run from an administrator prompt (Warning: if you run this, it will change your system in a way that requires a reboot to completely reset)

C:\WINDOWS\system32>w32tm /leapseconds /add:+2021-11-30T23:59:59 /force
The command completed successfully.

C:\WINDOWS\system32>w32tm /leapseconds /getstatus /verbose
[Leap Seconds]
Enabled: 1 (Local)
Number of Leap Seconds (after June 2018): 1 (Local)
Leap Seconds List (Local):
+2021-11-30T23:59:59


C:\WINDOWS\system32>w32tm /leapseconds /remove:+2021-11-30T23:59:59 /force
The command completed successfully.

C:\WINDOWS\system32>w32tm /leapseconds /getstatus /verbose
[Leap Seconds]
Enabled: 1 (Local)
Leap Seconds State is inconsistent with the Settings.
Leap Seconds Runtime State: Enabled: 1  Leap Second Count: 1
Leap Seconds Settings State: Enabled: 1 Leap Second Count: 0
Either a reboot is needed to sync up the state or the settings are invalid.
See verbose output for details.
Leap Seconds List (Local):
        Runtime State   Settings State (X == invalid entry)
+2021-11-30T23:59:59

After adding, the registry value contains binary data "e507 0b00 1e00 1700 0000 0000", corresponding to 2021, 11, 30, 23, 0, 0. After removing, the registry value is empty, yet the "Runtime State" remembers the data. The first questions are, does this "Runtime State" matter? If/when a real leap second is added, are we sure it will be in this registry value?

2. I think I can read the data correctly, but it would be nice if someone at MS can confirm.

c:\leapsec>type leapsec.cpp
#include "Windows.h"
#include <cassert>
#include <cstdint>
#include <cstdio>
#include <vector>

struct LeapSecondInfo {
    uint16_t year;
    uint16_t month;
    uint16_t day;
    uint16_t hour;
    uint16_t negative;
    uint16_t unknown;
};
static_assert(sizeof(LeapSecondInfo) == 12, "");
using namespace std;

int main()
{
    constexpr auto reg_key_name = __TEXT("SYSTEM\\CurrentControlSet\\Control\\LeapSecondInformation");
    constexpr auto reg_subkey_name = __TEXT("LeapSeconds");
    HKEY leap_sec_key = 0;
    RegOpenKeyEx(HKEY_LOCAL_MACHINE, reg_key_name, 0, KEY_READ, &leap_sec_key);

    vector<LeapSecondInfo> leap_sec_info;
    DWORD size = 0;
    RegQueryValueEx(leap_sec_key, reg_subkey_name, nullptr, nullptr, nullptr, &size);
    assert(size % 12 == 0);
    leap_sec_info.resize(size / 12);
    RegQueryValueEx(leap_sec_key, reg_subkey_name, nullptr, nullptr, reinterpret_cast<BYTE*>(leap_sec_info.data()), &size);

    puts("Leap second list:");
    for (const auto& ls : leap_sec_info) {
        printf("%c%04d-%02d-%02dT%02d:59:59\n", ls.negative ? '-' : '+', ls.year, ls.month, ls.day, ls.hour);
    }
    return 0;
}

c:\leapsec>cl leapsec.cpp /EHsc advapi32.lib
Microsoft (R) C/C++ Optimizing Compiler Version 19.28.29812 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

leapsec.cpp
Microsoft (R) Incremental Linker Version 14.28.29812.0
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:leapsec.exe
leapsec.obj
advapi32.lib

c:\leapsec>leapsec
Leap second list:
+2021-11-30T23:59:59
+2021-12-31T23:59:59
-2022-01-31T23:59:59

c:\leapsec>w32tm /leapseconds /getstatus /verbose
[Leap Seconds]
Enabled: 1 (Local)
Number of Leap Seconds (after June 2018): 3 (Local)
Leap Seconds List (Local):
+2021-11-30T23:59:59
+2021-12-31T23:59:59
-2022-01-31T23:59:59

c:\leapsec>

3. As far as updating while a program is running, I think the simplest approach is to make use of the tzdb_list, which will anyway require a never-invalidated collection of vector<chrono::leap_second>. It will only be necessary to go to the registry on initialization and when reload_tzdb is called. My understanding is that this can be done in the import library, where new functions are allowed. Does this sound like a reasonable design? If so, could someone please point out an example of a function that goes in the import library that I could study?

[mnatsuhara]

Thanks for looking into this and creating this discussion for more...discussion 😝

I can go and look further into details around using the registry key, but I did find this article online which seems to indicate that we can't get past leap seconds from the registry key, only future ones (looking at the section "Previous Leap Seconds"). Similarly, in the code you provide above it doesn't look like any past leap seconds are reported - only future ones. Looking at time.zone.leap, we'll need to also be able to provide data about past leap seconds, so the registry key may not be a complete solution for us. Were you envisioning using the registry key to do automatic updates to the vector<chrono::leap_second> and store a static table of past leap seconds in addition?

[MattStephanson]

I can go and look further into details around using the registry key

Thanks.

Were you envisioning using the registry key to do automatic updates to the vector<chrono::leap_second> and store a static table of past leap seconds in addition?

Exactly, yes. I think I have the details sorted out, so the PR should be ready Real Soon Now, after I review things and tear out operator<=> (since, umm, the duration and time_point overloads aren't merged).

[mnatsuhara]

Update: I have sent mail to Dan Cuomo (the author of the article I linked above) passing along your two questions.

Regarding question (3): I do note that I believe we are not required to provide updates while running to the leap second data: "While the program is running, the implementation may choose to update the time zone database." So, if the amount of work to implement updates while running is minimal, we should provide that support. If the additional work is substantial, we should strongly consider foregoing the updates for now and recording it as a vNext TODO.

Unfortunately, we are not able to instantiate any vectors in the import lib.

@StephanTLavavej's explanation: "Unfortunately, vector is prohibited from being instantiated in the import lib. The reason is that we have only 2 import libs (release and debug), but users have 5 possible IDL settings (release 0/1, debug 0/1/2), and the import lib is effectively statically linked, so any occurrences of vector or basic_string etc. would be lethal ODR violations. This is why I had to move syncstream.cpp's static map into a satellite DLL."

So the vector<chrono::leap_second> will need to be added to the atomic_wait satellite. This was recently done for #1564's static map, so you should be able to find good examples in that PR. Specifically, these comments should be helpful:

• <syncstream>: implement basic_syncbuf and basic_osyncstream #1564 (comment) (ODR violations)
• <syncstream>: implement basic_syncbuf and basic_osyncstream #1564 (comment) (single copy per STL DLL)
• <syncstream>: implement basic_syncbuf and basic_osyncstream #1564 (comment) (Scary Comment when import lib stuff is added)

(Again, thanks @StephanTLavavej)

[MattStephanson]

I believe we are not required to provide updates while running to the leap second data

True, but it's not much more difficult than reading them once during initialization.

Unfortunately, we are not able to instantiate any vectors in the import lib.

Yeah, I discovered that after some very funky behavior in the tests with certain MT[d] and IDL settings, which surprised me because it looks like there are 5 types of import libraries being built. You'll see when the PR drops, but I settled on storing a static void * in the import lib, which is initialized from a function pointer passed from the user's code into the import library. Since the import lib doesn't manipulate the vector in any way, I don't think there are any IDL problems. The alternative is an inline variable, though that would need an exception from your convention against initializing data that isn't used. It's much simpler, though.

[StephanTLavavej]

it looks like there are 5 types of import libraries being built.

We have 2 DLLs (msvcp140_oss.dll and msvcp140d_oss.dll in out\build\x86\out\bin\i386) and 2 corresponding import LIBs
(msvcprt.lib and msvcprtd.lib in out\build\x86\out\lib\i386) used for /MD and /MDd regardless of _ITERATOR_DEBUG_LEVEL setting. We have 5 static LIBs (libcpmt.lib, libcpmt1.lib, libcpmtd.lib, libcpmtd0.lib, libcpmtd1.lib also in the lib directory) used for /MT and /MTd with every available _ITERATOR_DEBUG_LEVEL setting.

The 5 static LIBs are why we don't have to worry about IDL mismatch when static linking (because the STL's separately compiled code matches the user's code). However, the 2 import LIBs, which are effectively statically linked, means that we have to be very careful about what's "injected" into them.

I settled on storing a static void * in the import lib, which is initialized from a function pointer passed from the user's code into the import library. Since the import lib doesn't manipulate the vector in any way, I don't think there are any IDL problems.

Hmm, that sounds correct (I hesitate to be absolutely confident regarding separately compiled scenarios). This is a novel scheme AFAIK, but I think it works. It's important not just that the import lib doesn't manipulate the vector, but also that this is specifically stored in the import lib and not the DLL. In vNext, we plan to merge all of the satellites and import lib machinery into the DLL, but this void * would not be merge-able (cat_idl0.dll and dog_idl2.dll loaded into the same process with msvcp200d_oss.dll would not be able to share a vector, regardless of void*). Every user DLL needs its own vector with a matching representation.

N4878 [time.zone.db.list]/1 says "The tzdb_list database is a singleton; the unique object of type tzdb_list can be accessed via the get_tzdb_list() function." Uniqueness inherently can't be achieved across multiple user binaries (because tzdb_list contains tzdb contains string/vector regardless of implementation scheme), but the Standard doesn't recognize the existence of DLLs, so anything we do is inherently implementation-defined. Unlike <system_error> where this created significant headaches, the addresses of the tzdb_list and its tzdb elements seem extremely unlikely to matter.

I think we can consider using an inline variable, and investigating how to make the MSVC-internal test accept it; not just for implementation simplicity, but also the possibility of not paying the cost when <chrono> isn't used by the user, or when it is but tzdb_list isn't used (inline variables inherently pay no cost when they aren't included in the former scenario; I am not sure if they can be elided in the latter scenario).

[mnatsuhara]

@MattStephanson we've finally heard back from the team who implemented the Windows leap second registry!

Leap second information has not been exposed as an API or documentation externally. We have only exposed it via w32tm.exe, as shown here as well. This was because there were no customer asks for an API.
Please be aware that leap second data is directly available in shared memory of each process. Using the registry works in most situations, but not every situation – eg: if a leap second is removed from the list or if there is an error that the Kernel validates.

They've pointed us to some code that w32tm uses to deal with leap seconds that should be more robust in dealing with these edge cases that the registry approach will fail to handle correctly. I'm going to look into their suggested approach to see if it's something that will work for us. Given our time constraints, we may need to ship the registry implementation initially. If we can maintain the __std_tzdb_get_reg_leap_seconds / __std_tzdb_delete_reg_leap_seconds / __std_tzdb_registry_leap_info interface, we should be able to come back and swap out the underlying implementation with no bincompat concerns.

[MattStephanson]

That seems to answer what the mysterious "Runtime State" that w32tm reports is. All we need from the STL side is the date, time, and positive/negative flag, so I hope the current interface should be adequate, even if the shared memory data has a different layout.

[mnatsuhara]

Yeah, I am pretty sure we should be able to maintain that interface (since it's pretty straightforward and simple).

[StephanTLavavej]

We got another update from the Windows team - we shouldn't use the undocumented structures in the Process/Thread Environment Block, but they're considering adding a new API to Windows for us to access this information (no promises, but there's a work item OS-32341669 in their database).