SNTP using fractional seconds?

[tomtobback]

Hi, i am trying to synchronise multiple ESP32s using the native time functions as in the ESP32>Time>SimpleTime example.
However, the transition to the next second does not occur at the same instant on my different ESP32s (up to several 100 of milliseconds out of sync). The only explanation i can think of is that the underlying SNTP functions do not take the fractional seconds of the NTP UDP packet into account. I have been trying to find the source code, with no success. Can anybody help me look into this.
Or any suggestions how to synchronise devices on the same network to millisecond precision.

[MarkusAD]

Correct it doesn't use the fractional part of ntp timestamps, its built into the precompiled library liblwip.a and defined by the macros SNTP_SET_SYSTEM_TIME(sec) and SNTP_GET_SYSTEM_TIME(sec). See the comments in tools/sdk/include/lwip/lwip/apps/sntp_opts.h. You can try compiling liblwip.a from esp-idf, a test sketch will do, and modify sntp_opts.h (the one local to the idf source tree) to use the _US(sec, us) versions of the macros, then copy the compiled lib to the arduino tree to get the changes to take effect. It can be tricky because this repo only periodically syncs with esp-idf. so you'll have to find a compatible idf commit/version that works with the version of this repo that you're using. Save a copy of the original liblwip.a before overwriting so you can revert if it bombs. I haven't tried using the fractional timestamps, but I've used liblwip from idf successfully in arduino-esp32. No idea if it'll work but certainly worth an hour or two of frustration if it ends up a success.

[MarkusAD]

@tomtobback its even easier if you install this repo as an esp-idf component and build your project within that. You can change settings in sntp_opts.h and liblwip gets compiled along with the project build.

[MarkusAD]

Looking further, lwip's sntp and settimeofday() should already be using the microseconds portion of ntp timestamps. Replacing liblwip.a with a newer one from idf didnt gain me any better precision, it still had several hundreds of milliseconds offset. Probably at least some of that comes with propagation delay over wifi and/or latencies in the wifi/tcpip stack. @tomtobback if you want to dig around the lwip source the relevant files in idf are:
components/lwip/lwip/src/apps/sntp/sntp.c
components/lwip/port/esp32/include/lwipopts.h
components/lwip/lwip/src/include/lwip/apps/sntp_opts.h

[tomtobback]

hi @markyad thanks so much for pointing me to the source file sntp_opts.h. there i found this:

/** SNTP macro to change system time in seconds
 * Define SNTP_SET_SYSTEM_TIME_US(sec, us) to set the time in microseconds instead of this one
 * if you need the additional precision.
 */
#if !defined SNTP_SET_SYSTEM_TIME || defined __DOXYGEN__
#define SNTP_SET_SYSTEM_TIME(sec)   LWIP_UNUSED_ARG(sec)
#endif

i changed that #define to SNTP_SET_SYSTEM_TIME_US(sec, us), saved the file, and compiled my sketch again in the Arduino IDE. it complained about SNTP_SET_SYSTEM_TIME_US already being defined in /tools/sdk/include/lwip/lwipopts.h and indeed i found it there at the bottom:

#define SNTP_SET_SYSTEM_TIME_US(sec, us)  \
    do { \
        struct timeval tv = { .tv_sec = sec, .tv_usec = us }; \
        settimeofday(&tv, NULL); \
    } while (0);

#define SNTP_GET_SYSTEM_TIME(sec, us) \
    do { \
        struct timeval tv = { .tv_sec = 0, .tv_usec = 0 }; \
        gettimeofday(&tv, NULL); \
        (sec) = tv.tv_sec;  \
        (us) = tv.tv_usec; \
    } while (0);

So i commented out the 3 above lines in the sntp_opts.h and my sketch compiled without errors. Now it seems to be accurate! I'm not sure why this is not the default, it does not seem to take any extra resources to use the accurate time?
Thanks again.

[tomtobback]

Hi @markyad i had not seen your latest reply before i posted the above. I can't really figure out from the source code whether it's using the microseconds functions or not. From my experiments so far, i get more consistent success when i comment out the 3 lines as in my previous post.

[MarkusAD]

I'm not sure how that worked, because just changing settings in sntp_opts.h is not enough. For example if you lower the SNTP_UPDATE_DELAY to something lower than the default 1 hour interval, it wont take effect until you compile that into liblwip.a and compile/flash the sketch using the new library. I have 3 of them running now with various builds and settings, but my wifi is so busy its difficult to see if any one is better than another.

[tomtobback]

hi @markyad just for your info: i don't think i'm seeing the compiling behaviour that you describe. I'm on Ubuntu, using the Arduino IDE 1.8.2 and when i modify the sntp_opts.h it seems to be compiling each time because if i change something (as my first attempt above), i see a compilation warning in the IDE. That means that file is compiled every single time, or am missing something?
I have changed the SNTP_UPDATE_DELAY to 20 seconds; not sure if it is doing that, but my 2 nodes (on the same wifi network) are synchronised now, as far as i can see (I have a screen on each that blinks on the second). I'm not worried about them being accurate relative to the absolute time, they just need to be in sync.

[MarkusAD]

I think part of the ambiguity is that most of the sntp routines reside in the precompiled library liblwip.a but the routine of interest (SNTP_SET_SYSTEM_TIME_US) is a macro that resides in lwipopts.h. Apparently commenting out the define for the truncating version in sntp_opts.h is enough to cause it to use the more accurate microseconds version without having to mess with the library.

The esp32 I tested by commenting out the 3 lines remains very close to my ntp servers and a network gps after about 7 hours with sync = 1hour... it was never sync so closely as it is now.

[tomtobback]

i'm closing this as it seems to work by commenting out of those 3 lines. imho it would make more sense to have this higher precision as default.