Synchronization service call has undefined behavior

[JD-ETH]

We observe strong non-deterministic behavior when trying to trigger the synchronization. The resulting time delay is much more than expected (~0.7 s instead of ~0.1s).

Hardware:
Linux Ubuntu 18.04, ifm o3x100.

On sensor configuration:
The relevant setting here is SynchronizationActivated, and is set to "false". This conflicts otherwise with the software disable. sync_clocks flag is enabled in the ros launch file.

Problem description
The synchronization seems to give reasonable result when the sensor boots up. however, when for some reason the ifm3d_ros interface is relaunched, or when the syncClocks is called as a service, the time stamp starts to jump around. We examine the delay between system time (ROS time) and sensor time (message time) using the ros diagnostics tool. Videos and small descriptions are attached under the following link.

The issue is posted here as it more closely relates to the underlying ifm3d library then the ros wrapper.

[theseankelly]

Hi @JD-ETH

As you've discovered, the O3X offers device-based time synchronization via an NTP server.

The ifm3d library also offers a secondary 'software' time sync approximation via the ifm3d::Camera::SetCurrentTime API. This API accepts a time epoch (where '-1' means 't_now'), and writes the current host system time to the device via an XML-RPC call. This method is subject to varying degrees of latency as it cannot account for network or device-side latency between when the host time is read to when it's written on the target device.

The syncClocks service in ifm3d-ros is a projection of the ifm3d::Camera::SetCurrentTime API, which explains the nondeterminism you're observing.

If you want to truly time synchronize multiple devices, the only viable solution is using the device-side mechanism with a common NTP server.

[JD-ETH]

@theseankelly thanks for the quick response!

We tested the hardware synchronization first, and couldn't get it to work reliably. I'm describing our procedure here and please let me know if anything's wrong:

1. setup chrony for ntp server on local network at the host.
2. set configuration to use SynchronizationActivated, wait until 5 tries.
   We then observe the sensor syncing status via ifm3d dump. When booting sensor and host at the same time, the synchronization doesn't seem to run and status keep showing server not available.
   If however ifm3d reboot is called, and one wait for several minutes, essentially the result of ifm3d dump shows synchronized status.

We are sure the ntp server is available, and the host has access to internet. Could you give some suggestions?

I will be closing this thread afterwards, as the nondeterminism seems inevitable and shouldn't be viewed as an issue.

[theseankelly]

I'll defer to @graugans on the NTP questions.

[graugans]

As I described in #96 the NTP server needs to be available when the camera is booting up. Otherwise the camera does not sync due to the risk of time jump effects. I wonder why you do have a sync time of several minutes. This is not intended behaviour. I can check this on monday when I am in the office.

[JD-ETH]

I have found a fix to the problem.

Indeed, the syncing fails if the ntp server is unavailable. Which is to be expected on a mobile robot platform by the way, how would your PC boots up earlier than your sensor?

The interesting and unexpected fact of the ifms is however, when rebooting the sensor ifm3d reboot after the NTP server is available, it doesn't attempt or does not immediately attempt to reconnect with the server. Seems like a "number of sync tries" was somehow saved and the syncing was skipped. Sometimes I observe syncing however after a long time after the reboot, which is unreliable for our purposes.

I can confirm however, by deactivating and reactivating the SynchronizationActivated flag the ifms can immediately be synchronized with the server. The command I ended up using was:

ifm3d dump | jq '.ifm3d.Time.SynchronizationActivated="false"' | ifm3d config
ifm3d dump | jq '.ifm3d.Time.SynchronizationActivated="true"' | ifm3d config
For all our ifm sensors.

Feel free to close this MR, as I have found my fix. But there is obviously something unexpected going on, so I will let you decide whether you want to close this thread here. @graugans .