PPS source have timeout since update on March 17th

[Etifloyd]

Describe the bug

For more than one year I'm running chrony with a Waveshare NEO-M8T GPS/PPS module to have a stratum 1 NTP server. It was running correctly until I let Raspberry PI OS make a big update on March 17th. Since chrony is no more able to synchronize with the PPS signal.
Running the following command: "sudo ppstest /dev/pps0" give time-out errors.
To be sure that the GPS module is still ok, I restore a SD card with an image of december and it runs correctly.

Steps to reproduce the behaviour

Create an SD card with the last version of Raspberry Pi imager and apply all updates.
Update the /boot/config.txt (see attachement).
Install the following packages: chrony, gpsd, gpsd-clients, gpsmon, pps-tools.
Update the following files: /etc/chrony/chrony.conf (see attachement) and /etc/default/gpsd (see attachement).
Stop and disable the systemd-timesyncd.service.
Enable chrony and gpsd services.
Reboot the raspberry.
Once rebooted the "sudo ppstest /dev/pps0" command shall give timeout errors.
And the "chronyc sources" command shall show 0 reach for the PPS0 source.
RPi_config.zip

Device (s)

Raspberry Pi 4 Mod. B

System

Raspberry Pi reference 2022-04-04
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 226b479f8d32919c9fe36dd5b4c20c02682f8180, stage4

Mar 17 2023 10:50:39
Copyright (c) 2012 Broadcom
version 82f3750a65fadae9a38077e3c2e217ad158c8d54 (clean) (release) (start)

Linux ntpserver 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 GNU/Linux

Logs

Here are my update logs from Feb. 14th when everything was running correctly and Mar. 17th where problem starts:
UpdateLog.zip

Additional context

No response

[pelwell]

Is that a 32-bit image you are using? And was it previously running a 32-bit kernel? If so, and if you still have (or can easily recreate) the failing image, try adding arm_64bit=0 to config.txt.

[Etifloyd]

Yes, it a 32 bits image that I'm using. I use the default image proposed in the imager: 32 bits with desktop.
I add arm_64bit=0 in the config.txt file and it did the trick, now it's working correctly!
Thank you for the quick answer and the trick!
Kind regards

[pelwell]

Cool - consider that a workaround for now, at least until we understand the problem. I've a horrible feeling its a problem with the ioctl() interface from the 32-bit userland to the 64-bit kernel...

[pelwell]

My configuration is:

• 64-bit 6.1 kernel
• 32-bit userspace, including pps-tools
• Fake PPS source (something waggling GPIO18 roughly every second)

If I launch ppstest without the fake PPS source I see:

pi@raspberrypi:~$ sudo ppstest /dev/pps0
trying PPS source "/dev/pps0"
found PPS source "/dev/pps0"
ok, found 1 source(s), now start fetching data...
[ nothing happens ]
^C

With the fake source:

pi@raspberrypi:~$ sudo ppstest /dev/pps0
trying PPS source "/dev/pps0"
found PPS source "/dev/pps0"
ok, found 1 source(s), now start fetching data...
source 0 - assert 1681896676.611779347, sequence: 419 - clear  0.000000000, sequence: 0
source 0 - assert 1681896677.633878166, sequence: 420 - clear  0.000000000, sequence: 0
source 0 - assert 1681896678.655616653, sequence: 421 - clear  0.000000000, sequence: 0
source 0 - assert 1681896679.677251561, sequence: 422 - clear  0.000000000, sequence: 0
...

I've not seen anything like a timeout.

Can you run the following test?

1. Disconnect your GPS module, and add a link/patch cable between GPIOs 17 and 18 (pins 11 and 12)
2. I'm assuming you already have dtoverlay=pps-gpio in config.txt, and that the default GPIO of 18 is being used.
3. sudo apt install gpiod pps-tools
4. In a shell run the following:

   $ while true; do gpioset gpiochip0 17=1; gpioset gpiochip0 17=0; sleep 1; done &
   $ sudo ppstest /dev/pps0
   

   What output do you get?

[pelwell]

5. If you remove the link or kill the background task (probably kill %1, or fg and Ctrl-C), does the output just stop or do you get a timeout?

[Etifloyd]

So to be back in the faulty situation I comment the "arm_64bit=0" line in the config.txt file, I shutdown the pi, remove the gps module, put a jumper between pin 11 and 12 and reboot the pi
I launch the while loop and the pps test output is the following:
trying PPS source "/dev/pps0"
found PPS source "/dev/pps0"
ok, found 1 source(s), now start fetching data...
source 0 - assert 1681899996.974862821, sequence: 11 - clear 0.000000000, sequence: 0
source 0 - assert 1681899998.012749800, sequence: 12 - clear 0.000000000, sequence: 0
source 0 - assert 1681899999.051649106, sequence: 13 - clear 0.000000000, sequence: 0
source 0 - assert 1681900000.092132863, sequence: 14 - clear 0.000000000, sequence: 0

When I remove the jumper, the stream simply stops, no timeout.

[pelwell]

Interesting. What is the minimum you need to do to get ppstest to fail with a timeout? Obviously you won't be running the while loop, but what else?

[Etifloyd]

I put the gps module back on the pi and now I have timeout.
The two other stuff I'm running on this Pi is NUT to monitor an UPS and Cacti to monitor network devices through SNMP.
Both packages installed with apt-get.
Globally, my raspberry do almost nothing.
It's look like the timeout appear when the gps module is present. May it be releated to GPSD?

[Etifloyd]

I also try the while loop with a sleep of 0.9s and no timeout

[Etifloyd]

I try to reproduce the problem with a Pi 3B+ which also have a 64 bits core, but no timeout for the moment. I use jumper wires to reroute TX and PPS signal from the Pi4 to the Pi3B+. I have no timeout and chrony does sync correctly.
Maybe I run bullseye from too long time on my Pi4 and something went different over all the regular updates regarding a freshly created image. May it be a reason?

[pelwell]

Could you install a fresh Raspberry Pi OS on spare SD card, then do just enough to run ppstest with the HAT attached?

[Etifloyd]

I just create a new SD card with default Raspberry Pi OS 32 bits of Feb. the 21st with imager version 1.7.2.
I apply all the updates and enable the UART, SSH and VNC from Raspi config.
After reboot, I update the config.txt file to disable wifi and bt and enable the pps on pin 18.
After the reboot, I install pps-tools, gpsd and gpsd-clients.
In /etc/default/gpsd file I set the device as following:
DEVICES="/dev/serial0 /dev/pps0"
I restart the gpsd service and launch the gpsmon command. Both NMEA and PPS data are visible.
Then I launch the sudo ppstest /dev/pps0 command, and ... I have timeout

[pelwell]

Thanks - that's useful (but annoying).

Does stopping the gpsmon service prevent the timeouts? I won't be surprised if you don't get any pulses back.

[Etifloyd]

No, I just stop and disable both gpsd.service and gpsd.socket, no difference. Timeout is still present.