CM4 is missing IEEE1588-2008 support through BCM54210PE

[tschiemer]

Is this the right place for my bug report?
Yes: I asked on the forum, I asked the raspberry foundation through the contact form but didn't get any response - it is very much kernel related as is explained in the forum post.

Describe the bug
The CM4 has an undocumented inofficial PHY (BCM54210PE) that supports hardware timestamping (and a hardware clock, actually??), but there seems to be no driver supporting these features.

So CM4 does actually not provide hardware based IEEE1588-2008 support as opposed to what the raspberry pi foundation actually communicates in the CM4 datasheet.

To reproduce
sudo ethtool -T eth0 only shows software timestamping capabilities.

Expected behaviour
Depends - the raspberry pi foundation never communicated what IEEE1588-2008 features are supported such that is is unclear what exactly can be expected; also the PHY documentation is not available such that possibilities (ie optimal expected behaviour) can not be described.

Expected (according to https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/s1-using_ptp):

• eth support for:
  • SOF_TIMESTAMPING_TX_HARDWARE
  • SOF_TIMESTAMPING_RX_HARDWARE
  • (possibly) SOF_TIMESTAMPING_RAW_HARDWARE
• shows PTP Hardware Clock (because BCM54210PE has on-chip clock)
• Hardware Transmit Timestamp Modes ??
• Hardware Receive Filter Modes ??

The CM4 datasheet lists SYNC_IN, SYNC_OUT pins ment for IEEE1588-2008 support but has not further specified this.
It actually is a question wether these pins are needed in the first place - because the PHY would seem to have a hardware-clock itself.
Further it must be clear wether the PHY's internal clock (if any) is routed to a CM4 pin in any form and if not this should be fixed in future revisions because custom boards for the CM4 might require tight synchronization to a hardware clock.

Actual behaviour
sudo ethtool -T eth0 only shows software timestamping capabilities.
There is no (documented) synchronizable IEEE1588-2008 clock signal going from the CM4.

System
Raspberry Pi Compute Module 4

Raspberry Pi OS Lite, January 11th 2021, Kernel version: 5.4
Ubuntu Server 20.04.2 LTS,

Logs
none

Additional context
BCM54210PE documentation is required to implement to see actual features provided and configuration options - IEEE802.3-2018 seems to define said capabilities etc.
Possibly also required are CM4 schematics and BCM2711 documentation.
Also see IEEE802.3-2018 Section 90. Ethernet support for time synchronization protocols

I can try to take care of the implementation, but I would like to have the mentioned documents.

[pelwell]

@ghollingworth?

[tschiemer]

Just compiling a kernel with PTP timestamping support enabled (by default not). Will see if this changes anything.

I'm new to kernel topics, so I missed the obvious. Will report back

[tschiemer]

Built a new kernel according to docs and changed the following settings (through make menuconfig):

CONFIG_NETWORK_PHY_TIMESTAMPING=y

#
# PTP clock support
#
CONFIG_PTP_1588_CLOCK=m
# CONFIG_DP83640_PHY is not set
# CONFIG_PTP_1588_CLOCK_INES is not set
# CONFIG_PTP_1588_CLOCK_IDT82P33 is not set
# CONFIG_PTP_1588_CLOCK_IDTCM is not set
# end of PTP clock support

Linux cm4-1 5.10.14-v7l-MY_CUSTOM_KERNEL+ #1 SMP Tue Feb 16 14:05:19 GMT 2021 armv7l GNU/Linux

No change observed, ie

pi@cm4-1:~/linux $ ethtool -T eth0
Time stamping parameters for eth0:
Capabilities:
	software-transmit     (SOF_TIMESTAMPING_TX_SOFTWARE)
	software-receive      (SOF_TIMESTAMPING_RX_SOFTWARE)
	software-system-clock (SOF_TIMESTAMPING_SOFTWARE)
PTP Hardware Clock: none
Hardware Transmit Timestamp Modes: none
Hardware Receive Filter Modes: none

pi@cm4-1:~ $ sudo lshw -class network
  *-network:0
       description: Ethernet interface
       physical id: 2
       logical name: eth0
       serial: dc:a6:32:XX:XX:XX
       size: 1Gbit/s
       capacity: 1Gbit/s
       capabilities: ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=bcmgenet driverversion=5.10.14-v7l-MY_CUSTOM_KERNEL+ duplex=full ip=192.168.2.197 link=yes multicast=yes port=MII speed=1Gbit/s
...

I figure because there is no particular PTP hardware clock (driver) there's no recognized PTP clock (...) - but this would depend on wether BCM54210PE actually would have a hardware clock (?)

bcmgenet does not seem to use phy_devices struct mii_timestamper *mii_ts;

so enabling timestamping has no effect...

For reference: https://github.com/raspberrypi/linux/blob/c9226080e513181ffb3909a905e9c23b8a6e8f62/include/linux/mii_timestamper.h

[ghollingworth]

We're currently discussing getting support for writing a driver from Broadcom, although we have access to the datasheet, to me it's pretty meaningless!

Unfortunately, so is the Linux interface to the clock...

But we'll endeavour to get there

[tschiemer]

Ok, thanks!

Hoping to be hearing more from you!

Just to say, as for the docs, I wouldn't mind signing an NDA to have a look to see if can make sense of it..

[wowczarek]

Damn you Broadcom and your closed datasheets! Timestamping at the PHY level is the best one can get, but not if unimplemented in software...

I also do hope those SYNC_IN and SYNC_OUT pins are actual event inputs/outputs for external sync and were preserved on the I/O board and not repurposed for something else. I've seen countless SBCs where this potential was wasted.

So, @ghollingworth, aside from a PTP driver eventually being developed - do we know if the two pins mentioned (pins 16 and 18 on the CM4) are presented on the I/O board?

Thanks!

[tschiemer]

@wowczarek

From what I see in the CM4IO datasheet (page 7) SYNC_IN, SYNC_OUT are available on J2 pins 8+9

[pelwell]

Damn those open datasheets!

[wowczarek]

@pelwell haha, senior moment :)

[jaycordaro]

I saw @wowczarek message on TimeNuts. I used to work for Broadcom's Ethernet PHY group and worked on the 5421x series including the IEEE1588 part of the chip. I remain on good relations with my former colleagues. There are several packages and two dies which map to the BCM54210 part number. I think the part on the Raspberry Pi (BCM54213PE? is that correct) does not pin out the SYNC_IN and SYNC_OUT pins even though they are on the die (36 pin package). I will check with them. Even if the HW SYNC pins are not brought out, some functionality of the IEEE-1588 portion of the PHY may be able to be enabled.

@ghollingworth, does the datasheet you have access to mention IEEE-1588?

[dp111]

On the CM4 we use the BCM54210PE and the SYNC_IN and SYNC_OUT are exposed to pins on the CM4 connector. The CM4IO board exposes these signals on connector J2. @jaycordaro if the PHY team are looking at a driver for this it would be great.

[jaycordaro]

ok I spoke to some friends who are still at Broadcom. The PHY does support HW timestamping, the register writes required to enable HW timestamping and the SYNC pins are described in an applications note, "IEEE-1588 for Timing Applications," not in the datasheet, so that may explain why the datasheet didn't provide anything useful. Someone who has Broadcom DocSafe access for BCM54210PE could request that document to be added to their account. The Product Line Manager would then approve the request. The PHY team people that I talked to (senior ASIC mgr for the PHY and someone in Marketing) are not aware of any effort inside the company to support a driver but they will try to help.

[ghollingworth]

Thanks to everyone on this... Especially to Philphil on the forums and @jaycordaro for their help! I do have access to the register information and Broadcom Docsafe and am currently trying to play catchup with the PTP kernel interface and ioctl interface to understand what it is I need to implement to get a working timestamping system.

What's clear is that there are a number of interfaces implementing different functionality, just need to decide on the order of implementation.

What is it that people actually want out of this?

1. external signal timestamping (timestamping of the SYNC_IN signal)
2. external event creation (SYNC_OUT periodic output signal)
3. TX and RX packet hardware timestamping (the BRCM MAC contains a FIFO from which you can read hardware timestamps)
4. PTP Hardware clock (which I think is implied with 1 and 2 but maybe not 3

Gordon

[wowczarek]

@ghollingworth actually 4. is implied exactly in 3., at least from the kernel interface point of view, for network sync (IEEE 1588) to work.

What is needed is all of the items you list, but as to the order:

a. 4. is required because this is the interface to controlling the clock.
b. 3. is required to sync this clock over the network
c. 1. is required to sync with an external hardware pulse such as from GNSS receivers or other quality clocks
d. 2. is required for external measurement and characterisation, which is priceless

• Items 1. and 2. are rare opportunities for hardware this common, giving this platform a chance to be the prime off-the-shelf testing ground for time sync related projects (*)
• Items 3. and 4. are the part that any other 1588-compliant chip has driver support for.

So unselfishly I'd say 3,4, then 1,2, but really it's the whole that people / we are / should be after.

(*) While we are at it, a set of u.fl (or ideally SMA) connectors could be considered for these two pins in the next revision of the carrier board instead, and freeing up the pins for something else. That would mean plug and play for serious timing, but it's not a big issue otherwise, we have the header. This is debatable though - a GNSS receiver on a board will also have a pin, but a more serious GNSS driven oscillator will have a coax connector.

Thanks!

[ghollingworth]

Thanks that helps a lot.

If you've got a suggestion for an Ethernet device that does actually implement these functions from the standard set of drivers:

I would suggest one of these (i.e. drivers that reference the ptp_clock_register function)

https://elixir.bootlin.com/linux/latest/C/ident/ptp_clock_register

That would help me with an initial implementation...

Also how would you test it? I can run ptp4l which I'm assuming would test the hardware timestamping, but may not test the hardware clock interface...

[tschiemer]

many thanks also!

Well, simply put I would think that all features might serve one use case or other and thus - optimally - all should be available to make the CM4 an attractive platform for any of those use cases.

The PTP hardware clock would seem the most reliable/precise possible and thus if it would be available that would be a definite yes.

Sync'ing external components to the PTP clock is - as I would think - also a core feature 1588 capable platforms should offer; an example being where audio ADCs and DACs should be driven by the sampling frequency derived from the network clock (thinking of AES67 here). As I understand from your wording, the periodic signal is adjustable within a particular range/frequency?

Analogously sync'ing the PTP clock to an external component would be quite interesting a feature to have, but as I understand the way SYNC_IN works is just timestamp generation.

Knowing the precise timestamp of an event on an external component would seem as important; ex knowing the precise time of sample acquisition, in particular if external components are not sync'ed to the PTP clock.

So I would consider 1, 2 and 4 to be quite important features to have for a module aiming at deeply embedded applications and would make the CM4 an interesting base indeed.

As for 3 (selective TX/RX hardware timestamping) nothing immediately comes to mind (not working with such tech) but I'm sure there are meaningful use cases.

wowczarek: I'm not sure 3 is required for actual network/ptp syncing, depends a bit how the 1588 functionalities are implemented (on-chip?)

but yes, I agree, 1+2 are indeed rare opportunities for hardware this common and it's indeed the whole package that should be aimed for

[wowczarek]

@tschiemer indeed, so this is why I made a distinction of sorts in the items from Gordon's list. However, IEEE 1588 == network time sync, don't forget. Without hardware packet timestamping (3,4) you cannot have a working PTP stack.

[tschiemer]

Also how would you test it? I can run ptp4l which I'm assuming would test the hardware timestamping, but may not test the hardware clock interface...

Spontaneous ideas... use a (HW or peripheral) clock for comparison? Observe SYNC_OUT on some external component use SYNC_IN to generate a timestamp and compare values?.... if it's an on-PHY clock, you just have to see how it runs, I guess?

Ja, I'm aware, just don't know what magic the PHY is capable of by itself.. thinking bout it, it likely won't have a PTP stack of it's own.. :/

[wowczarek]

@ghollingworth

If you've got a suggestion for an Ethernet device that does actually implement these functions from the standard set of drivers:

I think the closest are Intel i2xx/i3xx etc. where there are the two SDIO pins that can be ordered to be event inputs / outputs. The Linux PTP/PHC API has a way to grab the input event timestamps I would prefer a /dev/ppsX device, but that's just me.

Also how would you test it? I can run ptp4l which I'm assuming would test the hardware timestamping, but may not test the hardware clock interface...

Yes, ptp4l for the h/w timestamping and PTP Hardware Clock. I mean really what the "PTP hardware clock" is, "merely" two things, plus some initialisation logic:

1. A counter + register that counts either arbitrary ticks, with maybe another register to hold time of day or a lower/higher register pair for precise and rough time - or like Intel did it, a register/counter that ticks in human time. I am expecting the former. Plus some code to handle rollovers, etc.
2. Another register that controls the counter increment - this is how you steer the clock.

The timestamps captured are obviously latched counter values with a FIFO. So the timestamping driver part will pick these up, associate with frames/packets and attach these as out of band mesages (SCM) to the socket kernel buffer.

So the PHC driver provides an interface to this, allowing us to read time, set time, and steer the clock. If you use ptp4l as a PTP slave ('slave' term is under debate these days), you will utilise h/w timestamps to compute offset and control the hardware clock, this is why timestamps and PHC are one item.

[wowczarek]

@tschiemer

Ja, I'm aware, just don't know what magic the PHY is capable of by itself.. thinking bout it, it likely won't have a PTP stack of it's own.. :/

It won't. The PHY only timestamps packets, that is it, you need a software stack like ptp4l.

[wowczarek]

1. The hardware clock interface or PTP Hardware Clock / PHC, in Linux kernel terms, is only a standard interface for reading, writing and steering increment of the time register(s) on the chip and converting to human time. In hardware terms it's a register or a few, and that is that. This is not the same as the in/out pins; these are an extra.

2. "IEEE 1588 support" means some time register as above, and hardware timestamping, which means only one thing: timestamping packets/frames. Either the ability to timestamp every packet, which Intel, Solarflare and some others support, or, more commonly, only timestamp PTP packets, so the hardware matches a PTP packet using either the multicast group destination MAC address or PTP Ethernet multicast address or link-local variants of those (right at the beginning of frame), or if it's more sophisticated, UDP port 319, which also allows for unicast IP PTP timestamping. Hardware matches these, latches the timestamp counter and pushes that up a FIFO, long story short, they end up being human time attached to packets arriving at the socket. Obviously I don't know this chip and what's in the magic document, so I don't know which of the above matching options it does. There are also things like VLAN tag matching. What I do know is Broadcom's switching ASICs, and their PTP support is solid and extensive.

3. The event input pin should generate a timestamp aligned with the PTP clock on input rise transition (well, typically rise, for most timing applications). I hope it will be that simple. TI's AM335x network subsystem for example does it differently - event input has its own counter, but luckily it can be correlated with the PTP counter on rollover - more work, but it works. The problem with the event input and testing it, is that because it's meant to be handled in the PTP API, it's mostly ignored by people with GNSS receivers and there is near-zero software support for it. Really it should be a /dev/ppsX device.

4. The event output pin should allow, at minimum and by default, a rising pulse of certain width at the top of every second, of a certain width - make it 100 ms to keep all sync applications happy.

Testing 3. and 4, at least preliminary, can be done with two boards back to back, but ideally you need a time interval counter / frequency counter. Time nuts around the world will handle the rest. Two boards, Ethernet back to back, crossed over event pins, one is a PTP master, the other a PTP slave (ptp4l), PTP syncs them, and the inputs will show the clock difference between the two. That's the minimal DUT/SUT setup.

[marcohabeck]

Unfortunately, I was able to find out in December. I put an i210 in the PCIe slot and built a kernel. The PHY clock is synchronized and so is the board. Unfortunately the clock of the CM4 is not good enough and the whole thing runs away after hours and the clock can no longer be synchronized with the PHY .

[wowczarek]

@marcohabeck this is off-topic - but if you mean it runs away after being synchronised once and then leaving it, then welcome to the real world. In a PC or server it would be no different. This hardware can timestamp, but it can't keep time, that's just the way it is. Poor oscillators. Timing cards with oscillators good enough to hold better than hundreds of microseconds per day and few and far between. Oregano/Meinberg and high-end Exablaze are the only ones I know can do decent holdover. Solarflare right behind them. These systems have to be kept actively synchronised all the time, they drift very quickly.

But technically PTP NICs can be steered much more aggressively than the OS clock (I assume this is what you mean by 'clock' vs. PHY), so the CM4 or drivers or software would have had to be severely broken if the OS clock couldn't keep up with the NIC. OS clock control is limited to +/- 512ppm (~half a millisecond per second) adjustment, which is huge. People would have noticed drift of this magnitude. You'd have to check your software stack. Was the PHY (MAC really) synchronised to PTP, or were you just synchronizing the OS clock to the NIC without PTP?

[neggles]

I'm quite happy to help out with hardware-side testing on this stuff; the PHY hardware timestamping support is most of the reason I picked up a CM4+CM4IO board, and it would be absolutely killer for something like a next-gen LimeNET Micro (and for time sync applications in general)

I've got a number of PTP-equipped devices and 1PPS sources - some commercial, some homemade - available for testing, though I doubt you'd have much trouble finding plenty of people who're interested!

[cfdez-tech]

Hi @ghollingworth, I've been working on the ptp 1588 driver for the bcm54210pe, at the moment I've got the full skeleton implemented and the kernel can check configuration available for the PTP and PHC:

root@rpi-cm4:~# ethtool -T eth0
Time sta[   64.191025] DEBUG: Passed bcm54210pe_ts_info 258 
mping parameters for eth0:
Capabilities:
        hardware-transmit     (SOF_TIMESTAMPING_TX_HARDWARE)
        hardware-receive      (SOF_TIMESTAMPING_RX_HARDWARE)
        hardware-raw-clock    (SOF_TIMESTAMPING_RAW_HARDWARE)
PTP Hardware Clock: 0
Hardware Transmit Timestamp Modes:
        off                   (HWTSTAMP_TX_OFF)
        on                    (HWTSTAMP_TX_ON)
        one-step-sync         (HWTSTAMP_TX_ONESTEP_SYNC)
Hardware Receive Filter Modes: none

Also I can get and set the PHC using phc_ctl tool:

root@rpi-cm4:~# date
Fri Mar  9 13:03:13 UTC 2018

root@rpi-cm4:~# phc_ctl /dev/ptp0 get
[  174.782856] DEBUG: Passed ptp_clock_gettime 109
[  174.787636] DEBUG: Passed ptp_clock_gettime 114
[  174.792269] DEBUG: bcm54210pe_gettime 176
[  174.796722] DEBUG: Get time 4386 13124 21862 0 0
[  174.801471] DEBUG: Current PCH time 860116326.0
[  174.806145] DEBUG: Passed ptp_clock_adjtime 126
[  174.810797] DEBUG: Passed ptp_clock_gettime 109
[  174.815454] DEBUG: Passed ptp_clock_gettime 114
[  174.820073] DEBUG: bcm54210pe_gettime 176
[  174.824450] DEBUG: Get time 4386 13124 21862 0 0
[  174.829163] DEBUG: Current PCH time 860116326.0
phc_ctl[174.832]: clock time is 1234549248713053542.000000000 or (null)
root@rpi-cm4:~# phc_ctl /dev/ptp0 set
[  176.947297] DEBUG: Passed ptp_clock_gettime 109
[  176.951944] DEBUG: Passed ptp_clock_gettime 114
[  176.956614] DEBUG: bcm54210pe_gettime 176
[  176.961005] DEBUG: Get time 4386 13124 21862 0 0
[  176.965790] DEBUG: Current PCH time 860116326.0
[  176.970450] DEBUG: Passed ptp_clock_adjtime 126
[  176.975122] DEBUG: Passed ptp_clock_settime 98
[  176.979653] DEBUG: Original time 1520600486 102643396
[  176.984846] DEBUG: Set time 0 23202 34214 1552 14020
phc_ctl[176.988]: set clock time to 1520600486.102643396 or Fri Mar  9 13:01:26 2018

root@rpi-cm4:~# phc_ctl /dev/ptp0 get

[  179.635848] DEBUG: Passed ptp_clock_gettime 109
[  179.640502] DEBUG: Passed ptp_clock_gettime 114
[  179.645127] DEBUG: bcm54210pe_gettime 176
[  179.649521] DEBUG: Get time 0 23202 34214 1552 14020
[  179.654601] DEBUG: Current PCH time 1520600486.101725892
[  179.660022] DEBUG: Passed ptp_clock_adjtime 126
[  179.664650] DEBUG: Passed ptp_clock_gettime 109
[  179.669267] DEBUG: Passed ptp_clock_gettime 114
[  179.673883] DEBUG: bcm54210pe_gettime 176
[  179.678254] DEBUG: Get time 0 23202 34214 1552 14020
[  179.683331] DEBUG: Current PCH time 1520600486.101725892
phc_ctl[179.687]: clock time is 1520600486.101725892 or Fri Mar  9 13:01:26 2018

The current blocking point is that, despite reading and setting the PHC using phc_ctl set/get tool, this counter is not increasing its value, so it's not counting. Has any of the contributors in here got a contact in Broadcom in order to know whats happening with the register configuration? I've checked several times and I think it's correct. I cannot get any more ideas from the datasheet.

In parallel, I'm currently developing the time stamping routines in order to get the info from the Broadcom internal FIFO. I think I will have the first draft maybe during next week.

I'll be glad to contribute to this issue. Maybe we can create an branch in order to start collaborating here.

[ghollingworth]

Can you share your code, so I can see your implementation?

[ghollingworth]

I have access to support from Broadcom and have been using it to get the PHY setup correctly, currently I've got it set up timestamping packets and working correctly, but am also having trouble reading back the current timer counter. Whick register set are you using? The original time stamp or the heartbeat time stamp registers?

Gordon

[cfdez-tech]

I'm currently using the Original Time register, I can read and write in it, but it's not counting.

Mi register setup is the following one (extracted from Broadcom .vbs scripts and the other is added from my side in order to configure the SYNC_OUT)

 
REGISTER, VALUE
0x0F5, 0x0001
0xA10, 0x0101
0xAF8, 0x0101
0xAC0, 0x0F0F
0xAC1, 0x0F0F
0xAC3, 0x0101
0xA21, 0x0001
0xA22, 0x0000
0xA54, 0x0000
0xA55, 0x0000
0xA56, 0x5566
0xA57, 0x3344
0xA58, 0x1122
0xA5C, 0x07C0
0xA5D, 0x07C0
0x82D, 0x407E
0xA5E, 0x0002
0xA5B, 0x0160
0xA11, 0x00FF
0xA19, 0x0000
0xA78, 0x8000
0xA79, 0x400
0xA7A, 0x240
0xA7F, 0xD012 => In order to work with internal sync and SYNC_OUT pulses. We've got nothing connected on SYNC_IN pin.

[cfdez-tech]

Please @ghollingworth tell me how can I send you my code as soon as I've got green light to share it.
Thanks,
Carlos

[ghollingworth]

email me gordon@raspberrypi.com

[kuro68k]

Has anyone got this working at the moment? I've tried on a CM4, and ts2phc cannot lock to a PPS. Older versions can, but fail after a few hours.

I'm trying to debug it but not getting very far at the moment. Something changed in the last year or so that made it worse, but even code from around the time Jeff Geerling is only stable for a few hours.

[jclark]

If you follow the instructions at https://github.com/jclark/rpi-cm4-ptp-guide and it doesn't work for you, open an issue there with more details and I will try to help. My guess would be that your GPS isn't generating a reliable PPS signal.

[kuro68k]

Thanks, I have been following those instructions. The GPS is known to be reliable, and I checked it with an oscilloscope. I will open an issue there.

[Florob]

I've got a major issue with the current version of the PHY driver. As is, this code configures the PHY to replace all timestamps. This is incorrect, especially for HWTSTAMPS_TX_ON, which is not supposed to modify any data. In my specific case this completely breaks two-step peer-to-peer delay requests. I'm not convinced generated packets are 1588 compliant for other messages though.
Even for HWTSTAMPS_TX_ONESTEP_P2P this behavior is incorrect. Only Sync and Pdelay_Resp message should be modified. For the later the modification should be in the correctionField, not in the timestamp, according to the patchset that added it.
I have a small patch that changes some of this, but without access to the datasheet I have no idea how a proper implementation of HWTSTAMPS_TX_ONESTEP_P2P might look.

[jclark]

The code here has some special handling for HWTSTAMPS_TX_ONESTEP_P2P and HWTSTAMPS_TX_ON.

[anthonio9]

What is the current state of this issue? Does raspberrypi kernel for CM4 finally support PTP hardware timestamping? Is SOF_TIMESTAMPING_TX_HARDWARE and SOF_TIMESTAMPING_RX_HARDWARE present on the ethtool -T eth0 output? Does the kernel require some special configuration to enable the feature?

[kuro68k]

I can only speak from experience. I am evaluating the CM4 for PTP and sync4e for a work project. For a while it was broken, but an update some time this year seems to have fixed it and the results are now acceptable, pending longer term tests.

There are some limitations due to the hardware. It's not clear if the official IO board is badly designed or the CM4 is, but you can't use one of the IO pins which means you can't have both a PPS in and PPS out at the same time. At least it has PPS in though, some SoCs rely on GPIOs and interrupts.

There is also no facility for using an external PTP clock, which seems to be a limitation of the SoC. The steering of the internal clock is okay in my limited testing, but is affected by system load (i.e. heat, particularly when ramping up or down).

Once I have some better data I'll see about sharing it. Overall it looks good but does need further validation. I think it was a firmware update that broke and then fixed it, so careful version control is probably required. The CM5 is also on the horizon. Think about your requirements, for sub microsecond I think it is fine, and maybe sub 0.1µS over temperature.

Thanks to everyone who worked on this and made it happen.

[anthonio9]

@kuro68k what is your ethtool -T eth0 output? I'm getting what's below...

# ethtool -T eth0
Time stamping parameters for eth0:
Capabilities:
        hardware-transmit
        hardware-receive
        hardware-raw-clock
PTP Hardware Clock: 0
Hardware Transmit Timestamp Modes:
        off
        on
        onestep-sync
        onestep-p2p
Hardware Receive Filter Modes:
        none
        ptpv2-event

and it seems not to be enough to run ptp with hardware timestamping.

[kuro68k]

Sorry for the delay in replying, I'm a bit busy. I'll get the output of my PI when I have a chance.

[jclark]

@anthonio9 That is the expected output. What is it that makes you say that it is not enough to run ptp with hardware timestamping?

[schorsche32]

Should that also work with the kernel being used for Ubuntu 22.04.4 LTS? If not, is there any chance of getting it running without recompiling the kernel?

[jclark]

Ubuntu 22.04.4 uses kernel 6.5, which includes the necessary support, so it should work, although I haven't tried it.

[anthonio9]

@jclark on some other platforms with the hardware timestamping support the output to ethtool is slightly different, with the SOF_TIMESTAMPING_TX_HARDWARE flag mentioned next to the listed capabilities:

Why does RPI not list these flags like in the picture?

An app I'm working with requires eth0 to be bonded with /dev/ptp0 and cannot see that this is true on CM4. It's not yet known if it's a problem with the app or CM4.

[jclark]

The line PTP Hardware Clock: 0 in the ethtool output means that the PTP hardware clock for eth0 is /dev/ptp0.

From a quick glance at the ethtool source code, it looks like the (SOF_*) part will be present when it is getting the information from the kernel using ioctl rather than netlink. The absence/presence of the (SOF_*) part does not imply any difference in the capabilities.

[schorsche32]

Ubuntu 22.04.4 uses kernel 6.5, which includes the necessary support, so it should work, although I haven't tried it.

I also saw that, but the ubuntu 22.04 for the raspberry still uses the kernel 5.15 (and is somehow not upgradeable), which does not include the necessary support.

[kuro68k]

It's becoming a little bit moot now because linuxptp has dropped support for the CM4. The maintainer won't accept patches that fix the bug in the BCM54210 that prevents it accepting packets with a version number of 2.1, so only the old V3.5 release works.

Hopefully the CM5 fixes it.

[jclark]

The maintainer changed his mind and it's fixed in linuxptp 4.2 nwtime/linuxptp@02a13b0

[lhoward]

I think the RPi5 (and thus presumably the CM5) has a BCM54213PE, FWIW. (Out of interest does anyone have a reference indicating the issue is in the transceiver and not the MAC in the SoC?)

[nullr0ute]

I think the RPi5 (and thus presumably the CM5) has a BCM54213PE, FWIW. (Out of interest does anyone have a reference indicating the issue is in the transceiver and not the MAC in the SoC?)

I don't think that can be assumed, the PHY between the RPi4 and CM4 are different. The RPi5 also moves from the Broadcom eth to a Cadence ethernet mac [1 ch7] (linux driver: MACB) although it does appear to be a broadcom PHY [2].

[1] https://datasheets.raspberrypi.com/rp1/rp1-peripherals.pdf
[2] https://github.com/raspberrypi/linux/blob/rpi-6.6.y/arch/arm/boot/dts/broadcom/bcm2712-rpi-cm5.dtsi#L219

[kuro68k]

It's probably too late now, but is there a good way to contact the RPF about requirements for the CM5?

[lhoward]

Ah, re: Cadence MAC, see RP1 [1].

[1] https://datasheets.raspberrypi.com/rp1/rp1-peripherals.pdf

[JamesH65]

It's probably too late now, but is there a good way to contact the RPF about requirements for the CM5?

Much too late. Please see the forward guidance here https://pip.raspberrypi.com/categories/945-forward-guidance/documents/RP-005110-WP/CM5-Forward-Guidance.pdf

[kuro68k]

It's probably too late now, but is there a good way to contact the RPF about requirements for the CM5?

Much too late. Please see the forward guidance here https://pip.raspberrypi.com/categories/945-forward-guidance/documents/RP-005110-WP/CM5-Forward-Guidance.pdf

Thanks. They have removed the SYNC_IN pin, which suggests that PTP is low priority. Seems like it will remain unsuitable for most PTP master mode applications, unless your requirements are note very demanding.

[lasselj]

Yes, well, what one can glance from this document is that the SYNC IN pin (which after 1588 was implemented for the PHY used in the CM4 had no purpose) has been repurposed, but also implicitly that the SYNC OUT pin (which in reality can be used for both SYNC IN and SYNC OUT) for the CM5 has been retained. Conversely the RPI5 has no SYNC pins because its PHY doesn't support hardware timestamping. In the RPI5 timestamping is done (less accurately) in the MAC inside the RP1.

It is left to the reader to draw the conclusions that follows therefrom :-)

/Lasse

Much too late. Please see the forward guidance here https://pip.raspberrypi.com/categories/945-forward-guidance/documents/RP-005110-WP/CM5-Forward-Guidance.pdf

[kuro68k]

Indeed, but aside from wanting both IN and OUT pins to make testing easier and avoid having to multiplex signals when switching roles, the lack of an external clock option does limit the stability of the PTP clock. It's a shame because the CM5 could be a very powerful device for PTP and SyncE if the full capabilities of the peripheral were available, even via a test point.

[JamesH65]

@dp111 Any thoughts?

[regymm]

Kinda curious is there a way to output the PPS signal to a GPIO (or other measurable ways) on the Raspberry Pi 5? Came across this question but there seems no answer yet... https://forums.raspberrypi.com/viewtopic.php?t=360556

[lasselj]

Hi,No PHC driven sync pins on the Pi5. We explored in the past with Broadcom if the 54213 PHY on the Pi4 had any undocumented features, but it wasn’t the case. For sync, the CM4 is your friend and odds are the CM5 will be too as the sync pin is retained in the forward guidance. Alternatively, in the interim if you want to play, you could always attach an Intel 225/226 retail card to the PCIe bus. It has three SDPs exposed. On the opentimecard.org we combine the CM4 and the i226… it’s a winner for sync. I can’t wait to see how the CM5 performs. We are big fans.-- Best regards,Lasse L. JohnsenE           ***@***.*** 7 Jul 2024, at 14:06, regymm ***@***.***> wrote: Kinda curious is there a way to output the PPS signal to a GPIO (or other measurable ways) on the Raspberry Pi 5? Came across this question but there seems no answer yet... https://forums.raspberrypi.com/viewtopic.php?t=360556 —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[kuro68k]

If you don't have €7,000 spare for an OpenTimeCard, a cheaper alternative is to get some Microchip eval boards. I'm playing with them at the moment. I'll write it up when I have it fully working.

Another option for the Pi5 is interrupts. I don't know if you can interrupt from the PPS, but if you can then you could use the interrupt to toggle a GPIO. The interrupt introduces some random latency, but it's already pretty bad due to the very poor clock being used for PTP (same on the CM4).

[regymm]

Thanks for the information, I probably need to go for a CM4.