Repository navigation
CM4 is missing IEEE1588-2008 support through BCM54210PE #4151
Description
Activity
Just compiling a kernel with PTP timestamping support enabled (by default not). Will see if this changes anything.
I'm new to kernel topics, so I missed the obvious. Will report back
Built a new kernel according to docs and changed the following settings (through make menuconfig):
CONFIG_NETWORK_PHY_TIMESTAMPING=y # # PTP clock support # CONFIG_PTP_1588_CLOCK=m # CONFIG_DP83640_PHY is not set # CONFIG_PTP_1588_CLOCK_INES is not set # CONFIG_PTP_1588_CLOCK_IDT82P33 is not set # CONFIG_PTP_1588_CLOCK_IDTCM is not set # end of PTP clock supportLinux cm4-1 5.10.14-v7l-MY_CUSTOM_KERNEL+ #1 SMP Tue Feb 16 14:05:19 GMT 2021 armv7l GNU/LinuxNo change observed, ie
pi@cm4-1:~/linux $ ethtool -T eth0 Time stamping parameters for eth0: Capabilities: software-transmit (SOF_TIMESTAMPING_TX_SOFTWARE) software-receive (SOF_TIMESTAMPING_RX_SOFTWARE) software-system-clock (SOF_TIMESTAMPING_SOFTWARE) PTP Hardware Clock: none Hardware Transmit Timestamp Modes: none Hardware Receive Filter Modes: none pi@cm4-1:~ $ sudo lshw -class network *-network:0 description: Ethernet interface physical id: 2 logical name: eth0 serial: dc:a6:32:XX:XX:XX size: 1Gbit/s capacity: 1Gbit/s capabilities: ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation configuration: autonegotiation=on broadcast=yes driver=bcmgenet driverversion=5.10.14-v7l-MY_CUSTOM_KERNEL+ duplex=full ip=192.168.2.197 link=yes multicast=yes port=MII speed=1Gbit/s ...I figure because there is no particular PTP hardware clock (driver) there's no recognized PTP clock (...) - but this would depend on wether BCM54210PE actually would have a hardware clock (?)
bcmgenet does not seem to use
phy_devicesstruct mii_timestamper *mii_ts;so enabling timestamping has no effect...
For reference: https://github.com/raspberrypi/linux/blob/c9226080e513181ffb3909a905e9c23b8a6e8f62/include/linux/mii_timestamper.h
We're currently discussing getting support for writing a driver from Broadcom, although we have access to the datasheet, to me it's pretty meaningless!
Unfortunately, so is the Linux interface to the clock...
But we'll endeavour to get there
Reacted by David Christle, Chris Hills, Nathan Huisman, 林博仁 Buo-ren Lin, David Busby and ritchieOk, thanks!
Hoping to be hearing more from you!
Just to say, as for the docs, I wouldn't mind signing an NDA to have a look to see if can make sense of it..
Damn you Broadcom and your closed datasheets! Timestamping at the PHY level is the best one can get, but not if unimplemented in software...
I also do hope those SYNC_IN and SYNC_OUT pins are actual event inputs/outputs for external sync and were preserved on the I/O board and not repurposed for something else. I've seen countless SBCs where this potential was wasted.
So, @ghollingworth, aside from a PTP driver eventually being developed - do we know if the two pins mentioned (pins 16 and 18 on the CM4) are presented on the I/O board?
Thanks!
Reacted by Seamus and ritchieFrom what I see in the CM4IO datasheet (page 7) SYNC_IN, SYNC_OUT are available on J2 pins 8+9
Damn those open datasheets!
Reacted by Jeff Geerling, Alasdair Allan, deepthought-64, William Desportes, Nathan Huisman, Guillaume Khayat, Christian Kuhtz, Maximilian Schulz, Adyanth Hosavalike, ritchie and 1 moreI saw @wowczarek message on TimeNuts. I used to work for Broadcom's Ethernet PHY group and worked on the 5421x series including the IEEE1588 part of the chip. I remain on good relations with my former colleagues. There are several packages and two dies which map to the BCM54210 part number. I think the part on the Raspberry Pi (BCM54213PE? is that correct) does not pin out the SYNC_IN and SYNC_OUT pins even though they are on the die (36 pin package). I will check with them. Even if the HW SYNC pins are not brought out, some functionality of the IEEE-1588 portion of the PHY may be able to be enabled.
@ghollingworth, does the datasheet you have access to mention IEEE-1588?
Reacted by SeamusReacted by Ted BehlingOn the CM4 we use the BCM54210PE and the SYNC_IN and SYNC_OUT are exposed to pins on the CM4 connector. The CM4IO board exposes these signals on connector J2. @jaycordaro if the PHY team are looking at a driver for this it would be great.
Reacted by ritchieok I spoke to some friends who are still at Broadcom. The PHY does support HW timestamping, the register writes required to enable HW timestamping and the SYNC pins are described in an applications note, "IEEE-1588 for Timing Applications," not in the datasheet, so that may explain why the datasheet didn't provide anything useful. Someone who has Broadcom DocSafe access for BCM54210PE could request that document to be added to their account. The Product Line Manager would then approve the request. The PHY team people that I talked to (senior ASIC mgr for the PHY and someone in Marketing) are not aware of any effort inside the company to support a driver but they will try to help.
Reacted by philip, Wojciech Owczarek, LloydS, Mark A. Ziesemer, Jeff Geerling, deepthought-64, Chris Hills, Marco d'Itri, Nathan Huisman, 林博仁 Buo-ren Lin and 9 moreReacted by Cameron Tacklind and Ted BehlingThanks to everyone on this... Especially to Philphil on the forums and @jaycordaro for their help! I do have access to the register information and Broadcom Docsafe and am currently trying to play catchup with the PTP kernel interface and ioctl interface to understand what it is I need to implement to get a working timestamping system.
What's clear is that there are a number of interfaces implementing different functionality, just need to decide on the order of implementation.
What is it that people actually want out of this?
- external signal timestamping (timestamping of the SYNC_IN signal)
- external event creation (SYNC_OUT periodic output signal)
- TX and RX packet hardware timestamping (the BRCM MAC contains a FIFO from which you can read hardware timestamps)
- PTP Hardware clock (which I think is implied with 1 and 2 but maybe not 3
Gordon
@ghollingworth actually 4. is implied exactly in 3., at least from the kernel interface point of view, for network sync (IEEE 1588) to work.
What is needed is all of the items you list, but as to the order:
a. 4. is required because this is the interface to controlling the clock.
b. 3. is required to sync this clock over the network
c. 1. is required to sync with an external hardware pulse such as from GNSS receivers or other quality clocks
d. 2. is required for external measurement and characterisation, which is priceless- Items 1. and 2. are rare opportunities for hardware this common, giving this platform a chance to be the prime off-the-shelf testing ground for time sync related projects (*)
- Items 3. and 4. are the part that any other 1588-compliant chip has driver support for.
So unselfishly I'd say 3,4, then 1,2, but really it's the whole that people / we are / should be after.
(*) While we are at it, a set of u.fl (or ideally SMA) connectors could be considered for these two pins in the next revision of the carrier board instead, and freeing up the pins for something else. That would mean plug and play for serious timing, but it's not a big issue otherwise, we have the header. This is debatable though - a GNSS receiver on a board will also have a pin, but a more serious GNSS driven oscillator will have a coax connector.
Thanks!
Reacted by ritchieThanks that helps a lot.
If you've got a suggestion for an Ethernet device that does actually implement these functions from the standard set of drivers:
I would suggest one of these (i.e. drivers that reference the ptp_clock_register function)
https://elixir.bootlin.com/linux/latest/C/ident/ptp_clock_register
That would help me with an initial implementation...
Also how would you test it? I can run ptp4l which I'm assuming would test the hardware timestamping, but may not test the hardware clock interface...
386 remaining items
Ah, re: Cadence MAC, see RP1 [1].
[1] https://datasheets.raspberrypi.com/rp1/rp1-peripherals.pdf
It's probably too late now, but is there a good way to contact the RPF about requirements for the CM5?
Much too late. Please see the forward guidance here https://pip.raspberrypi.com/categories/945-forward-guidance/documents/RP-005110-WP/CM5-Forward-Guidance.pdf
It's probably too late now, but is there a good way to contact the RPF about requirements for the CM5?
Much too late. Please see the forward guidance here https://pip.raspberrypi.com/categories/945-forward-guidance/documents/RP-005110-WP/CM5-Forward-Guidance.pdf
Thanks. They have removed the SYNC_IN pin, which suggests that PTP is low priority. Seems like it will remain unsuitable for most PTP master mode applications, unless your requirements are note very demanding.
Yes, well, what one can glance from this document is that the SYNC IN pin (which after 1588 was implemented for the PHY used in the CM4 had no purpose) has been repurposed, but also implicitly that the SYNC OUT pin (which in reality can be used for both SYNC IN and SYNC OUT) for the CM5 has been retained. Conversely the RPI5 has no SYNC pins because its PHY doesn't support hardware timestamping. In the RPI5 timestamping is done (less accurately) in the MAC inside the RP1.
It is left to the reader to draw the conclusions that follows therefrom :-)
/Lasse
Much too late. Please see the forward guidance here https://pip.raspberrypi.com/categories/945-forward-guidance/documents/RP-005110-WP/CM5-Forward-Guidance.pdf
Indeed, but aside from wanting both IN and OUT pins to make testing easier and avoid having to multiplex signals when switching roles, the lack of an external clock option does limit the stability of the PTP clock. It's a shame because the CM5 could be a very powerful device for PTP and SyncE if the full capabilities of the peripheral were available, even via a test point.
Kinda curious is there a way to output the PPS signal to a GPIO (or other measurable ways) on the Raspberry Pi 5? Came across this question but there seems no answer yet... https://forums.raspberrypi.com/viewtopic.php?t=360556
- Hi,No PHC driven sync pins on the Pi5. We explored in the past with Broadcom if the 54213 PHY on the Pi4 had any undocumented features, but it wasn’t the case. For sync, the CM4 is your friend and odds are the CM5 will be too as the sync pin is retained in the forward guidance. Alternatively, in the interim if you want to play, you could always attach an Intel 225/226 retail card to the PCIe bus. It has three SDPs exposed. On the opentimecard.org we combine the CM4 and the i226… it’s a winner for sync. I can’t wait to see how the CM5 performs. We are big fans.-- Best regards,Lasse L. JohnsenE ***@***.*** 7 Jul 2024, at 14:06, regymm ***@***.***> wrote: Kinda curious is there a way to output the PPS signal to a GPIO (or other measurable ways) on the Raspberry Pi 5? Came across this question but there seems no answer yet... https://forums.raspberrypi.com/viewtopic.php?t=360556 —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>Reacted by regymm
If you don't have €7,000 spare for an OpenTimeCard, a cheaper alternative is to get some Microchip eval boards. I'm playing with them at the moment. I'll write it up when I have it fully working.
Another option for the Pi5 is interrupts. I don't know if you can interrupt from the PPS, but if you can then you could use the interrupt to toggle a GPIO. The interrupt introduces some random latency, but it's already pretty bad due to the very poor clock being used for PTP (same on the CM4).
Thanks for the information, I probably need to go for a CM4.
There is a discussion in #5904 about a non-compliance of the current implementation of the CM4, Pi5 and CM5 with regards to IEEE 1588-2008 compliance in the presence of IEEE 1588-2019 traffic. As far as I understand it:
- A reserved field in IEEE 1588-2008 (v2) now indicates the minor PTP version in IEEE 1588-2019 (v2.1).
- A compliant implementation of IEEE 1588-2008 should ignore that reserved field, and, the rest of IEEE 1588-2019 is designed so that the packet remains compatible.
- The current implementation ignores timestamps on IEEE 1588-2019 packets tagged with a PTP minor version of
1.
I am hoping this is an oversight in the driver implementation and not a more fundamental issue with the PHY.
@mfurseman My understanding is that for the CM4/CM5 it is a hardware problem. As mentioned in that thread, ptp4l as of version 4.2 has the ptp_minor_version option that can work around this for ptp4l. Before I was aware of that, I developed https://github.com/jclark/bpf-ptpver20/ which uses a BPF TC egress filter to set the PTP minor version to 0; this can workaround the problem for any application. The Pi5 has a different PHY, which doesn't support hardware timestamping (the Pi5 does hardware timestamping in the MAC), so that's a separate problem.
The Pi5 has a different PHY, which doesn't support hardware timestamping (the Pi5 does hardware timestamping in the MAC), so that's a separate problem.
None of the B models have had PHYs with HW capable of timestamping. They are built to a price and it adds cost. The Pi5 had moved from onboard Broadcom MACs to one on the RP1 from Cadence, at least the upstream linux driver has PTP support, so hopefully it's all wired up properly in the RP1 chip. The CM5 has the same BCM54210PE that the CM4 had, and they explicitly mention "supporting IEEE 1588" so should have similar capabilities of the CM4.
None of the B models have had PHYs with HW capable of timestamping. They are built to a price and it adds cost. The Pi5 had moved from onboard Broadcom MACs to one on the RP1 from Cadence, at least the upstream linux driver has PTP support, so hopefully it's all wired up properly in the RP1 chip.
Do you happen to know, since the schematics are not available, if that means you can inject an external clock for the Cadence part? One of the major limitations of the Pi as a PTP master is that only the rather poor internal clock is available, which is steered towards sync with a PPS but is far from ideal. Being able to inject a more stable 25MHz clock would be very useful.
For the moment our design uses a Microchip PCIe ethernet IC that includes the PHY and accepts an external 25MHz clock, for that reason.
Do you happen to know, since the schematics are not available, if that means you can inject an external clock for the Cadence part?
TBH I don't know, I'm guessing even if the MAC IP could accept an external clock it would depend on how RPi wired it up on in the RP1 as to whether it could accept that clock source, we would need someone from RPi to confirm those details.
Reacted by kuro68k
Is this the right place for my bug report?
Yes: I asked on the forum, I asked the raspberry foundation through the contact form but didn't get any response - it is very much kernel related as is explained in the forum post.
Describe the bug
The CM4 has an undocumented inofficial PHY (BCM54210PE) that supports hardware timestamping (and a hardware clock, actually??), but there seems to be no driver supporting these features.
So CM4 does actually not provide hardware based IEEE1588-2008 support as opposed to what the raspberry pi foundation actually communicates in the CM4 datasheet.
To reproduce
sudo ethtool -T eth0only shows software timestamping capabilities.Expected behaviour
Depends - the raspberry pi foundation never communicated what IEEE1588-2008 features are supported such that is is unclear what exactly can be expected; also the PHY documentation is not available such that possibilities (ie optimal expected behaviour) can not be described.
Expected (according to https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/s1-using_ptp):
The CM4 datasheet lists SYNC_IN, SYNC_OUT pins ment for IEEE1588-2008 support but has not further specified this.
It actually is a question wether these pins are needed in the first place - because the PHY would seem to have a hardware-clock itself.
Further it must be clear wether the PHY's internal clock (if any) is routed to a CM4 pin in any form and if not this should be fixed in future revisions because custom boards for the CM4 might require tight synchronization to a hardware clock.
Actual behaviour
sudo ethtool -T eth0only shows software timestamping capabilities.There is no (documented) synchronizable IEEE1588-2008 clock signal going from the CM4.
System
Raspberry Pi Compute Module 4
Raspberry Pi OS Lite, January 11th 2021, Kernel version: 5.4
Ubuntu Server 20.04.2 LTS,
Logs
none
Additional context
BCM54210PE documentation is required to implement to see actual features provided and configuration options - IEEE802.3-2018 seems to define said capabilities etc.
Possibly also required are CM4 schematics and BCM2711 documentation.
Also see IEEE802.3-2018 Section 90. Ethernet support for time synchronization protocols
I can try to take care of the implementation, but I would like to have the mentioned documents.