ttyAMA0 Problems with 4.9.x Kernel #1913
Comments
|
Interestingly this only appears to affect ttyAMA0 (!) If I swap out the UARTs and put ttyS0 on pins 32/33 instead of ttyAMA0 : And re-run all is well |
|
Is it possible that this is the problem? https://github.com/raspberrypi/linux/commits/rpi-4.9.y/drivers/tty/serial/amba-pl011.c How can I narrow down further which commit broke this functionality for me? |
|
I have the same kernel: After installing and building mbpoll and the windows server (I have a USB serial port) I get: Running it for the first time after a boot, and repeatedly in a loop, I saw no errors. You haven't said which Pi you have - my tests were with a Pi 3 using If reading this hasn't led you to a solution, please tell me what Pi you have, and post the (non-commented-out) contents of config.txt and cmdline.txt. |
|
Hi Phil Thanks for replying, appreciate your time I'm running on a CM3 in a custom carrier board. The RS485 is opto-isolated from the main board, but I don't think that's relevant as I'll explain later I've dug a bit further and found a kind of work around that might help us track this down. Here's the config : config.txtdtoverlay=mmc cmdline.txtdwc_otg.lpm_enable=0 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait maxcpus=4 I was able to kind of get the system into a working state by disabling address 1 on the windows simulator, running the mbpoll program a few times, then re-enabling address 1 on the windows sim. This made me remember seeing some weird behaviour with ttyAMA0 when talking to a Microchip LoRa module the other week, in that case to get any sense out of the setup you had to do echo "anything" >/dev/ttyAMA0 and then toggle the LoRa module reset line after booting to get it in a constantly known working state. So going down the same path, yes that appears to make it behave as expected. In the log below I've powered off/on each time and then logged into the system via putty and run the commands shown. I don't understand why this should make it happy, that's really way beyond my software knowledge. As I say this works fine without any of the above on 4.4.50, and even on 4.9.x ttyS0 works without issue (after removing the console and swapping pins over of course) If you can see something obvious please let me know, or if there's a way I can step through commits using rpi-update can you point me at the web page to find the list? Thanks! |
|
You can find historical firmware versions here: https://github.com/Hexxeh/rpi-firmware/commits/master I suggest you start with the two commits either side of the jump to 4.9 - both on Feb 20th. and |
|
Hi Unless I'm missing something (which is entirely possible due to bad head cold) I tried both of those as I mention in the OP Any ideas as to why the echo-ing solves the problem? I'm happy to work on the basis I've muffed something up, but I can't immediately see what it can be? |
|
If you've tried adjacent commits and found one succeeds and the other fails then there is little point in trying any others. Opening and closing the UART probably causes it to flush its FIFOs. The minimal command to try would be: i.e. echo nothing. |
|
Yep, that does the same : |
|
Can you post the output of |
|
|
That's weird, I didn't get the -n to work this time |
|
Am I right in thinking that your uart0 overlay is based on the standard uart1 overlay, and preserves the pin pulls? |
|
TBH I had quite forgotten about that, I only remember making one to add in the RTS/CTS Here's the source of the two overlay files I have for uart0 This one is for adding the RTS/CTS lines (not installed currently): Is there a new one I can use now that works with pins 32/33? |
|
for interest three test/reboot cycles : |
|
root@raspberrypi:/boot/overlays# ls uart* -l |
|
There is no "official" uart0 overlay - create an issue (or a pull request) if you want one - but yours look fine. |
|
Do you have a USB serial port? If so:
I'm curious to see if this works for you - whether all data gets through correctly, and whether any extra characters are inserted. When you've finished the ttyUSB0->ttyAMA0 test, try the reverse direction. |
|
Hi See below |
|
Was that the first access to ttyAMA0. as far as you know? |
|
And the reverse direction? |
|
Yes that's the first thing done once logged in Here's the reverse flow Trying to make it count down rather than up (so we don't start with 00) I've modified the tx code to But that's not working as I expected |
|
That initialises the array to the same values but working backwards - try: |
|
Or: |
I've just done a test by unplugging the USB cable (same as used above) from the Pi board and into my Windows box, reboot the Pi board and then ran mbpoll as the first thing from login. The same problem/fix as seen above also applies with that setup (which improves my sanity level a bit..) If you can't replicate this on your setup then there must be a difference in the linux config between my system here and yours. Here's the boot area on mine |
|
What's in your dt-blob.bin? Can you try without it if that makes sense? |
|
Yeah I just tried that, no difference What I'm going to do next is download the Jessie lite from the RPi website, flash that and upgrade it via rpi-update and see if it still breaks then |
|
Do you have access to a logic analyzer? It would be interesting to 'scope the pins and see what's really going on. The Saleae Logic 4 is the cost of 3-4 Pis and can decode serial, I2C, SPI and CAN. The Bitscope Model 5 is cheaper and looks more capable. |
|
Progress! Flashed the stock jessie lite image and then copied across the overlay and blob files I've been using I found I saw the same problem. So to test a hunch I went back to my original setup image. I removed the uart0 overlay files, dt-blob I've added and the altered the config.txt file to use pins 14/15 (which I think it turns out is irrelevant as the config looks hard coded) That works fine! |
|
However, I'm currently having a problem getting TXD0 and RXD0 to work when set to appear on 32/33 Exact same hardware setup as before, (using USB-TTL) just moved to different pins and added in a new dtblob.bin and overlay file Here's the config /boot/config.txt dtblob source : uartzero overlay root@raspberrypi:~# ls /boot/overlays/uart* -l |
|
Still broken as of |
|
Still no joy as of : Looking at dmesg, this is interesting :
Then when the first time the mbpoll command is run the below appears at the end of dmesg
Once you run echo "anything" >/dev/ttyAMA0 it comes good again, but nothing more that I can see is added to dmesg. Does this give any more info? |
|
Those "errors" are unfortunate but transient. and expected. They are a symptom of the fact that device load order is essentially undefined, and that sometimes (or with some configurations) a device that needs a resource may be loaded before the device that provides the resource. When that happens, the device probe functions returns -EPROBE_DEFER and the kernel tries again after some more devices have been loaded. The "no DMA platform data" message is always there. The firmware behaves differently depending on whether it thinks uart0 (ttyAMA0) or uart1 (ttyS0) is the primary (console) uart. It decides the uart roles like this (in pseudo-code): What do you see after running these commands: |
|
ah OK, was just looking for anything that appeared unusual What's puzzling here is if I put ttyAMA0 on pins 14/15 (as I show above) it works fine, but if I put ttyAMA0 on pins 32/33 that's when it breaks until I do the echo. Oddly If I put ttyS0 on 32/33 that works fine. It's very strange behaviour... Here's what you've asked for pi@raspberrypi:/root $ grep -a . /proc/device-tree/aliases/serial* pi@raspberrypi:/root $ ls -l /dev/serial* And one level down pi@raspberrypi:/root $ ls -l /dev/serial/* /dev/serial/by-path: |
|
The output tells me that the firmware thinks that ttyAMA0 is the primary (console) UART, and also that you aren't running Raspbian (if you were, /dev/serial0 would be a symlink to /dev/ttyAMA0 and /dev/serial1 to /dev/ttyS0). This is probably not what you want, since you want ttyAMA0 to be left for your own use. If you look at the pi3-miniuart-bt overlay you will see that it configures the aliases to be the standard (non-Pi3) way round. If you reverse the assignments in one of your overlays it should make your CM3 behave like a Pi3 with respect to UARTs. |
|
Hi Phil I really appreciate your time on this It is Raspbian, honest! :) It's Jessie Lite (if my memory serves correct from March 2016 that's had rpi-update through 4.1 => 4.4 => 4.9) I don't know why the symlinks are not there (or what generates them) but I had the same issue on this even with the latest core image version downloaded from the website. I tried to narrow down if it was an obvious OS problem earlier. Pardon my slowness here but I'm not quite up to speed on the reasoning here, sorry. If I'm following this, are you saying that the problem here lies with how the firmware assignment of uart0 & 1 to serial0 &1 alias has been reversed by default - presumably to suit the Pi3 board So what we need to do is create a "unswap" overlay, borrowing from the bt overlay you reference, to "correct" this back to how we expect them to be for CM3 usage. This could be done by just creating an overlay with a single a section like fragment@4 : I'm really confused how, if this is the problem, that it works at all at the moment. That does not make sense and the more I think about it the more my head hurts! For what it's worth here's the kernel cmdline currently being used root@raspberrypi:~# cat /boot/cmdline.txt |
|
I'm not at the point of saying where the problem lies, I'm just eliminating possibilities by eliminating differences. What I am saying is that currently you are using the UARTs backwards compared to how they are meant to be used - serial0 is meant to designate the console UART and serial1 the BT/spare UART. I am suggesting that you swap the roles in your system so that you can use them as they are intended to be used. It might not make any difference to your problem, but it removes a difference to my test setup. To that end, your overlay looks correct to me. |
|
YES! We have a winner :) So I created a unswap overlay :
Added that after the serial port assignments Next had to do was swap the cmdline console assignment:
Now works first time, interesting there's no error the first time you run it either I really can't fathom why something like this should /kinda/ work if it's wrong, or what at 4.9 was the straw that broke the camels back either. Thank you very much for your patience here |
|
Thank goodness for that. I would prefer you to call your overlay uart_swap since it points serial0 at uart1 and serial1 at uart0. Even though that is the default configuration on a Pi 3, it is still swapped with respect to other Pis. |
|
Sure, will do I'm just as thankful we've sorted it too :) I understand the main thrust of the work done here is aimed at the standard Pi units, but is it worth having some standardised CM overlays? |
|
Most overlays have been submitted or requested. Feel free to do either. |
|
hi everyone! |
|
@pelwell, I'm running into an issue related to this, and I'm trying to figure out how you managed to get what you did in #1913 (comment) working at all. The -5 option causes mbpoll isn't checking the return result from This is with kernel 4.9.35-v7+. Unless I'm reading something wrong (more than entirely possible), the pl011 driver has no support for the RS485 mode (unlike the SC16IS7xx driver, for example). The serial port is configured using But given how there's no RS485 support in the pl011 driver, I'm puzzled as to how both transmit and receive worked for you. Perhaps you were not using actual RS485, and since there was no RS485 transceiver actually being switched transmit and receive, it worked as a RS232 port would? |
|
Hi @jcwren I was using a MAX13487 which doesn't require RTS flow control. FWIW I don't think ttyAMA0 comes up with RTS/CTS lines by default, so you'll need to enable them (using a device tree overlay for example) if you want to use them I'd expect |
|
@jcwren You're right - I was using a straight RS232 connection. If the use of an RS485 adapter alters the requirements on the RS232 interface then I'm sorry - that wasn't obvious from the original post. @mypiandrew is correct that flow control is not normally mapped onto the header pins. You can either use an overlay or a utility such as raspi-gpio to change that: |
|
@mypiandrew, @pelwell, thanks. I've got RTS0 configured to pin 17, so I'm good there. Somewhere there's a fork of libmodbus that either uses a GPIO or manually farbles RTS to control the RS485 transceiver. Unfortunately, it's somewhat behind libmodbus master, so I'll have to figure out what it takes to bring it up to date. Is there any technical reason the pl011 driver couldn't support the RS485 modes? There's a lot of conflicting information out there that the pl011 does/doesn't support CTS/RTS, does/doesn't support automatic flow control, etc. With a casual glance at the SC16IS7xx, it looks like the RS485 support is not particularly hardware dependent, and perhaps could be ported into the pl011 driver. The problem with using a GPIO line and doing it outside the driver means that you now have to account for the delay in the FIFO. And that means either long enough fixed delays (slowing down overall throughput) or risking too short a delay and switching the transceiver back to RX mode before it's actually finished transmitting (bad data). |
|
The BCM2835 pl011 does do automatic flow control (although I can't rule out there being a subtle bug). We're using the standard upstream driver largely unmodified, so any serious change like you are suggesting, if possible, would have to go upstream. Have you considered using a USB RS485 interface? It might be easier and more performant in the long run. |
|
What's a little frustrating is that the autoconf tools can't tell the driver doesn't support RS485, only whether the kernel does or not. So when libmodbus builds on a Pi, it says "Yeah, sure, I have TIOCSRS485!" and merrily builds away. Of course, it doesn't know ahead of time that the actual port driver doesn't support it, but still... I already use a USB adapter, but I was trying to eliminate it, primarily because of size and awkwardness of building into to an enclosure. I laid out a Pi hat PCB using a MAX13451E (I use that on a lot of my other RS485 hardware) plus some other bits, with the expectation that because RTS was available, it should be easily doable. It wasn't until I actually had hardware I could test against that I discovered that the pl011 driver wasn't capable enough. Yeah, sure, maybe some deeper research would have revealed this, but RS485 over a UART with RTS control is pretty basic stuff. I haven't done any kernel development since the 2.x kernels, but I may take a poke at integrating it in. Seems there's a fair amount of interest in RS485 on Pi's out there, and a fully integrated driver would make that easier. |
|
From what you said you may be able to study the SC16IS7xx driver and apply the same logic to pl011 by analogy. Fingers crossed it is that easy, but there may be a reason why it hasn't been done already. Ooh - this may be helpful: https://www.spinics.net/lists/arm-kernel/msg597518.html |
|
Nice find! I've followed up with him to see if there's any other changes. The patch is missing a lot of the supporting infrastructure, but looks pretty cookie cutter to add. He's concerned about the mdelay() call while waiting for the FIFO to clear, but I can live with that. At 115200 baud, that shouldn't be more than a 100us stall. |
|
Hey sorry for the long wall of text - I really need some help here and I want to include everything I can.
after which the Pi restarts. This is the last and only message I see in syslog after my code begins executing. Also while booting I'm getting the error : I'm using a Pi 3
to switch AMA0 and s0 Any chance I could be doing something wrong here? What should I investigate? Thanks in advance. |
|
Never mind, it was due to excessive power consumption. |
I'm seeing problems when running an RS485 adapter circuit (MAX13487 with auto flow control) added to ttyAMA0 on the latest 4.9 Kernels. I don't see this problem on the 4.4.50 kernel.
I can't immediately see it's a hardware issue as I can flip the version back and forth from 4.4.50 to 4.9.x and the fault moves with that change.
To test I'm using this on the Pi
https://github.com/epsilonrt/mbpoll
and the below on a windows box with a proper serial port (not USB)
https://sourceforge.net/projects/modrssim2/
Looking here for commits :
https://github.com/Hexxeh/rpi-firmware/commits/master
See below log if that helps
Can anyone suggest what I can do to isolate this further?
It looks like it's the first character of the receive that's getting corrupted/added each time as the windows box is sending back
<01><03><0A><00><6F><01><4D> etcand not
<FE><01><03><0A><00><6F><01><4D> etcNote that I am aware there's a "funny" with the first time the serial port is opened and that run has been omitted from the above
The text was updated successfully, but these errors were encountered: