SerialPorts Linux issues

[krwq]

Collective issue for all issues while work is still happening.
Most of the issues were identified when working on async implementation: dotnet/corefx#33027

There are still lots of failures, all of them are marked with custom [KnownFailure] - most of the tests won't run on CI because it doesn't have physical port.

Things known to still be broken on Linux:

• (Issue dotnet/corefx#33374; PR System.IO.Ports native package corefx#35126) System.IO.Ports package cannot be used with netcoreapp2.1 even though it depends on netstandard2.0 (requires 3.0)
• software emulated RTS (XonXoff) is not working (or at least all related tests are failing)
• There are issues with parity replacement - likely just doesn't work
• There are issues with DTREnable - those could be test issues
• BytesToWrite (looks ok through code inspection but seem to be misreporting, possibly data from the queue should be added there)
• SerialPort.ReadBufferSize is not respected (see below conversation)
• in certain cases DataReceived event can fire more than once (SerialStream does it correctly but code on top of that in SerialPort is not) - this is likely also issue on Windows - I've added a comment why this happens how this can be fixed on Linux (likely something similar can be done on Windows) - we need to explore if this is something we want to fix
• Some tests related to StopBits are failing when measuring the transmission time - this is likely a test issue but needs to be looked at to double check
• non standard BaudRates are currently not supported but I've seen some article explaining how to emulate those - there are also couple of test failures (or rather hangs) but likely those are test issues (except for when non-standard baud rate is used) - currently allowed baud rate ranges are hard coded on linux (couldn't find an easy way to get that info from OS - perhaps just trying to set it and seeing if OS allows it is a way to go)
  • (Method group allocates when it shouldn't #33747) High baud rates - I've removed the artificial limit
  • (allow to set custom serial speed on Linux #80534) Custom baud rates
• flushing is not working (currently no-op)
• BytesToRead may return -1 when error occurs which is causing APIs such SerialPort.ReadLine throw ArgumentException in some cases where there are errors during communication
• [test] HasHardwareFlowControl is not working correctly - this can be checked similarly as RTS related tests are doing it (this might have a broader meaning than just RTS/CTS so needs more look)
• (Running CoreCLR tests on Mono - Update the issues.target file with Git hub links as new issues are created for each failing excluded test #33258) Uart port is not always detected
• (ConcurrentDictionary.GetOrAdd is not always thread safe. #33221) Dispose implementation (i.e. Shutdown call might need to be moved to a new SafeHandle class)
  • exclusive access need to be removed explicitly since some devices do not get rid of it making device not openable after closing

[krwq]

Moving this to future since I've addressed issues which I found useful to fix. Remainder is stuff which is very rarely used and we can address as we need.

Please upvote/comment on issues if you find this and want to have something sooner so that we can prioritize accordingly.

[ss4adam]

Setting SerialPort.ReadBufferSize property doesn't seem to affect the Linux tty buffer:

_serialPort.ReadBufferSize = 3000000; // three megabytes, this works fine in windows and lets me read a massive chunk of data, but in Linux I can't seem to read anything past 4096 bytes.

later if I use _serialPort.Read(dataBuffer, 0, dataBuffer.Length); then I miss anything outside of this 4K range. I like reading a large buffer rather than lots of small reads because I'm sending files and worried that the computer freezing up could make me lose a piece of the data in the middle of the transfer

I know in the past the linux kernel had an internal buffer of 4096 bytes.
If this buffer is full and a new character arrives on the serial port,
the oldest character in the buffer will be overwritten and thus will be lost. I'm using RedHat7

Is this a problem that .NET can even fix if the kernel is limiting the buffer size? I'm not strong at Linux so I'm not sure. If this can't be fixed, should an exception be thrown?

[krwq]

Hi @ss4adam, yes, this is fixable (assuming kernel allows you to do so) although we might not have cycles to fix this so any help would be appreciated (note: we will be doing repo move soon so might be better to wait until that's finished).

I'll update the list shortly

[ss4adam]

Hey @krwq, not sure what "might not have cycles to fix" means, but I would love to help. Just give me some direction so that I know where I'm looking to make the fix since I'm new to github (I'll probably need to brush up on com development). Another solution I was thinking about in the case that tty buffer might not be dynamically changeable (I looked at some linux source and saw it was hardcoded in a header file N_TTY_BUF_SIZE), maybe have it just read the 4k buffer in a looping/polling manner under the covers and update bigger buffer incrementing the index as it moves along, much like normal stream reading, but then .NET users will be able to use _serialPort.ReadBufferSize = 3000000 and never know the that the linux buffer was actually never changed.

[krwq]

@ss4adam "no cycles" = no time

Thanks for offering the help!

Note corefx will not merge any PRs after tomorrow and new repo will show up which merges coreclr and corefx. I'd recommend to wait until then before making PRs.

The fix will likely be here:
https://github.com/dotnet/corefx/blob/master/src/System.IO.Ports/src/System/IO/Ports/SerialStream.Unix.cs#L373

but you can also backtrace from here:
I'd start with searching who uses this:
https://github.com/dotnet/corefx/blob/master/src/System.IO.Ports/src/System/IO/Ports/SerialPort.cs#L377

You'll likely need to add whatever syscall you need in one of those places:
https://github.com/dotnet/corefx/tree/master/src/System.IO.Ports/src/Interop/Unix
https://github.com/dotnet/corefx/tree/master/src/Common/src/Interop/Unix

[danmoseley]

@ss4adam also, general contributing info here: https://github.com/dotnet/corefx/#contributing-guide

Note, this repo will be merging into another one in a few days. Don't panic -- if you start your work against this one, it will be trivial to continue the work in the new one.

[valeriob]

Just out of curiosity, @ss4adam why do you need such a large buffer ?
What speed is the serial configured to, and why are'nt you able to keep up ?

[ss4adam]

@valeriob Each machine I use is different, the highest is usually over 300K baud, but most are 115200 (which would never have a problem with 4k buffer). I am able to keep up using BaseStream.Read() in a polling manner, but on windows environments I'm able to make the buffer bigger, which is useful if the application freezes and I miss one of my polling iterations.

[valeriob]

Thanks for explaining @ss4adam, my 2c : if you need that reliabilty you should think about doing your own polling loop in a different thread, that's how i handle it, it works fine 😄

[ss4adam]

@valeriob I actually do it it's own thread, but I'm still seeing dropped data when the machine gets overloaded with other applications, not exactly sure why yet.

[skang0401]

I have a device with a baud rate of 14400. How can I use it under Linux with system.IO.Ports

[krwq]

@skang0401 currently not supported. See note above about "Non-standard low baud rates" if you're interested in adding support

[Ellerbach]

I have a device with a baud rate of 14400. How can I use it under Linux with system.IO.Ports

@skang0401 if you need this quite fast, you can use a project I've been doing before the Linux System.IO.Ports which is perfectly working for low baud rates starting at 50. Check here: https://github.com/Ellerbach/serialapp/, there is an existing Nuget as well: https://www.nuget.org/packages/NetCoreSerial/

[krwq]

@Ellerbach I'm not seeing you using any custom dividers in your code, I wonder if we could just simply add this specific baud rate in order to get this supported (assuming it's available on all OSes)

[Ellerbach]

using any custom dividers in your code

Correct, those are the official baud rates.

I wonder if we could just simply add this specific baud rate in order to get this supported

It needs to be tested. My guess regarding the various sources is that the final check of being or not a valid baud rate is done at the driver level. So if the hardware supports any baud rate and the driver will be ok with any value, it should just support it. Historically, those baud rates are what they are because of specific oscillators used and then frequencies were basically multiplied. In our days, most serial related hardware can support a larger range of rates based either on software serial either on precise high frequency dividers.

[skang0401]

@Ellerbach
I have also tested the serialapp project. Although it can be started normally, it does not seem to work properly at the baudrate I want.The result is that the custom baudrate does not take effect.
@krwq
I use ”cutecom” on the official Linux OS of the Raspberry Pi4. It can work normally at a baud rate of 14400.
Here is the repository of cutecom: https://gitlab.com/cutecom/cutecom
Is it possible to refer to his implementation？

[krwq]

@skang0401 unfortunately that's GPL project which we can't reference here. As mentioned above I think this can be achieved with
serial_struct.baud_base & serial_struct.custom_divisor but this will require a bit of testing to claim that we support it and I currently don't have much time to do that (leaving on paternal leave in ~1month with some vacation in between).

If you have some time doing the testing I can support you with the pull request

[gmkado]

@krwq has there been any progress on the non-standard low baud rates? This pyserial implementation seems to work for my application but now I'm trying to port it over to dotnet and would be nice if this were supported

[krwq]

@gmkado unfortunately we're not currently doing any active work for serial ports in dotnet/runtime. There is an orthogonal effort going on in dotnet/iot though: dotnet/iot#1832 - you can mention your scenario in there.

I'd be happy to review any contributions here...

[GryBsh]

@krwq has there been any progress on the non-standard low baud rates? This pyserial implementation seems to work for my application but now I'm trying to port it over to dotnet and would be nice if this were supported

It doesn't support non-standard high bit rates on anything but Windows it seems either. I'm trying to set 111856, and it chokes on it. I'm sorry but the above deflection to the IoT team is subpar. This isn't some implementational difference of opinion, this is the behavior of straight broken software.

[krwq]

@GryBsh seems the issue you're mentioning might be addressed here: #80534

[slyshykO]

Is it possible to add an ability to enable RS-485 support on Linux according to https://www.kernel.org/doc/Documentation/serial/serial-rs485.txt ?

Thanks in advance.

[danmoseley]

Would that need a new API @slyshykO ?
Would it be something you'd be interested in working on?

[slyshykO]

@danmoseley
It looks like this feature needed a new API.
It needs to cover struct serial_rs485 from Linux API and add the possibility to apply it through ioctl call to the serial port file descriptor.

I can help with tests on the proper device, but I have a little experience in C#.