Inconsistent operation

[jimmie11]

I have a sensor which outputs data continuously every 7 ms. I know this is accurate because I am using the same sensor on a PC and the message comes from the sensor every 7 ms.

I modified one of your examples to read from that sensor and the code is attached. As you can see, every 10 (or so updates), something happens and the code does not update for over 100 ms.

What is causing that? Is it my code or the library?

Thanks in advance.
output_2023-06-02_09-11-47.log
QNTest.zip

[ssilverman]

First question: you're basing your application on the altcp-based example. Do you need TLS?

[jimmie11]

Hello Shawn: Thank you very much for responding, I appreciate it. I am a newbie. A friend of mine has written that code and I do not even know what TLS is! What do you recommend? Thanks again.

…

On Fri, Jun 2, 2023 at 5:09 PM Shawn Silverman ***@***.***> wrote: First question: you're basing your application on the altcp-based example. Do you need TLS? — Reply to this email directly, view it on GitHub <#41 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG5ZBNZIYJAJ6MXF635Q7PLXJJJB3ANCNFSM6AAAAAAYYMGH5U> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ssilverman]

I really don’t think the problem is with the QNEthernet library. Is it possible you’re either dropping packets, or maybe the sensor just takes a while to respond? Are you polling for received data fast enough?

Can you describe exactly what you’re trying to do in a sequence of steps (possibly referring to each other, eg. for “repeats”). Describe what’s connected, the protocols you need, how communication needs to be done, and what you want done with the received data.

What I’m hearing right now is that your code doesn’t work, you don’t know why, and that someone else wrote it. Does the person who wrote it have any thoughts?

[jimmie11]

I had attached the code I am using to the PJRC forum you responded to.  I will attach it again on GitHub when I return. Basically, I connect to the sensor then issue a command so the sensor sends its data continuously.  I only write once to the sensor after which it starts streaming its data every 7 ms.After that I simply capture the data to a buffer and decode it.  Pretty standard. There is no polling. The problem cannot be with the sensor because I did the same thing with .NET.  The sensor sends its data reliably every 7 ms.  I tested over a few million messages.  Over 99.9 % of the time, data was sent every 7 ms. If it helps, I can give you remote access to the sensor and the PC it is connected to over a lan.Thanks again.On Jun 2, 2023, at 6:26 PM, Shawn

[jimmie11]

Here is the code.
testCode.txt

[ssilverman]

Do you have any documentation on the sensor and how it communicates?

[ssilverman]

I'm also not seeing the program print what's in the log "number Diff: number"
I think I'm missing a bunch of information here.

[jimmie11]

Attached is the document explaining the telegram sent by the sensor. It is a very long document but you only need pages 112-115 which explains parsing in my code.

The code I attached is the running code I am using. There is nothing missing other than the tls_template.c which is attached.

==================================================================
telegram_listing_telegram_listing_ranging_sensors_lms1xx_lms5xx_tim2xx_tim5xx_tim7xx_lms1000_mrs1000_mrs6000_nav310_ld_oem15xx_ld_lrs36xx_lms4000_lrs4000_multiscan100_en_im0045927.pdf

[ssilverman]

Maybe the problem is with the readBytesUntil() call? (Note: I haven't dived too deeply yet because lack of time right now; just trying to point out some possibly-obvious things to me.) That does a timedRead() call internally, which waits up to the Stream's timeout value (I think the default is 1 second) for data. I believe a state machine would run better than waiting on any call. In fact, that's how I write most of my code that reads from external equipment (I've written lots of drivers that talk to hardware having various protocols); it never has calls that wait or delay.

[jimmie11]

Thank you Shawn. You have gone beyond the call.

You mention ".....I believe a state machine would run better". I do not know what that is. Can you please refer me to how this may be used?

Can the readBytesUntil() call problem be fixed by specifying a very short timeout for the timedRead()? That sensor is continuously outputting data so would a timeout of a few ms fix this issue?

Thank you.

[ssilverman]

What I meant by not having all the information is that the log doesn't match what the program outputs. The log has "Diff:", but the code prints "Difference:". Basically, the first thing I see is that we're not looking at the results of the code.

[jimmie11]

Oh, I understand now! I made changes to the Serial.print statements thinking that it was the potential cause of the delay. The program logic is the same though ....

I am traveling and do not have access to the sensor so as soon as I return tomorrow, I will send the exact log corresponding to the code.

Thank you.

[ssilverman]

I’ll add: TCP isn’t a real-time protocol, so sometimes delays are to be expected. I’m going to convert this issue to a discussion because I think that’s more appropriate. I’ll create an issue if necessary in the future.

[jimmie11]

Yes, TCP is not a real-time protocol. But neither is Windows a real-time OS. I have implemented similar .NET code, and the latency is only exceeded every few hundred to few thousand scans. And when it does, the latency is only about 3-5 ms. While I do not know enough about micro-controllers, I would think that timing issues should be better not worse than Windows. (I may not be comparing apples to apples though)

Using the QNEthernet library, the latency occurs every 10-20 scans (all the time) and it can be as high as 100 ms (which is an eternity for my application).

So the culprit is either my Teensy code or the QN library. Shawn did point out that "readBytesUntil()" can cause a delay up to one second, so I will change that to see if it makes a difference.

Given my limited knowledge of TCP programming, I would certainly appreciate the community's input.

[ssilverman]

Here's another thing I thought of: Another possibility is you're not sending data when you intend to send it. When writing data to a TCP connection, the data isn't set out right away. Instead, any queued data waits for the next "TCP timer tick", which could be up to a few hundred milliseconds later. Is it possible this is the case here, where the receiving side isn't receiving the data in time? You can fix this by calling flush() on your connection after sending the data.

Relevant information:
https://github.com/ssilverman/QNEthernet/blob/6ae54bbae98451e7f59a677b07948f1f7c887863/README.md#write-immediacy
(and its parent section)

[jimmie11]

Thank you very much Shawn for the followup. I was waiting to update you after changing the code and completing testing. The problem has been resolved thanks to your earlier suggestion.

The problem was caused by readBytesUntil(). When I changed the code to receiving all received data to a buffer, the latency issues all but disappeared (99.9% of the time). The rest is more likely because TCP is not a real-time protocol. (BTW, I had been using flush()`)

If you please indulge me in one last question, currently I am using the following statement to read from the buffer: bytesRead = client.read(message, 2048);

It works fine. But what is the impact of changing the "2048" to a lower or a higher number?

Thanks again for your help and for creating a wonderful library.

[ssilverman]

That number is the “up to” value. The function call, in your case, reads up to but not more than 2048 bytes. If more than 2048 bytes are available, it will read only up to 2048 bytes, possibly less. If less than 2048 bytes are available, it won’t read more than that count.

See the docs for the Arduino Stream API or source code (included with the Teensy distribution). Or, look at the QNEthernetClient.cpp source code for more information.

[jimmie11]

Thank you.