ReadBytes() method cannot detect :WAVeform:DATA eof correctly

[luyi1888]

0x0a in the chunks end is not sufficient to detected eof. It will return in the middle of data.

Device: Siglent SDS 2000X Plus

https://siglentna.com/wp-content/uploads/dlm_uploads/2024/03/ProgrammingGuide_EN11F.pdf

I make the following changes to workaround this bug. Maybe read length from header will be better idea.

ScpiConnectionExtensions.cs

public static async Task<byte[]> ReadBytes(this IScpiConnection conn, int numOfBytesToRead, int specialTimeout = 0, CancellationToken cancellationToken = default)
{
    var result = new MemoryStream();
    var buffer = new byte[conn.DefaultBufferSize];

    ReadResult chunk;
    while (result.Length < numOfBytesToRead)
    {
        chunk = await conn.Read(buffer, buffer.Length, specialTimeout, cancellationToken);
        result.Write(buffer, 0, chunk.Length);
    }

    return result.ToArray();
}

[klasyc]

Hi and thank you for your feedback. The ReadResult.Eof attribute should be a reliable way to detect the end of a message, because this flag is set by the instrument itself. I think, the real cause of your problem is that the instrument sends data slowly and the default reading timeout expires just before the instrument prepares the entire waveform (the loop is broken by the chunk.Length > 0 condition). This happens to me normally when I download a waveform from an oscilloscope.

For this reason, I introduced the specialTimeout parameter, which allows increasing the reading timeout just for the "big reads", which are quite rare, at least in my experience. Have you tried to set the specialTimeout parameter, e.g. to 5000 ms?

Your workaround works, because you entirely removed the timeout requirement, and you strictly wait for the given amount of data. But if the instrument (or USB connection) crashed or sent less data, you would probably wait forever.

Using the read length from the header is not always possible, because some waveforms are too big to be transmitted at once. Therefore, I read them in chunks, which means that the read length in the header does not correspond to the total length of transmitted data.

To achieve the best result, I would experiment with the following:

• UsbScpiConnection.Timeout - this is the default timeout used for regular commands. Here I would prefer a short timeout to quickly detect connection drops/command errors.
• UsbScpiConnection.DefaultBufferSize - if the waveform is not read at once, this defines the length of a single transfer. Optimum value depends on the instrument speed and its internal buffers. I usually try different buffer sizes and measure which gives me the fastest transfer speed.
• specialTimeout of the ReadBytes() method - this gives the extra timeout to your waveform read. Here I usually start with a big value, measure the real time of transfer and then I decrease the timeout value accordingly.

[luyi1888]

I am using the TCP connection and set 5000ms timeout.
using (IScpiConnection connection = new TcpScpiConnection(hostname, port, 5000))

I am not try specialTimeout yet.

But the bug is depends on waveform data.
I set oscilloscope memory depth 2Mpts.
Sometimes, it can capture the whole waveform data(2,000,011 bytes 8bits sample/plus header).
If Readbytes() return before the whole data captured. It will stop at the same location (eg. always stop 700kb or other)

[klasyc]

Oh, you are using TCP connection - that's another story... and increasing the specialTimeout makes a little sense...

Yes, in this case EOF flag is not reliable, because there is simply no convenient way, how to detect it in TCP stream. Currently, the library sets EOF when the received buffer ends with LF character, but if the LF character is part of data (and is situated at the end of the reading chunk), then the detection fails and the reading function returns too early.

The first thing I have to fix is that TcpScpiConnection must not set the EOF flag because it does not have any reliable way on how to detect it. Then I will have to detect new lines in the ReadString() method in order not to break it.

Your updated/workaround ReadBytes method with explicit length of expected data actually works - when I read it for the first time, I did not realize that the timeout is handled in the Read() method - sorry for confusion. We could name it e.g. ReadExactly() and add it to the library. The only thing to deal with is that we should handle the case when the Read() function call returns more data than you actually wanted.

To the curve reading: currently I am using bare Read() method from the IScpiConnection interface and I do exactly what you proposed - first I read the length of data in the header, and then I read until I have all bytes. I could add this code as another extension method, but there is one catch: for big data reads it is better to read the curve by chunks and immediately process them. Maybe I could pass a delegate for block processing to that function...

Conclusion: thank you for reporting this issue. I will fix it next week - the EOF flag + extension methods.