[Bug] Socket->Send() Sometimes Corrupted Data / Resending Previously Sent Data / Slow Chunking

[davefilip]

You have found a bug in Circle? Please fill out this form!

Description of the bug

Rene,

I have recently started upgrading my Circle-based OS from Step 50 to Step 51, given that it sounds like the network layer has been improved, and I am excited to try the new IPC (pipe) functionality.

The upgrade was easy, and I didn't need to modify any code this time (vs. going from 49 to 50 which required some refactoring).

However, I did start having a problem with Step 51 related to network sockets.

I have a telnet server that I wrote 2+ years ago, that had been working reliably since. However, with Step 51 I've been having some reliability problems, which seems to be related to calling CSocket->Send():

CLogger::Get()->Write(LogSource, LogNotice, "[%d] %s", strlen(string), string);
m_Socket->Send(string, strlen(string), MSG_DONTWAIT);

I added the Clogger line before it to verify that the string being passed is good, and the length is valid (i.e., not is never an unterminated string, the parameters are valid as displayed on the conosle).

Although I have no problem receiving data from a telnet client, sending data to the telnet client with the same kernel may or may not work depending on the RPi model and WiFi access point.

Sometimes it works fine, but sometimes will re-send fragements of data that has previously been sent, or add what looks like lots of blank lines, and sometimes it also displays the proper text but gets very choppy (seconds between each fragment of text sent).

I have done a lot of testing, and I suspect that it might be related to some combination of processor and network speed, based on the following (all using the EXACT SAME kernel image):

A. RPi 3B + High Speed Mesh WiFi - have not seen any problems

B. RPi 3B + Medium Speed Pocket WiFi (1) - mostly reliable, can occasionally see a problem

C. RPi 3B + Low Speed RPi Hotspot (2) - Frequently works but can sometime see the problem

D. RPi Zero 2W + High Speed Mesh WiFi - Frequently works but can sometime see the problem

E. RPi Zero 2W + Medium Speed Pocket WiFi (1) - Sometimes works but frequent problems

F. RPi Zero 2W + Low Speed RPi Hotspot (2) - Lots of problems, mostly unusable

Note: 1 - MikroTik mAP (2x2 inch battery)
Note: 2 - Raspberry Pi 3B running HostAPD + DNSMasq

For my current project, ideally I need to get (F) working for an IoT project on a remote platform.

And of course reverting back to the old (Step 50.0.1 based) kernel still works flawlessly in all of these environments.

In your release notes you've stated:

This release offers a significantly improved implementation of the TCP/IP network stack. It supports TCP congestion control according to RFC 5681 with Fast Retransmit and Fast Recovery.

Just to eliminate those as a possibility, is there any define (-D) to turn off the congestion control and/or Fast Retransmit + Recover capability?

Otherwise, any ideas on how to further troubleshoot this?

Thanks,

Dave.

Affected version(s) and hardware

What version(s) of Circle show this bug? On which Raspberry Pi model(s) did you notice this bug? Which additional hardware has been used?

51, 3B and Zero 2W (see above for specific test results)

Log output or screenshots

Corrupted output with fragements of prior text appended to correct text, or sometimes correct text truncated, and sometimes a second or two between each fragment of text.

Textual log output goes here.

[davefilip]

Since the new bug reporting asks for some screen shots, I am including the type of behavior I am seeing here.

So I do a 'show task' command, and get the following output:

=== cut here ===
OK, show task

#  ADDR     STAT  FL NAME
00 01708700 ready    main
01 01A87F40 ready    wifireader
02 01ACBFC0 ready    wifitimer
03 01B125C0 ready    net
04 01B52A40 sleep    wpa_supplicant
05 01C0DDC0 sleep    mqtt
06 01C08040 block    telnet
07 01A85EC0 block    tftpd
08 01CE8440 block    httpd
09 01C08480 sleep    ntpd
11 01C15080 run      telnet_3
=== cut here ===

Which is correct. I then do a 'dir services' command and get the following output:

=== cut here ===
OK, dir services

/MFD/SYSTEM/SERVICES: 10 files (13 KB)

ANEMOMETER.cmd            2025-10-07 10:25    896 B
DADJOKES.cmd              2026-01-23 09:01     1 KB
DHT11.cmd                 2025-10-06 10:05     1 KB
DHT22.cmd                 2025-10-06 10:05     1 KB
DS18B20.cmd               2025-10-06 10:05    907 B
PUBWEATHER.cmd            2026-01-08 09:17     2 KB#  ADDR     STAT  FL NAME
00 01708700 ready    main
01 01A87F40 ready    wifireader
02 01ACBFC0 ready    wifitimer
03 01B125C0 ready    net
04 01B52A40 sleep    wpa_supplicant
05 01C0DDC0 sleep    mqtt
06 01C08040 block    telnet
07 01A85EC0 block    tftpd
08 01CE8440 block    httpd
09 01C08480 sleep    ntpd
11 01C15080 run      telnet_3
=== cut here ===

Whereby it starts giving me a services directory, but then I get the output of the 'show task' halfway in the middle. I then do a 'dir services' command again, and I get the correct, full output:

=== cut here ===
OK, dir services

/MFD/SYSTEM/SERVICES: 10 files (13 KB)

ANEMOMETER.cmd            2025-10-07 10:25    896 B
DADJOKES.cmd              2026-01-23 09:01     1 KB
DHT11.cmd                 2025-10-06 10:05     1 KB
DHT22.cmd                 2025-10-06 10:05     1 KB
DS18B20.cmd               2025-10-06 10:05    907 B
PUBWEATHER.cmd            2026-01-08 09:17     2 KB
PURPLEAIR.cmd             2026-01-18 12:33     2 KB
STATUS.cmd                2025-10-06 10:05    634 B
WATCHNET.cmd              2025-11-01 17:34     3 KB
ZENQUOTES.cmd             2026-01-23 09:01     1 KB
=== cut here ===

So, sometimes I get the correct, full response sent back, and sometimes I get part of a prior response instead thrown in. However, I always see the proper response on the console, since I am now sending all responses to both the console and the telnet (socket on port 23) session. Therefore, I do not believe the problem is with my command processing, because the console always receives the proper response, but the wrong response gets sent to the socket.

In addition, I should say that I am not seeing (at least so far) any problems with single request/response protocols. As far as I can tell, only when keeping the socket open with multiple read/write requests.

[rsta2]

Dave, thanks for the report. There have been many changes in the network stack and this problem is probably not easy to find. The new features cannot be disabled. I'm trying to reproduce the issue here. Please give me some time. Does this happen with WiFi only or also with a wired connection?

[davefilip]

I believe the problem only occurs on WiFi, and seems to be more prevalent on slower WiFi (e.g., worst using a RPi 3B hotspot running HostAPD + DNSMasq in the basement, best using a new TP-Link mesh hotspot a few feet away).

As far as why I am not using NetworkManager for a RPi hotspot ... available since Bookwork ... although I can configure and get that to work with my other (e.g., Mac) computers, I have never gotten Circle's WiFi to work, and it just never connects. [I believe you verified this over a year ago when I reported another problem, which turned out to be needing to wait about a minute after disconnecting from a HostAPD + DNSMasq hotspot before reconnecting, otherwise HostAPD gets confused thinking it is the same session).

It also seems to be more prevalent on a Zero, and I do have one RPi Zero 2W with a network adapter in the basement, so let me try doing some testing with that today. But my concern is that the problem may be present only on slow network connections, and how to simulate that.

[davefilip]

UPDATE: I installed the new kernel (build on 51) on a Raspberry Pi Zero 2W with a USB Ethernet dongle, and although it is difficult / rare, I've been able to replicate the problem twice, whereby the output of one command was not complete, but append with part of the output from a prior command. So it appears to affect both WiFi and Ethernet.

I should also add that any time this happens, the socket does not get disconnected, but becomes very weird, and I start getting lots of the wrong text sent back from subsequent commands.

However, once I force disconnect (from the client) and then re-connect (to a new socket), everything goes back to normal for a while.

[rsta2]

Unfortunately I had no success in reproducing the problem so far. I'm using a RPi Zero 2 W on an old 54 Mbit/s D-Link AP. My test program is sending about 680 bytes of data each second. This arrives OK in a Telnet session on my PC. I also tried to ignore 33% of the received TCP packets to simulate packet loss and TCP retransmissions with the same result.

[davefilip]

Have you tried both sending and receiving on the same socket? So not just sending data out, but receiving data, then sending something back, receiving some more data, then sending something back, receiving some more data, and repeating?

[rsta2]

Yes, my test program can also receive and echos the received data back. It's a modified echo server from sample/20-tcpsimple. But that's not exactly the same.

[davefilip]

It also does depend on the amount of data received, as if I send the command (just tried this):

print text "Hello, World!"

about a hundred times, it comes back successfully every time, even on my "worst case" (Zero 2W with HostAPD+DNSMasq). Usually after a directory listing is when I have a problem.

And to clarify the direction, small amounts of data are sent from the telnet client to the server, and a larger amount of data sent back from the server to the client. Doing small bits of data back from the server does not seem to exacerbate the problem.

Wondering if it has to exceed the frame size?

[rsta2]

That's why I was sending 680 bytes. I will implement a simple command interpreter next later.

[davefilip]

Great, and some more information, if I put the directory listing in a file and list it out:

OK, list dirlist.txt
/MFD/SYSTEM: 32 files (16 MB)

.PASSWORD                 2024-11-06 08:35     16 B
ANEMOMETER.LOG            2025-10-06 17:15   327 KB
COMMANDS                  2025-06-15 17:47    <dir>
DADJOKES.LOG              2026-02-05 19:54   101 KB
DHT11.LOG                 2025-09-07 16:22     1 KB
DHT22.LOG                 2025-11-20 12:25    34 KB
ENABLED                   2025-06-15 18:45    <dir>
EXAMPLES                  2025-06-15 11:30    <dir>
PURPLEAIR.LOG             2026-01-19 15:52    64 KB
SERVICES                  2025-06-15 11:31    <dir>
STATUS.LOG                2025-08-02 12:18   1016 B
WATCHNET.LOG;4            2026-07-05 05:10    53 KB
ZENQUOTES.LOG             2026-02-05 19:54   108 KB
kernel8.athos             2026-07-12 14:25     2 MB
kernel8.bedroom           2026-07-12 14:44     2 MB
kernel8.cellar            2026-07-12 15:18     1 MB
kernel8.hash              2025-11-18 18:01     17 B
kernel8.img               2025-11-18 18:01     2 MB
kernel8.kitchen           2026-07-12 14:09     2 MB
kernel8.livingroom        2026-07-12 13:39     2 MB
kernel8.office            2026-07-12 13:23     2 MB
kernel8.patio             2026-07-14 08:38     1 MB
kernel8.porthos           2026-07-12 14:27     2 MB
purple1.json              2025-08-24 17:44     2 KB
reboot.cmd                2026-01-23 09:08    446 B
shutdown.cmd              2026-01-10 12:23    376 B
small.json                2025-08-25 12:19    104 B
sms.json                  2025-08-28 11:40    778 B
startup.cmd               2026-01-22 19:28     1 KB
status.txt                2025-08-28 13:07     51 B
test.cmd                  2025-09-17 08:31     94 B
test.txt;3                2026-07-09 15:57     29 B

OK,

I can do that many times without seeing the problem. But doing a real directory listing I frequently do see the problem.

So what is the difference? When I list a file, I call Send() once per 1024 bytes, so that listing will call Send() only twice for that listing, since it is only 1.6K. In contrast, doing a real directory listing will call Send() thirty-six (36) times, once for each line.

So perhaps (just a guess) the problem is exacerbated by calling Send() many times in succession on a slow network connection?

Thanks for looking into this for me, and I hope the information I am providing is helpful.

[rsta2]

Thanks for this interesting info! I will continue with it tomorrow. You are welcome.

[davefilip]

FYI - I added a new 'blast' command to colorOS (my Circle-based OS) to send a line of text multiple times using individual Send() calls, simply for testing purposes. I have been able to replicate the problem on a slow WiFi access point using this command, e.g.:

OK, blast 40 ABCDEFGHIJKLMNOPQRSTUVWXYZ

Which when it works, will display something like this:

*** Starting Blast Test ***

01-ABCDEFGHIJKLMNOPQRSTUVWXYZ
02-ABCDEFGHIJKLMNOPQRSTUVWXYZ
03-ABCDEFGHIJKLMNOPQRSTUVWXYZ
.. etc ...
39-ABCDEFGHIJKLMNOPQRSTUVWXYZ
40-ABCDEFGHIJKLMNOPQRSTUVWXYZ

*** Finished Blast Test ***

But when it doesn't work, will stop before displaying all lines (example here is 40 lines but can specify 1 to 99 lines), or skip groups of lines (e.g., display lines 1-20, and then continue with line 38), or will combine this output with other commands output (like a file directory or a log display).

Although I probably don't need to, just to make clear exactly what I am doing to replicate the problem, I am providing the code here:

//
// blast.cpp
//
// Blast Text Lines to Network User for colorOS Copyright (C) 2026 David E. Filip <dfilip@colornetlabs.com>
//
// Built on Circle - A C++ bare metal environment for Raspberry Pi
// Copyright (C) 2017-2023  R. Stange <rsta2@o2online.de>
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program.  If not, see <http://www.gnu.org/licenses/>.
//

#include <circle/net/in.h>
#include <circle/string.h>
#include <circle/timer.h>

#include "blast.h"

static const char LogSource[] = "blast";
static const bool debug = true;

// Constructor

COSBlast::COSBlast(COSShell *shell)
{
	m_Shell = shell;
}

// Destructor

COSBlast::~COSBlast(void) {
}

// bool Semd(CSocket *socket, int count, char *text) - Send count number of text packets to network user

bool COSBlast::Send(CSocket *socket, int count, char *text) {

	char msd = '0';
	char lsd = '1';

	int I;

	if (count > 99) {
		m_Shell->Output(BLAST_COUNT_RANGE);
		return false;
	}

	if ((text == NULPTR) || (strlen(text) == 0) || (strlen(text) > (MAX_COMMAND_LINE - 5))) {
		m_Shell->Output(BLAST_TEXT_LENGTH);
		return false;
	}

	char buffer[strlen(text) + 5];

	buffer[0] = msd;
	buffer[1] = lsd;
	buffer[2] = '-';

	strcpy(buffer+3, text);
	int l = strlen(buffer);

	buffer[l] = '\n';
	buffer[l+1] = '\0';

	m_Shell->Output(BLAST_START_MESSAGE);

	CTimer* Clock = CTimer::Get();

	for (i=0; i < count; i++) {

		unsigned long start = Clock->CTimer::GetTicks();
		socket->Send(buffer, strlen(buffer), MSG_DONTWAIT);
		unsigned long stop = Clock->CTimer::GetTicks() - start;

		CLogger::Get()->Write(LogSource, LogNotice, "%s - %u ticks", buffer, stop);

		if (lsd == '9') {
			msd++;
			lsd = '0';
		} else {
			lsd++;
		}

		buffer[0] = msd;
		buffer[1] = lsd;
	}

	m_Shell->Output(BLAST_END_MESSAGE);

	return true;
}

//
// Blast Text Lines to Network User for colorOS Copyright (C) 2026 David E. Filip <dfilip@colornetlabs.com>
//
// Built on Circle - A C++ bare metal environment for Raspberry Pi
// Copyright (C) 2017-2023  R. Stange <rsta2@o2online.de>
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program.  If not, see <http://www.gnu.org/licenses/>.
//
#ifndef _blast_h
#define _blast_h

#include <circle/net/socket.h>

#include "shell.h"

// Messages

#define BLAST_COUNT_RANGE		"Count must be between 1 and 99! (BLAST_COUNT_RANGE)\n"
#define BLAST_END_MESSAGE		"\n*** Finished Blast Test ***\n\n"
#define BLAST_START_MESSAGE		"\n*** Starting Blast Test ***\n\n"
#define BLAST_TEXT_LENGTH		"Text line not specified or too long! (BLAST_TEXT_LENGTH)\n"

class COSBlast {

public:

	COSBlast(COSShell *shell);
	~COSBlast(void);

	bool Send(CSocket *socket, int count, char *text);

private:

	COSShell	*m_Shell;
};

#endif

I have also included each Send() line written to the log (Clogger), to verify that the text being sent is good, but also with the time (in ticks) it takes the Send() call to process. However, the time is always zero, even while I sometimes see the network output chunk, pausing sometimes seconds between lines. Even when the 'blast' output is running very, very slowly, the log shows it completes very quickly.

Although perhaps this is because of the MSG_DONTWAIT option, although I have also tried setting this to zero (although with Step 50, I always used MSG_DONTWAIT and never had a problem).

Not sure if this helps, but thought I'd pass it along.

Regards,

Dave.

[rsta2]

I modified my test program so that it waits for some input (a command and/or just return) and then sends 40 lines of text with a Send() call for each line. Unfortunately I still cannot reproduce the problem with it. We probably need an other idea. See the test program attached!

tcp-send-test.zip