reader.listen

[hamidjalili59]

hi bro 👋😅
can you implement a stream for read data and a method onErr or onDone 🙏
For example,🤔 when the port is disconnected from the system, the error method is executed🌹

[FengChendian]

Ok, I will finish it tomorrow(UTC+8)

[FengChendian]

done, maybe... I don't have time to test it today.
The API is different.

  Future<StreamSubscription> readBytesOnListen(
      int bytesSize, Function(Uint8List value) onData,
      {required Function() onListen}){
  /// implement
}

[FengChendian]

I will add listen to write and test it in the next few days

[sabin26]

@FengChendian readBytesOnListen is not working for me.
Reading data using other programs from c++ and also using with other flutter packages works as expected. But, using this package with same configurations is not working for reading. Writing is working as expected ! Please have a look asap. I am getting empty data [ ].
I tried the following code:
port.readBytesOnListen(256, (value) => print(value), onListen: () {},);
Here, value is returning [ ] first time and it is not updating when data is received again.

Is it not the function to constantly listen to receiving data ?

[FengChendian]

@FengChendian readBytesOnListen is not working for me.
Reading data using other programs from c++ and also using with other flutter packages works as expected. But, using this package with same configurations is not working for reading. Writing is working as expected ! Please have a look asap. I am getting empty data [ ].
I tried the following code:
port.readBytesOnListen(256, (value) => print(value), onListen: () {},);
Here, value is returning [ ] first time and it is not updating when data is received again.

Is it not the function to constantly listen to receiving data ?

Yeah... The function was called only once. Because I close the subscription when bytes was read or timeout. It is not a constant listener.

[FengChendian]

@sabin26 Maybe I should call it readBytesWithFunction.. .I achieve it for doing something when reading bytes.
And If you want to get data after reading, you should use this function with onData parameters again and again.

I think I should create an API for constantly listen to receiving data...
Sorry about that, It was my fault that caused the misunderstanding

[sabin26]

@FengChendian Yes. If there is an API to listen receiving data then this package would cover most use cases. I am blocked in my project because of the unavailability of this function. I hope the API to be implemented soon.

[sabin26]

@FengChendian I looked into event driven method for being notified when the data arrives and found the use of 'WaitCommEvent'. I have no experience in win32 else I would have written the code and submitted a PR to this repository. I am going to place some links that might be useful.

https://docs.microsoft.com/en-us/previous-versions/ms810467(v=msdn.10)?redirectedfrom=MSDN#serial-status
https://docs.microsoft.com/en-us/windows/win32/devio/monitoring-communications-events
https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-waitcommevent

I can confirm that, the required classes are available on the flutter win32 packages.

If your are able to understand and write code for its API, it would be really appreciated.
Also, Could you please at-least provide when will it be available ? I have been delaying my project.

[FengChendian]

@FengChendian I looked into event driven method for being notified when the data arrives and found the use of 'WaitCommEvent'. I have no experience in win32 else I would have written the code and submitted a PR to this repository. I am going to place some links that might be useful.

https://docs.microsoft.com/en-us/previous-versions/ms810467(v=msdn.10)?redirectedfrom=MSDN#serial-status
https://docs.microsoft.com/en-us/windows/win32/devio/monitoring-communications-events
https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-waitcommevent

I can confirm that, the required classes are available on the flutter win32 packages.

If your are able to understand and write code for its API, it would be really appreciated.
Also, Could you please at-least provide when will it be available ? I have been delaying my project.

I can try this tomorrow. Now I don't have any Serial Port device, because I can't go back to school due to the delta COVID-19 in Nanjing, China. So progress is slow. I will try to configure the virtual serial port to implement these method.

[FengChendian]

@sabin26 I didn't find CreateEvent function in win32 flutter package. So I violently achieved constantly listen function... 😂
I'm tesing this function. I think that it can call onData automatically if data is recevied...

Here is source code:

  void readBytesOnlisten(int bytesSize, Function(Uint8List value) onData) {
    if (_timer != null) {
      _timer!.cancel();
    }
    _timer = Timer.periodic(Duration(milliseconds: 20), (timer) {
      readBytes(bytesSize).then((value) {
        if (value.isEmpty) {
          return;
        }
        onData(value);
      });
    });
  }

[FengChendian]

@sabin26 done, you can test it using

port.readBytesOnListen(8, (value) => print(value));

Print will be called when receiving value/

[sabin26]

I saw this method but with byteSize 1 through your other repository: SerialPortTool. You did the exact same thing there. I will test it out.

[FengChendian]

I saw this method but with byteSize 1 through your other repository: SerialPortTool. You did the exact same thing there. I will test it out.

Yeah... I use timer to implement read listen in my small project. Beacuse I don’t know whether Stream has a suitable API.
I use byteSize 1 to refresh fast.

[sabin26]

@FengChendian Is it the bytesize or the timer of 20ms, I am getting wrong data sometimes. Should I drop my bytesize to 1 to refresh fast ?
Also, sometimes hex.encode(value) is returning me d0.10. The expected value should have been 00.10. I don't want alphabets. Do you know how to solve this ?

[sabin26]

Through my usage, I found that waiting for the bytes to arrive from the event as done in C++ or C# is the perfect way to solve the issue rather than looping the reading method continuously. So, I will be shifting my work to c# from flutter at-least for my current desktop project.
I appreciate your efforts made related to solving this issue.

[FengChendian]

@sabin26 Now library 0.2.2 API is

void readBytesOnListen(int bytesSize, Function(Uint8List value) onData, {void onBefore()?, Duration? duration})

You can set proper dutation using parameters duration.

About wrong data, I can't reproduce this bug. It works fine on my machine. d0.00 may be wrong data, too.

And about reading method continuously, win32 package doesn't supports CreatEvent to implement event. I will comment on win32 package.

[sabin26]

I saw that your issue made a contributor submit a PR to win32 repo on adding the CreateEvent method. I will follow up on that and I hope when win32 will merge the PR, this package will update the readBytesOnListen API based on the new method. You can test out the PR and be ready to update when win32 will support it.

Link:
https://github.com/timsneath/win32/pull/283

[FengChendian]

@sabin26 I saw it too. I will test out the PR and try to implement wait event to get data properly. I hope that win32 package will merge the PR later. Then I will update this package to support new API.

[sabin26]

@FengChendian The PR has been merged in the github repo of win32 package.

[FengChendian]

@FengChendian The PR has been merged in the github repo of win32 package.

I saw it. I will update this package if I have time in next few days.

[sabin26]

@FengChendian Any updates ?

[FengChendian]

@sabin26 Sorry... I am busy with my subject this week. I will try to update tomorrow.

[FengChendian]

@sabin26 Done in version 0.3.1.
But WaitCommEvent is not a Non-blocking Function... I found UI will be blocked when using listen. I am trying to fix it.

[sabin26]

@FengChendian
I was expecting the WaitCommEvent to emit an event when the data is received. But you are using while(true) function to continuous loop inside the readBytesOnListen function.

(My expectation is shown below in the form of an example)

var receivedData = "";

_serialPort.OnDataReceived = OnDataReceivedFunc; // or any other format 

OnDataReceivedFunc()
{
    // will be called whenever _serialPort receives any data
    var bytes = read(bytesToRead);
    // receivedData += ConvertToString(bytes);
}

The above example is just a representation of what I want to achieve with this readBytesOnListen function. The implementation can be a bit different.

Here is the screenshot of the first comment on the issue. It says implementing a stream for read data. I believe the 2nd line represents my scenario.

And I am sorry if my previous comments could not make you clear on what the issue was.

[FengChendian]

@sabin26 I use while(true) just because WaitCommEvent will be executed once when called. If you want to emit events when data is received, you must call WaitCommEvent again and again. Otherwise the function OnDataReceivedFunc will only respond once.

And most importantly, UI blocking is due to WaitCommEvent itself, not infinite loop. As long as you call WaitCommEvent, the UI will block.

Because this function will block until signal coming or error in FILE_ATTRIBUTE_NORMAL mode. It's a bad Windows API.

Or you just want an event? If OnDataReceivedFunc called once is okay, I will update a function which read once onListen, using compute function for avoiding blocking. It's easy for once read.

[FengChendian]

@sabin26 I re-read win32 Docs. I know the solution about function blocking when calling WaitCommEvent.
Firstly, openFile using FILE_FLAG_OVERLAPPED (Async I/O), so WaitCommEvent will return value imediately. When RXCHAR, it will get ERROR_IO_PENDING and can use ReadFile function. The solution needs to use CreateEvent.

But win32 package does't publish 2.2.6 version which contains CreatEvent Function. I can't update my package because win32 package is too big and my network is bad.......

Here is the C++ code which using Async Mode. If your network is good, you can compare the way I use win32 and translate these codes into dart code...
Actually, win32 package API is simple

#include <windows.h>
#include <tchar.h>
#include <assert.h>
#include <stdio.h>

using namespace std;

void test()
{
	DWORD dwCommEvent;
	DWORD dwRead;
	char  chRead = 0;
	OVERLAPPED o;

	HANDLE hCom;

	hCom = CreateFile(TEXT("\\\\.\\COM6"),
		GENERIC_READ | GENERIC_WRITE,
		0,    // exclusive access 
		NULL, // default security attributes 
		OPEN_EXISTING,
		FILE_FLAG_OVERLAPPED,
		NULL
	);

	if (hCom == INVALID_HANDLE_VALUE)
	{
		// Handle the error. 
		printf("CreateFile failed with error %d.\n", GetLastError());
		return;
	}

	DCB dcbSerialParameters = { 0 };

	if (!GetCommState(hCom, &dcbSerialParameters))
	{
		printf("Failed to get current serial parameters\n");
	}
	else
	{
		dcbSerialParameters.BaudRate = CBR_115200;
		dcbSerialParameters.ByteSize = 8;
		dcbSerialParameters.StopBits = ONESTOPBIT;
		dcbSerialParameters.Parity = NOPARITY;
		dcbSerialParameters.fDtrControl = DTR_CONTROL_DISABLE;

		if (!SetCommState(hCom, &dcbSerialParameters))
		{
			printf("Could not set serial port parameters\n");
		}
		else
		{

			PurgeComm(hCom, PURGE_RXCLEAR | PURGE_TXCLEAR);
			Sleep(100);
		}
	}

	if (!SetCommMask(hCom, EV_RXCHAR))
	{
		return;
	}


	o.hEvent = CreateEvent(
		NULL,   // default security attributes 
		TRUE,   // manual-reset event 
		FALSE,  // not signaled 
		NULL    // no name
	);


	// Initialize the rest of the OVERLAPPED structure to zero.
	o.Internal = 0;
	o.InternalHigh = 0;
	o.Offset = 0;
	o.OffsetHigh = 0;

	assert(o.hEvent);


	// Error setting communications event mask.
	for (; ; )
	{


		if (WaitCommEvent(hCom, &dwCommEvent, &o))
		{
			if (ReadFile(hCom, &chRead, 1, &dwRead, NULL))
			{
				cout << "Read" << chRead << endl;
			}
			else
			{
				cout << "error, read:" << chRead << endl;
				continue;
			}
		}
		else
		{
			if (GetLastError() == ERROR_IO_PENDING)
			{
				cout << " ERROR_IO_PENDING" << endl;
				if (WaitForSingleObject(o.hEvent, 500) == WAIT_OBJECT_0)
				{
					ReadFile(hCom, &chRead, 1, &dwRead, &o);
					cout << "Read:" << chRead << endl;
				}

			}
			ResetEvent(o.hEvent);
			continue;
		}
	}

}

int main()
{
	cout << "Hello CMake." << endl;
	test();
	return 0;
}

[sabin26]

@FengChendian The package win32 has been updated to version 2.2.6 that contains CreateEvent function. I tried to modify readBytesOnListen as per your C++ code.
Here is the result:

  void readBytesOnListen(int bytesSize, Function(Uint8List value) onData, {void onBefore()?}) async {
    Uint8List uint8list;
    if (SetCommMask(handler!, 0x0001) == 0) {
      throw Exception("SetCommMask EV_RXCHAR failed");
    }
    if (onBefore != null) {
      onBefore();
    }
    await _computer.turnOn();

    OVERLAPPED o = OVERLAPPED();

    o.hEvent = CreateEvent(
        nullptr, // default security attributes
        TRUE, // manual-reset event
        FALSE, // not signaled
        nullptr // no name
        );

    // Initialize the rest of the OVERLAPPED structure to zero.
    o.Internal = 0;
    o.InternalHigh = 0;

    assert(o.hEvent != 0);

    for (;;) {
      if (await _computer.compute(wait, param: handler) == 1) {
        uint8list = await _read(bytesSize);
        if (uint8list.isNotEmpty) {
          onData(uint8list);
        }
      } else {
        if (GetLastError() == ERROR_IO_PENDING) {
          if (WaitForSingleObject(o.hEvent, 500) == WAIT_OBJECT_0) {
            uint8list = await _read(bytesSize);
            if (uint8list.isNotEmpty) {
              onData(uint8list);
            }
          }
        }
        ResetEvent(o.hEvent);
      }
    }
  }

But there are 3 objects that give the error as they are not available on win32 package (or you could say that I could not find it).

ERROR_IO_PENDING
WAIT_OBJECT_0
ResetEvent()

I removed the below given lines as they were also not available.

o.Offset = 0;
o.OffsetHigh = 0;

I edited the SerialPort.Open function to openFile using FILE_FLAG_OVERLAPPED.

void open() {
    if (_isOpened == false) {
      handler = CreateFile(_portNameUtf16, GENERIC_READ | GENERIC_WRITE, 0,
          nullptr, OPEN_EXISTING, FILE_FLAG_OVERLAPPED, NULL);

      if (handler == INVALID_HANDLE_VALUE) {
        final lastError = GetLastError();
        if (lastError == ERROR_FILE_NOT_FOUND) {
          throw Exception(_portNameUtf16.toDartString() + "is not available");
        } else {
          throw Exception('Last error is $lastError');
        }
      }

      _setCommState();

      _setCommTimeouts();

      _isOpened = true;
    } else {
      throw Exception('Port is opened');
    }
  }

[sabin26]

Although the compute is called for WaitEvent, I believe that calling the for loop with for(;;) or while loop with while(true) on UI thread might cause an issue. It's just a hunch.

[FengChendian]

Although the compute is called for WaitEvent, I believe that calling the for loop with for(;;) or while loop with while(true) on UI thread might cause an issue. It's just a hunch.

I have tested compute function in version 0.4.1 on GitHub. There are no UI blocking when loop. Because most of time will be used to wait Event, which is isolated.

But if you use WaitEvent function in FILE_ATTRIBUTE_NORMAL, you can't close handle while waiting event. That's a serious problem.

Maybe there are some problems which I can't realize. But I think I can't find better solution for constantly listen now. Because of WaitEvent function, I can't use stream or timer. Data isn't iterable or periodical. And timeout is infinite in this function.

I think another solution is using overlapped mode in I/O. Win32 package has published version 2.2.6 today. I will test it and using stream or timer to implement listen.

[FengChendian]

@sabin26 Done. I think it is no UI-blocking or stuck problem. There are two ways to change onlisten function.

port.readOnListenFunction = (value) {
  print(value);
};
Future.delayed(Duration(seconds: 4)).then(
    (value) => port.readBytesOnListen(2, (value) => print('object$value')));

But I want to remind you that win32 package doesn't have ResetEvent(). Auto-reset event is a little strange in my computer. ReadByteSize must be less or equal to receiced buffer size. Otherwise, ReadFile function always gives me zero data read.
For example:
If your I/O device send char buffer[4];
you need read size 4 in dart.

I don't know why. And I will open an issue in win32 package to request ResetEvent().

[sabin26]

If only we could do serialPort.BytesToRead that gives the received buffer size then the simplest solution would be the following:

main() {
  .....
  serialPort.OnDataReceived = OnDataReceivedFromSerialDevice;
  serialPort.Open();
}

OnDataReceivedFromSerialDevice() {
 // showing byte[] instead of Uint8List only for demo
  byte[] buffer = new byte[_serialPort.BytesToRead]; // this will be very helpful
  _serialPort.Read(buffer, 0, buffer.Length); // read the data ourselves
  ......
  do something with the data 
  ......
}

Here, the OnDataReceivedFromSerialDevice will only be called if the data is received but it will not get any data. We will call the serialPort.Read() on our own.
Or,
You could do the same from the inside code and return the buffer as Uint8List after doing as shown in the code above.
Either approach is good.

@FengChendian What do you think ?

[FengChendian]

@sabin26 Do you mean I just let people know the number of BytesToRead and give a function named OnDataReceivedFromSerialDevice which doesn't have read function inside?

It's a good idea. However, there are some technical issues or win32 bug. About getting the precise number of BytesToRead in Dart.
In CPP, it's easy, using ClearCommError(hCom, &errors, &status);

void test()
{
	DWORD dwCommEvent;
	DWORD dwRead;
	DWORD errors;
	COMSTAT status;
	
	OVERLAPPED o;

	HANDLE hCom;

	hCom = CreateFile(TEXT("\\\\.\\COM6"),
		GENERIC_READ | GENERIC_WRITE,
		0,    // exclusive access 
		NULL, // default security attributes 
		OPEN_EXISTING,
		FILE_FLAG_OVERLAPPED,
		NULL
	);

	if (hCom == INVALID_HANDLE_VALUE)
	{
		// Handle the error. 
		printf("CreateFile failed with error %d.\n", GetLastError());
		return;
	}

	DCB dcbSerialParameters = { 0 };

	if (!GetCommState(hCom, &dcbSerialParameters))
	{
		printf("Failed to get current serial parameters\n");
	}
	else
	{
		dcbSerialParameters.BaudRate = CBR_115200;
		dcbSerialParameters.ByteSize = 8;
		dcbSerialParameters.StopBits = ONESTOPBIT;
		dcbSerialParameters.Parity = NOPARITY;
		dcbSerialParameters.fDtrControl = DTR_CONTROL_DISABLE;

		if (!SetCommState(hCom, &dcbSerialParameters))
		{
			printf("Could not set serial port parameters\n");
		}
		else
		{

			PurgeComm(hCom, PURGE_RXCLEAR | PURGE_TXCLEAR);
			Sleep(100);
		}
	}

	if (!SetCommMask(hCom, EV_RXCHAR))
	{
		return;
	}


	o.hEvent = CreateEvent(
		NULL,   // default security attributes 
		TRUE,   // manual-reset event 
		FALSE,  // not signaled 
		NULL    // no name
	);


	// Initialize the rest of the OVERLAPPED structure to zero.
	o.Internal = 0;
	o.InternalHigh = 0;
	o.Offset = 0;
	o.OffsetHigh = 0;

	assert(o.hEvent);


	// Error setting communications event mask.
	for (; ; )
	{

		char  chRead[20] = { '\0' };
		if (WaitCommEvent(hCom, &dwCommEvent, &o))
		{
			if (ReadFile(hCom, &chRead, 4, &dwRead, NULL))
			{
				cout << "Read" << chRead << endl;
			}
			else
			{
				cout << "error, read:" << chRead << endl;
				continue;
			}
		}
		else
		{
			if (GetLastError() == ERROR_IO_PENDING)
			{
				cout << " ERROR_IO_PENDING" << endl;
				if (WaitForSingleObject(o.hEvent, 500) == WAIT_OBJECT_0)
				{
					ClearCommError(hCom, &errors, &status);
					cout << "Que" << status.cbInQue << endl;
					ReadFile(hCom, &chRead, status.cbInQue, &dwRead, &o);
					cout << "Read:" << dwRead << endl;
				}

			}
			ResetEvent(o.hEvent);
			continue;
		}
	}

}

But in dart, you will find code can't respond correctly when bytesToRead(cbInQue) is 1... Data will be zero sometimes. Or you will get more data in a few seconds. Or data is right. 2,3,4,5 is fine.
Dart code:

  /// look up I/O event and read data using stream
  Stream<Uint8List> _lookUpEvent(Duration interval) async* {
    int event = 0;
    Uint8List data;
    PurgeComm(handler!, PURGE_RXCLEAR | PURGE_TXCLEAR);
    while (true) {
      await Future.delayed(interval);
      event = WaitCommEvent(handler!, _dwCommEvent, _over);
      if (event == TRUE) {
        ClearCommError(handler!, _errors, _status);
        data = await _read(_readBytesSize);
        if (data.isNotEmpty) {
          yield data;
        }
      } else {
        if (GetLastError() == ERROR_IO_PENDING) {
          /// WaitForSingleObject is often timeout, so remove it
          if (WaitForSingleObject(_over.ref.hEvent, 500) == 0) {
            ClearCommError(handler!, _errors, _status);
            print(_status.ref.cbInQue);
            data = await _read(_status.ref.cbInQue);

            if (data.isNotEmpty) {
              yield data;
            }
          }
        }
      }
    }
  }

In CPP, all is right:

I think I am going to be crazy because of the strange bug...

[sabin26]

@FengChendian If the idea was good enough and there is no other blocking factor besides this strange bug we could open up an issue on win32 package and get help to implement the bytesToRead properly. As per my understanding, if this is implemented correctly the feature as I described will be completed right ?
And don't worry, sometimes being crazy is worth it. It will be very helpful for many dart/flutter developers like me when this is completed.

[FengChendian]

@sabin26 Yeah. I will try to implement this feature if this bug fixed. And I will make OnDataReceivedFromSerialDevice and ReadOnListen are both optional in the future version.

About this bug, I think maybe it is due to auto-reset event in dart. So I will wait for updating ResetEvent to test it. If it's not working, I will open an issue in win32.
You can test these code and open an issue if you are free for doing something or in a hurry. I'm not sure that the issue can be reproduced in any computer.

[sabin26]

@FengChendian The issue that you raised on win32 package regarding the ResetEvent and OVERLAPPED structure has been closed. These have been implemented now in the win32 package. I hope the bug to be solved soon.

Also, my time is being occupied learning and developing a WinUI 3 based desktop application on C#. So, I am not in a hurry. You can take your time.