Problem with the read callback from Arduino

[OdysseusU]

Hi,

I have an issue in the read callback when connecting the arduino. It seems that I receive my bytes twice and sometime the callback is triggered but the package received is empty.

I only send a byte array with the arduino every 2 seconds and I just print what we receive with android.

Also I don't understand how the package is received. Sometimes I can get the whole array in one simple array and sometimes it's received truncated and the callback is called several time to get the whole array.

Thank you

[felHR85]

Hi,
In some devices data is split into more than callback because of the usb to serial chipset. You can onvercome this using two approaches:

• Append the received data into a buffer and use it when it makes sense.
• Use the sync vdrsion of the library (added in the last version). Methods are called like syncOpen, syncRead, syncWrite...With those methods you should be able to specify how many bytes you want to receive.

[OdysseusU]

Thank you for your fast reply! I had already been buffering but it was troublesome. I'll see with the sync version.

[felHR85]

It is the last thing I added so it would be nice to receive your feedback about if the sync version works correctly.
What USB-Serial converter does your arduino use?

[OdysseusU]

I will work on that today and get to you after that.
I am using the last Arduino UNO which uses their own firmware for USB-Serial converter on their Atmega8U2 chip but it should be like a FTDI chip.

[OdysseusU]

So after doing some tests with the sync version, it seems to be working but I get a lot of 0 in between my bytes when I (sync)read. I suppose that's because i limited my byte array to 13 bytes and i just receive a certain amount below 13. The thing is that I send from the arduino exactly 13 bytes. So I still get a truncated packaged received from the Arduino.

I did a simple test where I write 13 bytes from android every 2 seconds
and on the arduino it sends s13 bytes whenever it receives something (13 bytes).
And everytime android sends something it reads in a while loop (with a timeout) and prints what it received.

in summary :
Android --> Arduino --> Android
.........13 bytes.....13 bytes.........

I will look further to see how I can improve my code.

[felHR85]

Java arrays are set to 0 by default so probably you are receiving less than 13 bytes when the timeout raises.

Are you using UsbSerial 4.1.1 because that release solved an issue with the synchronous read in FTDI devices.

I would check the arduino code too.

[OdysseusU]

Yes I am using the final release.

So yes i suppose that it was because i was receiving less than what i expected that I had the 0s.

But the biggest problem comes from receiving multiple package when sending only one. And this is the case for both methods (sync and callback).
How would you suggest using the sync version the best way? Having a timeout seems to be the problem. Because the data could arrive at the end of the timeout, so it would cut the received data. And I can't use flow control.

The code on Arduino seemed fine. I went back with the callback version because it was simpler to work with and improved my buffering to get what I want.
But it could be great to have the received data in a whole.

I suppose the only solution is to buffer the received package.

[felHR85]

Set timeout to 0 :)

[jstonis]

OdysseusU, did you get the synchronous api to work? i wanted to attempt using it since I only need to keep reading a string of 13 characters from the Arduino.

[OdysseusU]

Yes do you have any problem with it? I didn't really have any. The main problem was that my frames were cut but that is device related.

If you want to use synchronous api just put a while in there with a syncRead and changing everything open by syncOpen and close by syncClose. This should be good.

[jstonis]

Seems i do..so i only get one reading and thats it..i put the syncRead in the broadcastReceiver...not in a while loop just in there and then used tvappend to append to display..1st error is decoding ..not converting to string, 2nd error is getting just 1 read and thats it

[OdysseusU]

The decoding is a bit strange with android. Look into other methods to convert into string (don't use the method .toString()) rather create your String and put your encoding ( new String(bytes, encoding UTF8 for instance). I don't really remember how to catch the string again but I think the .toString() method isn't good.

And the reading problem, are you sure your Arduino sends the data multiple times? Did you put the timeout to 0? this caused me some problem since it waits until the byte arrives and sometimes stops completely the application if nothing is sent (even if it was). Put a short timeout to be sure.

But generally the library works as it was intended to. So the problems you encounter comes from java or your arduino.

[jstonis]

So.i did that and it cuts my string..it produces the string but only a quarter of it...atrange..its still splitting up the data

[jstonis]

Idk why it still splits it up..but it does. I set the size of the array to 14, because the string im expecting is A: 000 B: 000...so idk what to do

[OdysseusU]

Yup i've got the same problem. Your solutions are :

• Set timeout to 0 (or longer to catch all the bytes) and wait to have all your bytes
• Set a buffer which adds up every bytes you have until you get your 13 bytes.

I took the second solution because i couldn't afford to wait.

This is not library related as every library that does the serial with arduino does the same. It's device related (or Android related, don't know).

[jstonis]

So you really could use the async..but i.noticed i was losiglng data..so you read until you have 13 bytes..what if you receive extra bytes so its the next set of data..

[OdysseusU]

Well you have to know when to stop the reading and do buffering right. But I never lose the data (only 1 every minute or so when sending every 50 - 100 ms)
But yeah I went with the async version.

[jstonis]

So you do lose a byte every min? Can i see how you did it

[OdysseusU]

I won't put the whole code here it would be useless since it's quite specific what I do.
But here is a sample in the read callback, b being the bytes received :
`

    if(b != null){
        if(b.length > 0){
            if (isStartByte(b[0])&& !startbytefound) { //look if its a new frame
                startbytefound = true;
                clearBytes(); // clears my buffer
                iNBbyte = /* formula specific to my use, just put the number of bytes you need */;
            }
            appendBytes(b):
            if(bufferSize >= iNBbyte && startbytefound){

                bufferSize = iNBbyte;
                byte[] buf = new byte[iNBbyte];
                System.arraycopy(buffer, 0, buf, 0, bufferSize);
                clearBytes();
                appendBytes(buf); // append to my buffer

                checkData(); //process the data
                startbytefound = false;
                isAck = false;
            }
        }
    }`

What I do is quite simple.
I check if i have the start data (in your case 'A')
then i begin to append the data in the buffer.
If i have what i need (bufferSize >= iNBbyte) then i process it.
I do all these checks (b.length > 0 or b != null) because sometimes i get empty bytes or bytes of length too short, and even sometimes i get the data 2 times.

[jstonis]

okay i tried something similar to this algorithm. in your arduino code, do you do serial.println..so theres a new line in the string, or just serial.print?

[OdysseusU]

Actually i use serial.write to write byte code. But you could serial.println and find the ´\n' character to find the end of frame

[OdysseusU]

It is actually simpler since you just need to find if your received bytes contains the ´\n' character and if it does process the buffer you have. But it is more optimized to not search for the ´/n' character if you already know the size of the frame you should receive.

[jstonis]

So can i just stick to the way you did it with serial.println in arduino?

[OdysseusU]

Yes the serial.println all it does is serial.print +'\n'

[jstonis]

byte [] data=new byte[0];
byte iNBbyte;

byte[] buffer=new byte[13];
int bufferSize=13;
boolean isAck=false;

public boolean isStartByte(byte firstChar){
    if(firstChar=='A'){
        return true;
    }
    else{
        return false;
    }
}
public void clearBytes(){
    data=null;
}
public void appendBytes(byte[] b){
   byte[] one=buffer;
    byte[] two=b;
    byte[] combined=new byte[one.length + two.length];
    for(int i=0; i<combined.length; i++){
        combined[i]=i<one.length?one[i]: two[i-one.length];
    }

    buffer=combined;

}
public void checkData(){
   tvAppend(errorCheck,buffer.toString());
}



UsbSerialInterface.UsbReadCallback mCallback = new UsbSerialInterface.UsbReadCallback() { //Defining a Callback which triggers whenever data is read.
    @Override
    public void onReceivedData(byte[] arg0) {

        data = null;

           // data = new String(arg0, "UTF-8");
            data=arg0;

            //try this for now

            if(data!= null){
                if(data.length > 0){
                    if (isStartByte(data[0])&& !startbytefound) { //look if its a new frame
                        startbytefound = true;
                        clearBytes(); // clears my buffer
                        iNBbyte =13; /* formula specific to my use, just put the number of bytes you need */;
                    }
                    appendBytes(data);
                    if(bufferSize >= iNBbyte && startbytefound){

                        bufferSize = iNBbyte;
                        byte[] buf = new byte[iNBbyte];
                        System.arraycopy(buffer, 0, buf, 0, bufferSize);
                        clearBytes();
                        appendBytes(buf); // append to my buffer

                        checkData(); //process the data
                        startbytefound = false;
                        isAck = false;
                    }
                }
            }

what am i doing wrong???

[OdysseusU]

It's my fault I didn't detailed my variables.
clearBytes() should clear the buffer and not data.
also bufferSize should be the actual bufferSize (which would be buffer.length) and iNBbytes the threshold at which you launch the check data (iNBbytes should be constant in your case at 13).
Also are you sure byte=='A' works? I don't know if it gives the same byte, but that i'm not sure.
And you don't need data as it's only used as a local variable here.
And finally, to copy the array you should use System.arraycopy which is faster and simpler too.

so the code would be :
`
int iNBbyte = 13;
byte[] buffer;
int bufferSize;

public boolean isStartByte(byte firstChar){
    if(firstChar=='A'){ // check if it works
        return true;
    }
    else{
        return false;
    }
}
public void clearBytes(){
    byte[] buffer=new byte[13];
    bufferSize = 0;
}
private void appendBytes(byte[] buf){
    System.arraycopy(buf, 0, buffer, bufferSize, buf.length);
    bufferSize += buf.length;
}
public void checkData(){
   tvAppend(errorCheck,buffer.toString());
}

UsbSerialInterface.UsbReadCallback mCallback = new UsbSerialInterface.UsbReadCallback() {
//Defining a Callback which triggers whenever data is read.
@Override
public void onReceivedData(byte[] arg0) {
        if(arg0!= null){
            if(arg0.length > 0){
                if (isStartByte(data[0])&& !startbytefound) { //look if its a new frame
                    startbytefound = true;
                    clearBytes(); // clears my buffer
                }

                appendBytes(arg0);

                if(bufferSize >= iNBbyte && startbytefound){
                    bufferSize = iNBbyte;
                    byte[] buf = new byte[iNBbyte];
                    System.arraycopy(buffer, 0, buf, 0, bufferSize);

                    checkData(); //process the data
                    startbytefound = false;
                }
            }
        }`

Maybe that would work. Check every passage with a Log.i() to see if everything's ok.

[jstonis]

hm..its still not working.. i mean its not displaying the string!

[jstonis]

for clearBytes..you mean to set the global variable buffer to new byte[13] right?

[jstonis]

Oh it looks like youre right..it never returns true for when byte==character..hmm im not sure how to do that conversion.

[OdysseusU]

Right i let a byte[] buffer you can erase byte[] it was an error.
Well you can log the bytes you receive (their values) knowing that you have an A coming just take the value and use that.
if for example you see that your 'A'=42 then use firstChar==42.

[jstonis]

Do you know why I am getting this error?

java.lang.ArrayIndexOutOfBoundsException: src.length=2 srcPos=0 dst.length=13 dstPos=13 length=2
at java.lang.System.arraycopy(Native Method)
at com.example.josceyn.walkerapp.UserDisplay.appendBytes(UserDisplay.java:108)
at com.example.josceyn.walkerapp.UserDisplay$1.onReceivedData(UserDisplay.java:137)
at com.felhr.usbserial.UsbSerialDevice$WorkerThread.onReceivedData(UsbSerialDevice.java:261)
at com.felhr.usbserial.UsbSerialDevice$WorkerThread.run(UsbSerialDevice.java:235)

[jstonis]

so it seems as though its trying to append to the buffer greater than 13 bytes..do i need to add some sort of check before appendBytes is called?

[OdysseusU]

Right thats because my buffer I create it with a length of 1024 to be sure not to have this kind of problem. You could add a check to be sure it never adds up to 13 or you can create the buffer at the beginning to 1024. Then for clearing you can do an Array.fill(buffer,0) to clear the buffer.
And thats why also i copy back the buffer to a new buffer of size 13 to get only the 13 bytes from the buffer

[OdysseusU]

So all you need to change in your code is in the clearbuffer change the new byte by array.fill and at the beginning create just once your buffer byte[] buffer = new byte[1024]
That should work now.

[jstonis]

So once found 13...then the extra stuff in the buffer, which is the next set of data is just gone? That doesnt seem like a good idea...

[jstonis]

Ohh actually nevermind.i understand now..sorry. i wl try this now and let you know. Thanks a loy

[jstonis]

Lot*

[jstonis]

it works..wooo..thank you so much!

[jstonis]

One more question for ya...i go to a new activity, a new intent, and its still running in the background...how do i stop the usb input..send a command to arduino?

[OdysseusU]

In the activity you should have the onPause method to override. In their you put a serial.close and on the onResume method you put the serial.open

[jstonis]

Okay sweet thanks. And when you want to recalibrate arduino, which is at the start of the activity, you could just send a 1 to the arduino telling it to perform thag, right?

[OdysseusU]

What do you mean by recalibrating? You mean reset? I think you can setDTR off and on for that

[GauravShahLumiraDx]

Update : Solved all this issues after Buffering the Data received.

Hello,

I am using this library to read data from arduino and split it to different textviews. I do not know from read call back it gives the whole string to tvAppend function or just each characters it received ?
I use settext instead of append in the last line of code(runonUi) and can not see any data on my textview. It gives some random data. I am sending a string of 25 different integers seperated by ", " from arduino. Ex. 1,2,3,4,5,6,7,8...25 and receiving the same in android. But I am trying to get the whole string in android, then split it to "1", "2"..and display it using settext / append on my UI's textview.
So that if I receive 1,2,3,4,5,6,7,...25
Results will be
Textview 1 = 1
Textview 2 = 2 and so on.

Question is:

1. Where do i split my data ? in tvAppend function or during received callback?
2. How the data is received ?
   public void onReceivedData(byte[] arg0) {
   String data = null;
   try {
   data = new String(arg0, "UTF-8");

            tvAppend(textView, data);

this gives characters received or the whole string ?

[GauravShahLumiraDx]

@felHR85
@OdysseusU
Please help if you get any idea on above question.

[Vijayapriyacs]

hi,
Serial port opened successfully but when i try to show serial read data ,nothing shows Toast at usbSerial read callback method.

[rohanagarwal94]

Hello, I am facing problem while reading with the help of synchronous API, could someone please post the code for it. Thanks in advance.

[Vijayapriyacs]

Will you please give more details about your issues.what kind of bugs. Hello, I am facing problem while reading with the help of synchronous API, could someone please post the code for it. Thanks in advance. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#37 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AXTW_HyCjMuW9dRNOkN09V9vAk0rtKW9ks5tYU6QgaJpZM4H2U5h> .

[imb-mgezer]

There are multiple solutiouns. If you don't have high performance-critical code, you can code something like below. You can check start and stop characters or byte size etc.

private StringBuilder sb = new StringBuilder();
private UsbSerialInterface.UsbReadCallback mCallback = new UsbSerialInterface.UsbReadCallback() {
    @Override
    public void onReceivedData(byte[] arg0) {
        String data = new String(arg0, StandardCharsets.UTF_8);
        sb.append(data);
        if (data.endsWith("#") || data.endsWith("#\n")) {
            //Do what ever you want
            sb.setLength(0);
        }
    }
};

[Selmeny]

There are multiple solutiouns. If you don't high performance-critical code, you can code something like below. You can check start and stop characters or byte size etc.

private StringBuilder sb = new StringBuilder();
private UsbSerialInterface.UsbReadCallback mCallback = new UsbSerialInterface.UsbReadCallback() {
    @Override
    public void onReceivedData(byte[] arg0) {
        String data = new String(arg0, StandardCharsets.UTF_8);
        sb.append(data);
        if (data.endsWith("#") || data.endsWith("#\n")) {
            //Do what ever you want
            sb.setLength(0);
        }
    }
};

Works like magic. Thank you.