Make BabelFish work for Arduino Micro #36
Comments
kervinck
changed the title
|
Gigatris uploads fine. I try to investigate on the software flow control side... |
|
Hmm, my version of Gigatris has 3 such bytes in it... The recent TinyBASIC only mentions it once, to name the address $13a0 where programs begin in memory. $13 = 19. A simple and safe hack is to change it to $14a0 and verify if the file loads. Edit: see attach such file: |
|
Thanks! The 14a0 version works fine. But in both of my downloaded TinyBASIC versions is no $13? Here is my log output from sendGt1.py for all three versions of TinyBASIC and frogstroll. I'm not so sure if it's up to XON/XOFF. Maybe I have to investigate more on the arduino side of this problem. |
|
The first failing BASIC says this: That is the segment with a0:13 in it. It looks like your debugging gets swallowed somewhere. Anyway, it is best to think of a solution that works the same on all platforms. From Googling I couldn't find a method yet to disable this behaviour. It is also not exactly clear who is responding to the XOFF byte: it could just as well be the serial driver on the PC-side, as far as I'm concerned, although not likely. Just brainstorming: An ideal solution would 1. work the same on all platforms (meaning: no runtime or compile-time flags), and 2. stays compatible with what is out there already. I'm not sure if that is achievable. We can always introduce escape sequences, but then both sides (Arduino and sendGt1) require a simultaneous update. Well, sendGt1 can always warn when it is sending a $13.. That's a start |
|
I see in the log that frogstroll also hangs. [I was confused by an earlier remark on that.] Hanging is consistent with embedded XOFF bytes. I'm convinced by now. Have to ponder on a simple fix... |
|
It isn't entirely clear if the XOFF is stopping the flow on the receiving side (Arduino) or in the sending side (PC). PySerial's _ _ init _ _ documentation suggests that the port has flow control disabled by default. But who knows what the underlying platforms are really doing... I would not want to meddle with OS and library specifics: lets just accept the fact that some byte values are forbidden from being transmitted in general. There are several ways then to go about it. We can do an encoding using only ASCII characters (base64 or simple hex dump come to mind). Base64 is a bit complex. Hexdump (base16) doubles the transfer size. That is not really bad, but also not nice. And it breaks all compatibility. We can also introduce an escape character. For example the common convention of 0x7D ('}') as escape code and XOR'ing the next character with 0x20. This is used in PPP for example: [Unfortunately there is no magic byte value that doesn't occur in any GT1 file published so far: every byte value 0..255 is out there in the wild already] I slightly prefer the escape character over the ASCII encoding. It looks as if it can become a local change with some chance of smooth introduction. Full compatibility will still not be possible. But we could get close. At least we should aim that "new" sendGt1 scripts still work with "old" versions of Babelfish.ino. To make the transition smooth, we can change the Arduino code to let it signal first that wants escaped data. If we do it right, we do it in such a way that old sendGt1 scripts will fail on the request. For example by sending a negative number as requested byte count. |
|
Lets see if e53ed17 solves it? The idea is that using the newer sendGt1.py will at least warn for these bytes. It warns for XOFF, as XON doesn't seem to cause harm (which is strange in itself, but oh well...). If a file worked before, it will continue to work. When BabelFish.ino is upgraded as well on the Arduino, the two will work together and escape the control characters. Then there will be no warnings displayed. [ The old sendGt1.py will not work well with the new BabelFish.ino. It will likely give timeout errors for programs that have the 0x7E byte in them. I don't see that as a realistic configuration to worry about. ] |
|
Now with your modification to BabelFish and sendGt1... Case a) Including $13# to force the problem but it transferred fine: Case b) No $13# but it fails. I always have to manually reset the arduino micro after the failure. I tried USB2 and USB3 hubs and iMac direct USB3 port. ... maybe I have to add a display to the arduino for more in depth logging ;-) or change to another arduino I have laying around. But my time is somewhat limited these days. |
|
Bummer and completely unexpected. I think I'll order an Arduino Micro so I can also test. Although it can also be a Mac-side thing. Wait, that can't be the case if a Micro reset is required to restore state. It is probably possible to craft a GT1 file that draws some recognisable progress to the screen, that could even be text.. And then send each byte, followed by a screen update. I'll see if I can make something like that, but it will be somewhere next week earliest. |
|
I just created such file. See attached. Also in the repo as Utils/SerialTest.gt1, including the Python program to generate the file. The result after sending should look as the attached screenshot. If it stops earlier, the value of the last successful byte will be shown.
|
|
Sorry for the delay. I am very busy at the moment. Family things ;). But SerialTest.gt1 works fine on Arduino Micro from start to finish. I tried it several times. ;( |
|
Interesting! If you have a specific file that causes you problems, would you mind to attach it here in order to freeze a reference point? TinyBASIC.gt1 keeps changing and growing all the time, as most of my focus is there this summer... Incidentally, I'm about to pick up an Arduino Micro this afternoon. But I'm also preparing vacationing, so I'm not thinking about this issue every day :-) |
I sent over TinyBASIC a couple of times on my own Arduino Micro setup which looks like this:
[Note that the red wire (Vcc) is coming out of my plug and is connected to the breadboard, but only so that I don't have it dangling. It isn't connected electrically.] [Edit: forget all I wrote about periodic reliability problems (I already deleted that part of this comment). The wiring got messed up in the photoshoot. ] |
kervinck
changed the title
|
Good news! I can reproduce the issue exactly as reported. I went back to the files posted on the forum, and found one in this posting. I attach it here, renamed with the same "_2" extension as reported above. |
The Arduino Micro hangs up after having processed a segment of 256 bytes. The message ":Loading 256 bytes at $500" means that 256 bytes were actually received from the host and are being passed on to the Gigatron. This succeeds as well, and the Arduino asks for the next segment header of three bytes. This request for 3 bytes is even answered by the host, but the Arduino doesn't see it. After a few seconds it gives a timeout. As if it stopped listening to the host (but sending to the host continues to work). The interesting thing is that if I split, in Python, the writing of those 256 bytes in two calls (on the host side in sendFile.py), this hangup doesn't occur. Somewhere in the pipeline there is a buffer that decides that it must stop working after having received 256 bytes at once. This could be the Arduino Micro firmware causing this: the obvious suspect, because none of this happens with the Arduino Uno that does serial/USB in a completely different way with an extra chip. However, sadly, the story is not complete: Terminal.gt1 also has (is) a 256-byte segment, and this one works: [Edit: in hindsight, the 256-byte segments reported here were transmitted over USB as larger blocks, due to the insertion of escape codes] |
|
Some notes after trial and error:
|
|
I found an issue in Arduino's USBCore.cpp library for USB handling on the ATmega32U4 processor. The Arduino code doesn't follow Atmel's datasheet with respect to clearing bits in the UEINTX register:
USBCore.cpp clears them at the same time, and that doesn't always work. Applying a workaround for it in the sketch solves the problem. Solutions in 94d167d and onwards. I'm confident enough to remove the whole escaping mechanism again from BabelFish.ino and sendFile.py. This simplification is not backwards compatible. However, sendFile.py will give an error to notify that an upgrade is needed. Not nice, but I shouldn't have added the escaping mechanism in the first place as the problem lies elsewhere. |
|
About 256 byte remarks above: in many of the cases the actual transmission was larger due to inserted escape characters. I missed that. It looks quite reproducible: send data in a multiple of 64 bytes, and USBCore.cpp gets stuck. There is a similar issue reported in the Arduino repo: https://github.com/arduino/Arduino/issues/6669 "Arduino Micro USB Serial cannot receive more than 255 data once" which points to arduino/Arduino#6886 "Handle receiving a ZLP in USB_Available" However, the proposed solution doesn't solve the issue we're seeing here. |
|
Submitted https://github.com/arduino/Arduino/issues/7838 (Arduino Micro USB Serial cannot receive exactly 64 bytes at once) on the Arduino project |
The dissassembly file can show comments when they are injected with
the C() function. But this is a bit ugly: it is harder to read in
the .py source than regular comments, while regular annotations
don't show up in the .asm file. This leads to two views on the code.
That isn't helpful.
A much better solution is to produce an old-fashioned listing of
the source code, along with the assembly result on the left. This
is possible with the Python inspect module!
Change: Replace .asm output files with .lst files.
For dev.rom, we now have the Python source lines listed in dev.lst.
Almost all C(xxx) calls could be eliminated between enableListing()
and disableListing().
The justification for the explicit enable/disable (instead of "always
on") is that introspection is rather slow. >90% of the ROM file is
storage area generated from the same macro. With that, looking up
the line number for each word in the storage area, and then deciding
that it was already listed, is very slow..
(Note: This can be remedied with more refactoring that eliminates
the _linenos[] list. Maybe later...)
The transition is quite a bit of work, so for the older ROM files
we don't change the output (for now).
Snippet from dev.lst:
-----------------------------------------------------------------------------------
0b2d fc32 bra .sysSb#38 3062 bra('.sysSb#38') #36
0b2e de00 st [y,x++] 3063 st([Y,Xpp]) #37
3064 label('.sysSb#35')
.sysSb#35: 0b2f 0200 nop 3065 wait(38-35) #35
0b30 0200 nop
0b31 0200 nop
3066 label('.sysSb#38')
.sysSb#38: 0b32 0124 ld [$24] 3067 ld([sysArgs+0]) #38
0b33 2001 anda $01 3068 anda(1) #39
-----------------------------------------------------------------------------------
It appears some Arduino versions implement in-band, software flow control (XOFF decimal 19 and XON decimal 17). This means you can't send arbitrary binary payload data over the line.
See also https://forum.gigatron.io/viewtopic.php?p=228#p228
One direction for a solution is to disable this feature. That sounds a lot better than going to a 7-bit encoding. [This is a bit ironic, because sendGt1.py only exists because flow control was missing on the Arduino Uno in the first place.]
Another idea is to send uncompiled .h files... They are text and not difficult to convert on the receiving end.
[Edit to future readers: After LOTS of debugging, the issue is NOT related at all to control bytes or XON/XOFF flow control, but concerns with the length of the data messages ]
The text was updated successfully, but these errors were encountered: