Implement connecting to OSD via Cleanflight in MWOSD GUI

[btimby]

@ShikOfTheRa I am planning to modify the GUI to add the following:

• Add a "Connect via Cleanflight" checkbox near the COM port selection.
• When checked, the GUI will connect to the indicated COM port and verify that Cleanflight is connected.
• The GUI will then get a list of USART ports connected to Cleanflight and ask the user which is the OSD.
• The GUI will then issue the CLI commands to enable Serial Passthrough mode in CleanFlight.
• Proceed with connecting normally to the OSD.

Think of this as the parallel of what BLHeliSuite does for ESC passthrough with CleanFlight. Do you have any thoughts on the above?

I started to work on this by trying to compile the GUI using the Processing IDE. Any tips you can give me to get the correct libraries installed (version numbers etc.) or any other information you can give for developing the GUI? For example, I am having trouble locating a version of controlP5 that allows me to compile the GUI without "undeclared function" errors.

I plan to record my findings as a 'Developer' document for the docs directory. This should helo others that want to hack on the GUI. I will be traveling next week and will use this as my entertainment in the hotel room. Pull requests will be inbound.

[ShikOfTheRa]

Hi - there is MSP passthrough code already that may be of use.
Have a look at gui.cfg and change passthrough to 1 - maybe some variation of that? when enabled - provides CF port 0,1,2 or something?

Use 2.0.4 from here : https://code.google.com/archive/p/controlp5/downloads?page=1
I thing 2.0.4 is the latest featured release.

Also, for awareness someone is in the process of developing a chrome app. he is making fantastic progress, but will take some time as there is a lot of functionality to replicate.

[ShikOfTheRa]

Just to add... I think you need to find something more interesting to do in your hotel room!!

[btimby]

Yeah, I am lame...

The passthrough config option in the MWOSD codebase appears to originate from this:

http://www.multiwii.com/forum/viewtopic.php?p=62207

Which is a proposed extension to MultiWii MSP commands that AFAICT is not yet accepted (not in github). However, I like that idea, rather than using the CLI to initiate passthrough, MSP is a better way to do it programatically (via the MWOSD GUI).

Therefore, I am modifying Cleanflight to support Passthrough initiation via MSP, and will issue a pull request to CleanFlight and Baseflight. Meanwhile, I will extend the changes already in MWOSD GUI to work with this.

The difference between the above proposal and my changes is that the serial port is configurable. The MultiWii version expects the OSD serial port to be #defined when MultiWii is compiled. My MSP changes require the port ID to be specified at runtime as a param to the MSP command. This is due to the fact that Cleanflight is not configured at compile time, but at runtime.

In any case, if MultiWii does eventually add this feature, my changes will not stop it from being supported by MWOSD GUI.

Also, I found the tools used to compile the GUI (Processing and controlP5) are very out-of-date. Processing 3 is unsupported as is the latest version of controlP5, which is 2.2.5. I reverted to Processing 2 and controlP5-2.0.4 and was able to compile the GUI (thanks for your help!). I have started a Development.md file to document the process of setting up a development environment for the OSD.

One last thing... The OSD name, is it scarab or MWOSD?

[ShikOfTheRa]

I do know some that are using the multiwii passthrough so if can leave that available to them in some way, that would be ideal.I think adding a com port into the MSP request would be perfect solution.

I think no more multiwii development now.... its been dead for a long time. Its a shame, but can't keep looking back! I musty admit I still use it. Have copter with the boards and they fly nice - so no need for me to upgrade just yet.

Good on the processing!

The name...
So I joined kv team a couple of years ago and made a few changes for my buddies who were all running Scarab copters and put a nice logo on for them. Mainly testing a few enhancements that I hoped to put into kvosd. Henri (Kataventos) went nuts for some reason that I still don't understand to this day and remove me from the team. So no real choice but to create a fork and do my own thing. It was only ever intended to be a few mods for myself and the guys but just kept adding to it..
As it got bigger and KVteam development seemed to have stopped I was encouraged to share improvements with others, but felt that having a manufacturers brand name like Scarab on it wouldn't get exposure so came up with the crap name of MWOSD - for MultiWii OSD
So in my mind... Scarab OSD is development. MWOSD is release.

Now it covers so many controller options I am thinking maybe MWOSD is no longer the right name! Id like to call it scarab - but would have to talk with the brand name owner. and do want it to remain neutral although he has been the most generous of supporters. Lucky if get $10 a month in donations and I am seriously out of pocket with all the kit I've purchased for this so having a copter from multiwiicopter was a dream

[btimby]

Do you have any desire to see things updated to newer tool versions?

[ShikOfTheRa]

For GUI - don't care!
For the atmega, I want to keep it simple and avoid what happened with Multiwii.

[btimby]

@ShikOfTheRa, my goal is to make working with the OSD a bit easier. I have in mind two methods to accomplish this:

1. Allow the user to open the MWOSD GUI and connect via their FC (passthrough). This would avoid the extra step of enabling passthrough via their FC GUI. The MWOSD GUI would enable passthrough in a single step. For this I would like to put the function in the MWOSD GUI rather than the config file. I would love your feedback on this and how you think this could be accomplished.
2. Allow the user to flash firmware from the MWOSD GUI. This is important because uploading via passthrough using Arduino is not possible as avrdude relies on a hard reset (DTR) which is not connected to the FC. The GUI however is able to soft reset the OSD. I have implemented this by adding an UPDATE button to the GUI.

Right now you have to find a .hex file to upload, which you can grab from Arduino's console output when you do a verify of the OSD firmware code. You can then open the MWOSD GUI, connect, then click UPDATE and browse to the .hex file Arduino generated.

My idea is that we could possibly provide one or several compiled .hex file firmware so users could avoid using Arduino altogether. Is this something you are willing to do? Do you think this is feasible? I am thinking some basic popular sets of options. Anything that is not precompiled would have to be user generated after customizing config.h using the method above (Arduino verify).

This would mean that the OSD could be installed physically into the copter, then flashed and configured in place using only the MWOSD GUI (when using a compatible FC).

I committed a working firmware updater to my fork. This is by no means DONE, just WORKING. Be careful, I have bricked my OSD several times while testing and had to recover via burning the bootloader :-).

btimby@60d7b2c

My next step is to implement 1 above and then make sure I can load firmware via FC passthrough. The only complication is that the baud rate changes for firmware uploading (57600 vs. 115200) and I am not sure how I will handle this, but I have some ideas.

[btimby]

Also, I have started a Chrome app, mwosd-configurator is the working name right now. I plan to implement only a few things:

1. A connection screen initially displayed that allows the user to choose their serial/passthrough options.
2. An optional firmware version check that offers to update the OSD if the firmware is old.
3. The firmware updater.

However, implementing the rest of the GUI is a daunting task. Do you have a handle on what is most important? The only stuff I use is Font upload and the toggles for enabling/disabling boxes.

[ShikOfTheRa]

For 1:
I’d suggest sending an MSP passthrough connect request at connect time. OSD will just ignore it if connected directly.

For 2:
Impressed!

Maybe use one of those “travis” online build type applications that can automatically compile with preset defines. Dumps them into a folder on github that will ship with GUI?.

For Chrome App – there is someone already working on this and is in quite advanced stages. Suggest maybe hook up with him. Error414 on fpvlab or github

Personally I use the GUI simulator features a lot – it’s one of the prime tools for developing the OSD. I’m happy to keep maintaining the current crap just for that purpose though. I have never used it though so only think thats what it can do...!

Not sure if normal end users need that. Error 414 has a lot of the screen stuff working now.

[CrazyCoder]

@btimby Does it work for you with the unmodified bootloader? Standard bootloader jumps to the app without waiting for the upload and software reset doesn't trigger upload mode (only hardware DTR does).

I've been experimenting with optiboot bootloader (it works on 115200 by default, so no need to change the passthrough port speed). I've modified optiboot to trigger upload mode on watchdog event, then I modified soft reset code in OSD to trigger watchdog (jmp 0x3E00 also works).

Now I can trigger bootloader upload mode via OSD configurator. While in this mode flashing via Arduino IDE works for me without DTR. I'm experimenting with FTDI at the moment, didn't try via the FC passthrough yet.

My test setup:

[CrazyCoder]

OK, so I've flashed OSD via serialpassthrough.

Modification to optiboot: In https://github.com/Optiboot/optiboot/blob/master/optiboot/bootloaders/optiboot/optiboot.c#L484 remove _BV(WDRF) condition so that it goes to upload mode on watchdog reset.

Modification to OSD resetFunc(): instead of jmp 0 trigger watchdog reset via

  WDTCSR = _BV(WDE);
  while(1);

This method has pros and cons.

Pros: once you trigger reset, it will stay in upload mode until you power cycle, no need to catch the exact moment to upload, can be easily flashed via Arduino IDE. There is no delay on normal boot which is a plus.

Cons: modification to OSD code is needed. If you flash unmodified OSD code by accident, you will no longer be able to trigger upload mode via software. Another con is that it stays in upload mode after you flash the new code or trigger a reset via OSD configurator, you have to power cycle it to boot normally, this one is minor, though. That is because uploader code in the bootloader triggers reboot via watchdog as well, and loops back into upload mode.

Another possible solution without OSD code modification would be to modify the bootloader so that it waits for upload 1-4 seconds on every boot. This will delay the boot a bit, but you will be always able to upload the new code, even without triggering reset via OSD, just catch the moment after OSD power cycle. Some FCs power OSD when you plug in USB. When OSD is powered by USB, you will have no way to power cycle only OSD without FC, so passthrough mode will cancel on FC reset.

For the clean solution we can probably separate normal restart via the stock resetFunc and introduce the new reset MSP command which will trigger watchdog reset and go into bootloader. This would require some tool or additional button in the UI to send this new command to put OSD into upload mode.

And the worst thing is that bootloader modification is absolutely needed, this is not something that anyone can perform easily. Stock Arduino bootloaders go into upload mode only on hardware reset via DTR pin (both stock optiboot and ATmegaBOOT_168_atmega328.hex used in Pro Mini and in Micro MinimOSD out of the box). Please correct me if I'm wrong with this assumption.

If anyone wants to give it a try, I can give you a jump start to save some time, here is the modified optiboot bootloader.

[btimby]

@CrazyCoder It is great to see your progress. Here is what I know.

I modified the UI to perform flashing. I compile via Arduino, then nab the hex and can flash it to the OSD. This is done using the software reset, so with the stock bootloader, soft reset DOES put the OSD into programming mode.

However, my current hurdle is that the BAUD rate is different. So, the process in the UI flashing code is:

1. Send 3 soft reset MSP codes
2. Close serial port & reopen at 57600
3. STK500 to load firmware
4. Board reboots and process is complete, the user must hit connect again in UI.

The problem with this is during passthrough, the serial BAUD is controlled by CleanFlight and there is no mechanism to adjust it.

It was suggested by someone during passthrough implementation that it may be necessary to add HAYES control code handling to CF. This would actually be really nice for several reasons.

1. HAYES control codes could be used to terminate passthrough, obviating the need for hard reset of CF when finished with OSD. For example OSD GUI could send command codes when user hits "Close Comm".
2. HAYES control codes could be used to adjust the baud rate to allow for flashing the OSD.

I will share with you the code and procedure to perform the flash via UI that I describe above.

[CrazyCoder]

@btimby Interesting. Does it mean that your OSD had some different bootloader rather than ATmegaBOOT_168_atmega328.hex which doesn't go to upload mode after software reset? I guess it's not something that can be relied on since the bootloader can be different depending on OSD hardware and batches. If it works for you, it doesn't mean it will work for other OSD hardware which has the stock Arduino bootloader of the recent version.

[btimby]

I did not modify the boot loader. I am testing with a cheap Chinese clone MinimOSD from Goodluckbuy. Admittedly I don't know what firmware was on it originally, but I have since used Arduino to unbrick it via ICSP and the firmware flashing still works.

[CrazyCoder]

Which bootloader did you flash via ICSP when unbricking? According to ATmegaBOOT_168.c code in Arduino 1.6.8 distribution, it doesn't go into upload mode on reset. I'm puzzled. Maybe older Arduino versions flash a different bootloader version which waits for upload on every reboot?

[btimby]

I am using Arduino 1.0.5 and I clicked "Burn Bootloader". I had to solder pins to MinimOSD and make a cable for my USBasp beforehand.

[CrazyCoder]

Which board did you use in Arduino IDE? I've checked 1.0.5 bootloader source and it should work the same:

    // Check if the WDT was used to reset, in which case we dont bootload and skip straight to the code. woot.
    if (! (ch &  _BV(EXTRF))) // if its a not an external reset...
        app_start();  // skip bootloader

[btimby]

Arduino Pro or Pro Mini (5V, 16Mhz) w/ ATMega328

[btimby]

I am new to Arduino, usually I have to Google things like the board setting before flashing etc. Above is what my board was set to from the last time I flashed the OSD.

And whatever bootloader it came with worked...

[CrazyCoder]

OK, that is really weird. Pro Mini is set to flash ATmegaBOOT_168_atmega328.hex bootloader which is compiled from ATmegaBOOT_168.c which doesn't go to upload mode on soft reset. If anyone has an explanation, it would be great to see it.

But if you say that it works, it's probably a good thing. How much delay is there while you can upload?

[btimby]

Have you looked at the reboot method that OSD F/W is using? IIRC it does a jmp 0 I don't know if that changes anything.

scarab-osd/MW_OSD/MW_OSD.ino

Line 659 in 7fbfcd7

void resetFunc(void)

[CrazyCoder]

Yes, I did. I've tried different methods. I may need to try with the stock bootloader again since I've flashed optiboot already (to use 115200 speed and avoid changing speed when configuring and flashing). jmp 0 doesn't trigger upload mode with optiboot. It starts the code right away, there is no LED flashing 3 times indicating that it's waiting for upload.

Oh, I can see that the code that skips upload mode is inside WATCHDOG_MODS define. I guess that stock bootloader is compiled without this define, so it goes into upload mode every time. That would explain it. I should have looked more carefully.

[btimby]

And honestly, the metric I am using for "success" flashing via my STK500 code is that I am receiving correct ACKs from the OSD. While developing the code, I did get failures when I was not following protocol. Fixing those allows the entire process to run through and I was receiving ACKs as the documentation described. Also, the fact that I bricked the OSD several times when my STK500 code was incorrect tells me I was in fact programming the chip. But I did not verify beyond that.

[btimby]

OK, try the stock bootloader, my experience says it works. Here is how I am doing flashing via the GUI:

First, you need Processing GUI, an older version. You also need a library called ControlP5. I wrote a simple document describing how to set up for hacking the GUI:

https://github.com/btimby/scarab-osd/blob/master/DOCUMENTATION/Development.md

The source code for the GUI is here:

https://github.com/btimby/scarab-osd/blob/master/OTHER/MW_OSD_GUI_SOURCE/MW_OSD_GUI/MW_OSD_GUI.pde

Open that with Processing. You should then be able to run the GUI from source code on your system.

I checked in code to perform flashing from the GUI, to use it do this:

1. Open Arduino and open the OSD firmware as usual.
2. Instead of clicking upload, click verify. This compiles the firmware, you will see hex file name in the output at the bottom. Note this path or move it somewhere (I moved it).
3. Run the GUI from Processing, connect to your OSD (I used an FTDI for all my testing). Click the update button at the bottom of the GUI, browse to your hex file.

Done.

This is the code that performs the update:

btimby@60d7b2c

[btimby]

If you have problems with processing let me know and I can try to help and also update that document. I only recorded the problems I ran into making it work. Newer versions of Processing won't work as the language has changed significantly.

[btimby]

Also, the goal for firmware is to compile some flavors for users, allowing them to select those from the GUI obviating the need for Arduino from an end user perspective.

My (our?) ultimate goal is to allow this flashing via passthrough.

[CrazyCoder]

Thanks, I now understand why and how it works with the stock bootloader. Will flash it back and give it a try with your patched GUI version. I've already compiled GUI via Processing before, so I know how to do it. I still do some modifications to config.h for my personal preferences, so building a .hex manually. For the end users it would be much easier to have a set of precompiled hex files with the most common options, advanced users will build their own hex files with the options they need.

[CrazyCoder]

Close serial port & reopen at 19200

I guess there is a typo, default upload speed for Pro Mini is 57600 and according to your code you open the port at 57600.

[btimby]

:-) Yes, 57600 is correct. I edited my previous comment.

[btimby]

These are my thoughts on HAYES control codes:

https://en.wikipedia.org/wiki/Hayes_command_set#Hayes.27_solution

Specifically the escape sequence. This would allow you to switch between data (passthrough) mode and command mode. So that a limited number of AT commands could be handled by CleanFlight (reboot, adjust speed etc). Then the OSD GUI could drive the entire process. Timing would be a potential problem as Hayes escape sequence requires a one second pause after the +++ sequence. But it could potentially work like this:

1. Send reboot MSP command to OSD. Possibly we could have a second reboot MSP command that would sleep before rebooting.
2. Send +++
3. Wait 1 second
4. Send AT &B57600 to adjust Cleanflight's passthrough BAUD rate. This would replace the need to close and open the serial port.
5. Start STK500 firmware upload.
6. Verify upload, OSD reboots.
7. Send +++
8. Wait 1 second
9. Send AT &B115200 to adjust Cleanflight's passthrough BAUD rate.
10. Reconnect to OSD (now with new firmware).

[btimby]

Actually, the +++ sequence only needs to exit passthrough mode. Passthrough can then be re-established at a different baud rate. CleanFlight won't need to handle any special control commands.

[CrazyCoder]

Trying with the stock bootloader, no luck so far. If you click just RESET in the GUI, does the OSD LED flash indicating it's in bootloader? For me it doesn't happen. OSD just reboots. On power on the LED flashes once, so it goes through the upload mode for a second.

[CrazyCoder]

I used an FTDI for all my testing

Did you actually test it via FC serialpassthrough? Did you have DTR pin connected from FTDI?

[btimby]

No, I have only tested via direct connection to FTDI.

I will test again here in a few minutes to confirm. I last worked on this 20 days ago according to github ;-). So my memory may be corrupted. I am almost done implementing escape sequence for CF passthrough.

[CrazyCoder]

If you have DTR connected, it will go to bootrloader on software reset. When DTR is not connected, it just restarts OSD code and there is no chance for upload.

[btimby]

Ah. So just having the pin connected changes the process of software reset? Below is what I am testing with. Looking at it now, it is a KV modded one probably from Witespy not GLB.

So if I disconnect DTR, it stops working?

[CrazyCoder]

So just having the pin connected changes the process of software reset?

Yes, that is what I'm observing. With DTR the LED flashes on soft reset from OSD GUI, without DTR no LED flash and it starts fast LED blinking right away skipping the upload mode.

[btimby]

Will confirm now.

Tested with DTR connected, I see the following on the console.

approved
restarted
closed
updating
serial open
entered programming mode
writing data: 74477
writing 0 to 127
...
writing 74368 to 74476
done writing
Success

And with DTR cut.

approved
restarted
closed
updating
serial open
ACK Failed

[proggod]

I don't suppose you guys want to get this working on raceflight as well? I'm willing to help out :)

[btimby]

It seems the reboot method jmp 0 is not working without DTR connected. I have read that another way to restart the Arduino is using the watchdog timer. Apparently this will allow upload after reset? What if MSOSD used this reboot method instead.

https://ariverpad.wordpress.com/2012/02/26/uploading-arduino-sketches-wirelessly-using-a-bluetooth-breakout-board/

https://github.com/WickedDevice/SoftReset

[btimby]

@proggod thank you for the offer. Please read this discussion as we are currently just trying to "get it working" at all. Working with Raceflight seems very far off given current progress ;-).

For raceflight, serial passthrough would be needed. A first version of this is in betaflight already (but not cleanflight). I have since added two more functions in my fork:

1. The ability to start serial passthrough via MSP (not just CLI).
2. The ability to exit serial passthrough via an escape sequence.

With this full implementation, it should be possible from the FC standpoint to do what we want. However, getting flashing working on the OSD itself is the blocker ATM. For this we have a proof of concept that is able to flash the OSD, but only if DTR pin is connected. Removing this necessity is required to allow flashing via FC passthrough.

[CrazyCoder]

I have read that another way to restart the Arduino is using the watchdog timer. Apparently this will allow upload after reset?

  WDTCSR = _BV(WDE);
  while(1);

The above code should do the same, however it doesn't seem to work properly with the stock bootloader. It goes in some weird state with LED flashing without stop and upload not working. Does it work for you?

Exactly the same code worked for me with the modified bootloader as I've described above.

[btimby]

Not tried it yet. I am still reading. I have read that using WDT for rebooting can cause an infinite reboot cycle as the watchdog is not cleared by some bootloaders. Perhaps this is what you observe?

I have read the possible solutions to this are to modify the bootloader so that it clears the watchdog or to use a long WDT interval and clear it first thing when your application starts. However, this probably would not help for uploading.

[CrazyCoder]

Well, if bootloader modification is required, see my comment above for the whole process. It's confirmed to work via FC serialpassthrough,

[btimby]

Seeming that way.

[btimby]

For reference, here are the locations of the ICSP pads on a Micro MinimOSD.

http://static.rcgroups.net/forums/attachments/4/8/9/0/7/4/a7808257-164-BOOTLOADER%20FLASH.jpg

[btimby]

@CrazyCoder what about jumping to the bootloader address rather than jmp 0?

[btimby]

@CrazyCoder also, I have not been able to determine WHY just having the DTR pin hooked up makes a difference? It is not like I am using flow control or changing it's state. avrdude does via an ioctl() but my toy firmware flasher does not.

Could this simply be jumpered to 5v or something?

[CrazyCoder]

what about jumping to the bootloader address rather than jmp 0

According to the docs, the stock bootloader address is 0x3E00. I've tried it and it didn't work with the stock bootloader. Worked with optiboot. Just noticed that stock one could be larger, so the address might be 0x3C00. Can you try it?

Could this simply be jumpered to 5v or something?

That might work, I've read that wiring directly to 5V is a bad idea, you may need to use a cap or a resistor. Not an expert here and didn't do my googling in this direction.

[CrazyCoder]

I've tried different addresses and it doesn't seem to work with the stock bootloader. No LED flashing at all.

[btimby]

This seems to be jumping to the bootloader. I see some faint flashes before the led starts flashing quickly. It looks exactly like when I press the reset button.

static const void* bootloader = (void*) (0x7800);

...

void resetFunc(void)
{
    goto *bootloader;
} 

[btimby]

However my flashing code in MWOSD GUI does not work. But this could be because of something else.

I need to write some specialized tools for testing this. I know there is an MSP python library, I will start hacking something together. The MWOSD GUI is too cumbersome to use for testing.

[CrazyCoder]

I've tried your reset code and it doesn't seem to work. It just restarts without one bright LED flash like it does after power on.

[ShikOfTheRa]

btimby - for serial testing sometiems I use processing, but just copy out the bit I want. I hate processing, but in small chunks like that its actually quite quick and easy!

[ShikOfTheRa]

If no objections I'd like to close this in the next week as no updates for a few months. CAn be re-opened if work restarts