Store the git SHA of test-flown aircraft

[dewagter]

change:

• tag which airframes were tested in which commit

personal use:

• commit-count since last flight
• instant-revert to flying code
• history of tests

users:

• provide information on what was tested in a release

project:

• airframe age computations & cleanup
• release candidate selection

[gautierhattenberger]

I'm not really found of doing this since it may cause a big mess if you have local changes, and if the checkout works, it will most likely require a make at top level.

[dewagter]

Yes indeed. Moreover you will then be in detached head mode, which might freak out some people. And a make will be needed so you need to go to the prompt anyway. A copy SHA could be better, but that didn't work out in OCAML. Any tips?

[dewagter]

The checkout option was removed in favor of a GITK button, which also allows to check out the revision but allows much more.

[flixr]

Please also keep in mind that this has to be something optional and has to fail gracefully if you don't have a git checkout (e.g. used the tarball).

[podhrmic]

Can this be made optional? For example an optional tag in the airfame config file? Seems unnecessary to show it for all users in the pprzcenter (I assume that is what the glade edit is for). At least in our workflow we never work in the detached state - instead we have "flight_tested" branches. It probably doesn't help with this problem, but at least we would like to keep our workflow if possible.

[dewagter]

@podhrmic

1. yes, you can keep your workflow: actually having flight-tested branches is precisely the valuable information that is missing in master: namely: for a given airframe: "Was it ever flown by anyone, and if yes: using which code?"
2. about showing to all users: I disagree: I think conf should not just have a list of airframes but also a description and a SHA from whoever made that conf about what this conf item is about and if/when/how it was ever tested: To me this is something to show to ALL users so any new user can browse for recent airframes and see that even without git (even in the tarbal, since it is in conf)

[podhrmic]

@dewagter OK, sounds good to me.

[gautierhattenberger]

@dewagter the information you want to keep doesn't have to be in conf after all. What about adding a button to save at the end of a flight relevant information: conf of the airframe, git SHA, a dialog box to enter extra relevant information, log file name, the file in which data are saved (might be a DB in the future). This way we would have a kind of flight log book, that you can also share.

[dewagter]

@podhrmic @gautierhattenberger

Thanks for the feedback and tips.

• I also like to idea to have everyone click an anonymous button to some server after each flight, just to have an idea what is flying, if some tags are well tested or to see how much code misses in master to make things fly.
• I also like the idea of a (local/shared) logbook, preferably with logfile, ideally automatic: I already use a log->pdf tool: https://github.com/tudelft/paparazzi_log_parsing and if useful this can be shared, moved to paparazzi and expanded.

But to me, this is DIFFERENT from the current pull request.

• We already store conf/airframes in master so CI can check if they still compile. We do this because it is important for the project that "as-much-as-possible-of-the-project" would still compile. This is great and very important.
• But whoever adds a conf item, or changes one at this point does not tell ANYTHING about what it does. A user must now look how old code is, who made it when to GUESS if it ever flew in the first place and a long search through gitk is needed to find when it is likely to have been flown (if it even represents an actual airframe at all).
• Almost all (our) pull request were test-flown at some point. We have many people flying a lot of things. But whenever a pull request is made of a test-flown feature, very often changes are needed, which is normal and which is very good. But due to many constraints, most often the code is NOT test flown anymore afterward, meaning that as soon as a development enters master, is might already be broken.

SO:

• I would like that anyone who COMMITS a conf item, also adds a comment line and if/when it was tested (that is why it belongs IN CONF)
• when pull requests are made, code should be cleaned up (great and important) but the commit that has flown should be saved and changes should be added ON TOP. So finding what broke a working airframe can be at least eased with git tools. Or if a sudden demo is needed, one can revert to that code without need for long searches nor phone calls or emails.
• upon releasing a new paparazzi release, automated tools can help to ensure the age of latest test are not too old.

In other words: this is about TESTING (all targets of an airframe, and all its modes), while the idea of gautier is more about logging, and other discussions are about keeping track of who flies on which branch. For me 3 very different things that can complement each other and live in parallel.

IMHO the most pressing one for me is the TESTING and SHARING/STORING test results.

[podhrmic]

@dewagter In some cases Hardware in the loop - HITL (#1831) can be useful for testing, but it is limited to using an external INS (Vectornav, Xsens).

I am curious what kind of errors you typically encounter when trying to fly (i.e. what errors are we trying to prevent by testing)?

1. Code doesn't compile (missing dependencies etc) - that is easy to catch (CI)
2. Flight plan errors, control loop errors (gains, orientation, logic errors, etc) - we can catch that in NPS
3. Driver issues, platform specific problems, some hardware problems (CPU utilization, numerical errors, timing/scheduling, wiring issues) - in general this should be possible to catch with HITL
4. computer vision algorithms, airframe tuning, esc calibration - this would have to be flight tested (currently no way to simulate it). On a side note - Pixhawk uses Gazebo and it apparently provides data for camera, so maybe that would be useful to have?

I am trying to break down the issues because it is not always practical to test fly (time, resources, place). For example, we had lot of problems with 3) so we had to upgrade HITL to catch them and cut down the cost of testing. If you are encountering lot of problems of a specific type, maybe it is worth developing some testing tool to prevent them?

Just a thought. Otherwise as I mentioned, having an optional SHA tag is fine for me, but for practical reasons I don't think we'll be using it.

[dewagter]

Most often the problems are of type 3) and 4).

E.g. optic flow stabilization is developed and tested. Then a pull request
is made. In the mean time thread priority is changed. The pull request is
accepted, but after the thread changes it behaves differently.

But also: indi is upgraded, and test flown on bebop. A pull request is made
with the new code. There are 10 bebop files in master. Which is the one
that was test-flown with the latest PR?

My point being: I know not everyone will go out to test every development
step. I'm not asking to either. CI does SHARE what was tested: compiling of
all conf_test.xml airframes + nighly all conf. 2),3),4) do not share yet,
but are equally... wait maybe for an autopilot even at least AS important.

Rephrased: "SHARE your test results." Any alternative idea is welcome to
achieve this.

Just out of interest I wonder when you say: "I don't think we'll be using
it." Is there any practical reason (e.g. airframes not in master) why you
might not want to inform the community which airframes are recently tested
(e.g. you simply test too often). Or did you mean you would never click the
button: see all paparazzi changes since my last flight?

-Christophe

On Fri, Nov 4, 2016 at 6:18 PM, Michal Podhradsky notifications@github.com
wrote:

@dewagter https://github.com/dewagter In some cases Hardware in the
loop - HITL (#1831 #1831)
can be useful for testing, but it is limited to using an external INS
(Vectornav, Xsens).

I am curious what kind of errors you typically encounter when trying to
fly (i.e. what errors are we trying to prevent by testing)?

1. Code doesn't compile (missing dependencies etc) - that is easy to catch
   (CI)
2. Flight plan errors, control loop errors (gains, orientation, logic
   errors, etc) - we can catch that in NPS
3. Driver issues, platform specific problems, some hardware problems (CPU
   utilization, numerical errors, timing/scheduling, wiring issues) - in
   general this should be possible to catch with HITL
4. computer vision algorithms, airframe tuning, esc calibration - this
   would have to be flight tested (currently no way to simulate it). On a side
   note - Pixhawk uses Gazebo and it apparently provides data for camera, so
   maybe that would be useful to have?

I am trying to break down the issues because it is not always practical to
test fly (time, resources, place). For example, we had lot of problems with
3) so we had to upgrade HITL to catch them and cut down the cost of
testing. If you are encountering lot of problems of a specific type, maybe
it is worth developing some testing tool to prevent them?

Just a thought. Otherwise as I mentioned, having an optional SHA tag is
fine for me, but for practical reasons I don't think we'll be using it.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1921 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAd6fImZ3gZ_BgHhF82xen-rePnuhOmFks5q62jzgaJpZM4KmEc0
.

[podhrmic]

Just out of interest I wonder when you say: "I don't think we'll be using it." Is there any practical reason (e.g. airframes not in master) why you might not want to inform the community which airframes are recently tested (e.g. you simply test too often). Or did you mean you would never click the button: see all paparazzi changes since my last flight?

it is mostly a logistical issue - we use a paparazzi fork, and although I am slowly merging our code with master, it is not ready yet (so airframes and certain custom modules not in master). When this happens it will be more relevant, although at this point I think we would use the latest stable branch, and update periodically to the new stable release. Also, we have different people developing the code (mostly me), and different people flying (pilots, operators). Operators aren't necessarily familiar with the code itself nor github, so the simpler for them, the better. Plus we are benefiting from much simpler code (using only a single MCU and no fun stuff like optical flow etc:-/ ), so we can fully test with HITL.

[gautierhattenberger]

@dewagter for this I don't think doing git checkout or diff is nice. But does it help if at least the SHA is stored ? Also it means that when using a tarball release, the function should properly fail and maybe just return the current version.

[dewagter]

@gautierhattenberger: we can do diff (of only the files used by the
airframe) and checkout externally.

Yes, only storing the sha (and preferably a comment/note) is OK.

I was thinking of an analyse button in the select-conf.py program that
would show the age of all airframes. I think this might help in getting rid
of old stuff and making sure during a release that at least a
representative subset of airframes was tested recently upon release.

-Christophe

On Fri, Nov 18, 2016 at 10:40 PM, Gautier Hattenberger <
notifications@github.com> wrote:

@dewagter https://github.com/dewagter for this I don't think doing git
checkout or diff is nice. But does it help if at least the SHA is stored ?
Also it means that when using a tarball release, the function should
properly fail and maybe just return the current version.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1921 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAd6fHush3akk1e_y-H2lnBsJGXn6Jg1ks5q_htGgaJpZM4KmEc0
.

[gautierhattenberger]

can you update the PR to only keep SHA/comment and fix conflict then ?

[podhrmic]

@dewagter can you resolve the conflicts here?

[podhrmic]

@dewagter any updates here?

@gautierhattenberger @flixr should we keep this open?

[OpenUAS]

All nice! Note that although airframe age computation or last testflown should should more indicative. since we have airframe years old, still fly well and even on a regular basis we should not be forced to push results else an airframe gets removed. maybe add a flag would be better like someting as "not testflown lately" This PR improvement will not hinder others so a merge would be great.

[dewagter]

If you add an airframe XML in master (which is not obviously called test/example), then most people will expect it existed one day.

If it existed, it probably flew one day. But it also might not fly well anymore in recent versions. (Especially vision based controlled stuff has this a lot)

My point: not working airframes are bad for the project.

If you simply tell which version worked for you, you provide extremely valuable information:

1 anyone can check it out quickly
2 you can check what changed (instead of checking everything)
3 you tell everyone which code is missing in master if your airframe is on a branch (see Delftacopter)
4 by seeing how old the code is, some people might be triggered to try new code
5 during refactoring of code during a pull request, the tested version (typically before the rewrite) might be stored

Problem:

• storing the SHA does not tell what was tested or how well it worked (1min versus 100h)

About "forced to push results"

There is no point in having an airframe in master which does not exist anymore, other than examples. If it still exists, but you prefer older code, then maybe we also want to remove the XML from master and link to the older version. And if you want to keep it in master, maybe you want to check if recent code also works for you.

[dewagter]

This is what it looks like:

[dewagter]

What else should I change to accept this pull request:

• I removed the checkout option which was not supported by everyone
• the conf is backward compatible: opening a conf with release info will not cause problems in earlier versions
• rebased on master