mavgen.py: correct handling of includes (updated)

[hamishwillee]

This supersedes #338 and #339. It updates @olliw42 recursive nesting code so I hope it can be merged. #339 will be closed.

Changes w.r.t 338.

1. Removes the changes to default mavlink version etc (were just debug changes that hadn't been cleaned up)
2. Comments out the success-path debug print statements. The fail case statements stay (e.g. indicating for example, unresolvable inclusion)
3. Removes the test xml from the tree.

@peterbarker I have taken the time to convince myself that this code works. I have also run both Ollie test code and some of my own for both c and Python. The only differences are those that are expected - ie it fixes up the parent dialect headers with the CRCs etc of the nested dialects.

I have not included the test xml because I'm not sure where it would make sense to do so if the testing cannot be automated - and this case would be difficult to do.

I will withdraw my original PR. What do we need to do to get this in?

@auturgy FYI

[hamishwillee]

@peterbarker Reminder. This would be damn nice!

[peterbarker]

@hamishwillee this looks like an unintended but beneficial change?

pbarker@bluebottle:~/rc/pymavlink((HEAD detached from hamishwillee/recursive_includes))$ diff -ur /tmp/ri-output/ /tmp/master-output/
diff -ur /tmp/ri-output/icarous/version.h /tmp/master-output/icarous/version.h
--- /tmp/ri-output/icarous/version.h	2020-08-20 13:04:38.938184480 +1000
+++ /tmp/master-output/icarous/version.h	2020-08-20 12:55:49.743378972 +1000
@@ -9,6 +9,6 @@
 
 #define MAVLINK_BUILD_DATE "Thu Aug 20 2020"
 #define MAVLINK_WIRE_PROTOCOL_VERSION "2.0"
-#define MAVLINK_MAX_DIALECT_PAYLOAD_SIZE 46
+#define MAVLINK_MAX_DIALECT_PAYLOAD_SIZE 255
  
 #endif // MAVLINK_VERSION_H
diff -ur /tmp/ri-output/uAvionix/version.h /tmp/master-output/uAvionix/version.h
--- /tmp/ri-output/uAvionix/version.h	2020-08-20 13:04:38.926184491 +1000
+++ /tmp/master-output/uAvionix/version.h	2020-08-20 12:55:49.735378961 +1000
@@ -9,6 +9,6 @@
 
 #define MAVLINK_BUILD_DATE "Thu Aug 20 2020"
 #define MAVLINK_WIRE_PROTOCOL_VERSION "2.0"
-#define MAVLINK_MAX_DIALECT_PAYLOAD_SIZE 41
+#define MAVLINK_MAX_DIALECT_PAYLOAD_SIZE 255
  
 #endif // MAVLINK_VERSION_H
pbarker@bluebottle:~/rc/pymavlink((HEAD detached from hamishwillee/recursive_includes))$ 

And when generating mavlink1:

pbarker@bluebottle:~/rc/pymavlink(master)$ diff -ur /tmp/ri-output/ /tmp/master-output/
diff -ur /tmp/ri-output/icarous/version.h /tmp/master-output/icarous/version.h
--- /tmp/ri-output/icarous/version.h	2020-08-20 13:10:27.478035117 +1000
+++ /tmp/master-output/icarous/version.h	2020-08-20 13:18:46.150180476 +1000
@@ -9,6 +9,6 @@
 
 #define MAVLINK_BUILD_DATE "Thu Aug 20 2020"
 #define MAVLINK_WIRE_PROTOCOL_VERSION "1.0"
-#define MAVLINK_MAX_DIALECT_PAYLOAD_SIZE 0
+#define MAVLINK_MAX_DIALECT_PAYLOAD_SIZE 255
  
 #endif // MAVLINK_VERSION_H
diff -ur /tmp/ri-output/uAvionix/version.h /tmp/master-output/uAvionix/version.h
--- /tmp/ri-output/uAvionix/version.h	2020-08-20 13:10:27.474035117 +1000
+++ /tmp/master-output/uAvionix/version.h	2020-08-20 13:18:46.146180474 +1000
@@ -9,6 +9,6 @@
 
 #define MAVLINK_BUILD_DATE "Thu Aug 20 2020"
 #define MAVLINK_WIRE_PROTOCOL_VERSION "1.0"
-#define MAVLINK_MAX_DIALECT_PAYLOAD_SIZE 0
+#define MAVLINK_MAX_DIALECT_PAYLOAD_SIZE 255
  
 #endif // MAVLINK_VERSION_H
pbarker@bluebottle:~/rc/pymavlink(master)$ 

That change actually has interesting implications for anyone allocating memory based on it; malloc of zero bytes is problematic.

[peterbarker]

OK, after reading through the code it's rather less daunting than I'd initially thought.

I've folded up Hamish's trivial fixes (thanks!) back into @olliw42 's original commit.

I've added to the repository a simple structure I used to test the generation - relatively easy to tell at a glance if things are working correctly and may form the basis of a regression test later.

I've also added a patch on top in which I:

 - add large comment explaining algorithm used
 - add comments
 - remove some state variables which could be inferred from other state
variables
 - invert the return values of the helper functions, so
"expand_oneiteration" returns true if it expanded something
 - removed a couple of bad comments
 - simplified a few state variables (e.g. inverting sense of one of
them)

Hopefully I haven't broken anything! More testing tomorrow and then I'll probably merge (@tridge).

[hamishwillee]

Thanks @peterbarker . The comments will be very useful - it all makes sense, but since Ollie doesn't think like I do it took some parsing for me to understand.

With respect to #436 (comment)

This indicates that the payload sizes have changed (e.g. from 0 to 255), right? That needs to be sanity checked. I would expect any dialect that includes common.xml to be 255 so it is possible that the calculation was broken before.

[peterbarker]

Merged!

Thanks @olliw42 , @hamishwillee and @auturgy for your patience on this!

[peterbarker]

This indicates that the payload sizes have changed (e.g. from 0 to 255), right? That needs to be sanity checked. I would expect any dialect that includes common.xml to be 255 so it is possible that the calculation was broken before.

Quite the opposite. These "mixin" files were getting an incorrect size of 255 (the diff is reversed)

[hamishwillee]

@peterbarker Thank you. Much appreciated.

@olliw42 And you thought that our glacial slowness meant this would never happen :-)