Add Payload and Autonomy Microservice Support from Marine Dialect #1998
Conversation
I don't think there is a good reason to use cm when it's a float. I'd just use the SI unit instead.
The payload_type fields are only uint8_t, so we have to use values smaller equal 255.
This state is required in response to the demand GO_TO_RECOVERY.
By using the command MAV_CMD_REQUEST_MESSAGE we have the advantage that we can re-use the existing infrastructure for commands which includes: - Retransmission in case of lost messages (instead of just a timeout). - Better acks to signal if something is unsupported or deny it if the payload ID is invalid (again, instead of just a timeout).
|
This is a pretty substantial pr: will likely take a little while to work through it. |
|
I'm happy to answer any questions about the implementation and the rationale behind this change For a little bit more background: UUVs tend to operate in very comms limited environments (no RF / limited acoustic comms). As with most robots they have an on-board "autopilot" or "front-seat" that controls low-level vehicle movement (Control Fins to set Heading, Control Motor to Set Speed etc). They also often run a generic Autonomy system on an additional "back-seat" computer, which accepts high-level commands and breaks them down for the front-seat vehicle to consume (Scan this point, Scan this area etc.) Typically each vehicle manufacturer creates their own bespoke interface to allow external-control from an autonomy system, we are pushing to standardise this approach with an open standard. We are using MAVLink now as our vehicle interface for this Autonomy <-> Frontseat link and have been working with vehicle manufacturers have MAVLink integrated as a "frontseat" control interface. |
|
Brief brain dump of some thoughts/concerns (not expecting any particular response at this time, just capturing first impressions). You may well have answered these already in the docs.
|
|
As @auturgy says, this is a lot to digest. I very much like the idea of standardization of component information on the network, including non-mavlink components.
Generally we can take a dialect like this, but if this is to be in standard it would have to be split and probably go through RFC process, as per my comment above. |
|
Thanks for your reply Hamish, really useful to get your views :) Splitting of Autonomy + Payload + Sensor things sounds good to me. I would like the new services to be available to others for maximum compatibility. I am open to iteration to get them up to scratch. I never liked running as a separate MAVLink dialect and I aimed to keep features as generic as possible. Do you suggest I pull out the services into their own files? So the concept of the payload service stems from us having the ability to turn on onboard power-intensive sensors/payloads that are not directly MAVLink aware, but the vehicle is. Examples include things like sidescan sonar, where you only want them enabled during segments of the mission. Payload ID is unique to payloads only. We also needed a way to configure payloads, which we achieved through a mapping between parameters naming and payload IDs (not included here, but effectively PayloadID 1 will have its params accessible via I like the idea of combining payloads with components. For our use case all the payloads were controlled internally by the vehicle. The payloads do not have the capacity to be MAVLink-aware themselves. We could have had the vehicle emulate a component for each payload (MAVSDK work here @julianoes?) as they were added or removed but i'm still unsure how controlling the payloads (on/off/arm) would have worked in this case. Perhaps this is something to look at. Our payloads are controlled entirely by the Autonomy at current, but we could embed them as the a mavlink mission item also. Now some background to the Autonomy service, the Autonomy is unrated to AP modes. The AP is almost always in control of the actual vehicle via Auto/Mission. The Autonomy is monitoring the situation and packaging missions and sending them to the AP to execute and cancelling/re-issuing where appropriate. You can think of it as an autonomous GCS. The Autonomy has a higher level view of the objectives than an Autopilot. Imagine a mission with tens of surveys. This would be an unwieldy single MAVLink mission for the AP to run. The Autonomy is aware of the full-scale mission and is planning on the fly based on the vehicle situation. The MAVLink AP is receiving at most a "chunk" of the mission that is up-to-date with the vehicles current situation. As the vehicle completes the mission, the second one is issued from the Autonomy with the latest up to date information. Imagine while the AP completes a mission "chunk" some future surveys in the overall mission are now no longer needed, the autonomy is tracking this and can adjust the future mission "chunks" as they are issued. It can also add in new chunks and respond to information recovered during the survey. The Autonomy demands are control messages to the Autonomy system only. They can come from a ground control system to start/stop the Autonomy. They can come from an AP if the AP has a reason to demand start/stop the Autonomy system (fault? Mission from elsewhere?). If the AP accepts a mission from QGC, it should probably stop the Autonomy (if it is running!) as it's unrelated to the Autonomy mission. Alternatively as you say, the AP should just ignore any requests if Autonomy is in control, until it gets stopped or booted out. It's worth detailing these scenarios. |
|
Thanks @bazfp - I'm on other tasks until Wednesday so I can't give this the attention it deserves until then. But yes, I think that this would be better split into 2 or three files with a protocol-specific name (might or might not eventually end up inside common.xml, or linked from it). This is mostly so that they can be digested and accepted or rejected in parts. It is such a lot to take in otherwise. I have added this to discussion in the dev call next week. Given how much there is, I expect that will be just a broad chat about our top level thinking. I am personally very interested in this but want to understand the problems that it solves/doesn't solve better. |
|
Payload stuff:
Another tool that might be useful for this is currently only available on PX4 - component metadata. The component metadata concept allows you to define deeply nested structures mapping to parameters, and has been used to define UIs. See the PX4 actuators setup - and the same concept came first in the camera definition file. Anyway, may take on this is that dedicated APIs mediated through companion or autopilot are generally more flexible. |
|
Re Autonomy - I get it. The model is pretty much
Just to be aware of, a flight stack always has to be able to seize control, and you can't guarantee that it will send a "taking control" message, or that the companion will get it. Normally the companion detects this by monitoring mode - i.e. if control is via a dynamically updating mission, then it will monitor the mission mode and detect it has lost control by the mode changing. The last case to consider is control from a companion in another mode. The example I know of here is that PX in a mission can enable obstacle avoidance. The mode is mission and the AP emits a desired trajectory. The companion responds by mirroring the trajectory or providing an avoidance path. The AP will just use its own path if no points come back. |
|
@bazfp We discussed briefly in the dev call https://github.com/mavlink/mavlink/wiki/20230607-Dev-Meeting. Would love to discuss next steps in the dev call in two weeks. Wr.t.. handover of control, we discussed in the dev call. A vehicle can always take control of itself and should already emit values to indicate this in the heartbeat state - e.g. https://mavlink.io/en/messages/common.html#MAV_STATE_CRITICAL, |
|
@bazfp Are you up for coming along to the MAVLink call in two weeks to discuss? Julian says he tried to skype you to no avail. FYI We talked about it in last nights call briefly. Some comments were similar
New questions
|
|
Thanks Hamish, appreciate these discussions and feedback. I was on leave last week so not available. We're keen to join the next discussion and discuss in detail the next actions.
We allocated the range based on usage, happy to change if needed.
Great to here :) we tried to maintain genericness as much as we could.
Yes this autonomy control is a new paradigm for MAVLINK, for UUVs this frontseat <-> backseat control has been used for a long time. I am keen to make sure we slot this in nicely. I will look to split up this PR in the next two weeks |
|
A couple of questions. Separated into the separate services. Payload ServiceNB: This assumes your system is running some form of MAVLINK router allowing multiple streams to be combined/split. Autonomy ServiceNB: Similar to the Payload service I'm not sure using the autopilot as an intermediary for status updates is necessary as the Autonomy component will need to be on the MAVLINK network anyway. |
|
@bazfp Mav call this evening.
Let's discuss. I'd prefer to restrict the range - probably to 100 ids in first place at 52100 - 52199 |
This has been in development for over a year and is currently operational on underwater vehicles (UUVs) like the popular Remus vehicle from Hydroid. It's stable enough now we have decided to present the changes upstream.
This is version v22.08.01
There are also matching functional implementations in MAVSDK which are also in operation on both vehicle-side and autonomy-control side.
The things of note are:
Payload Microservice
The Payload Microservice is used to discover, configure and control
Payloads<Payload>(which could be sensors, actuators or other connected components) registered on aMAVLinksystem.You can use this Microservice to:
It is expected that the AutoPilot maintains a list of registered Payloads, containing for each:
A system can use the Payload Microservice to request details about the Payloads on the MAVLink system and to configure said Payloads.
When an external system enables a Payload it will set the Payload state to:
When a Payload is disabled, An external system will set the Payload state to:
In order to request a
PAYLOAD_STATUSfrom the AutoPilot, A system willsend a MAV_CMD_REQUEST_MESSAGE with:
44206(the MessageID of
PAYLOAD_STATUS)Payload being enquired about
For a list of the enumerations used in the payload messages, please see
payload-enum-summary.The AutoPilot also needs to respect the implementation details layed out in
payload-microservice-operation-exceptions.Workflows
payload_discovery_processbelow shows the communication sequence external systems use to retrieve the list of registered Payloads from theAutoPilot(assuming all operations succeed).In more detail, the sequence of operations is:
PAYLOAD_REQUEST_LISTto start the discovery anddownload process
PAYLOAD_COUNTresponse from the AutoPilot. If no response isreceived during that time, Autonomy will re-send the
PAYLOAD_REQUEST_LIST message.
PAYLOAD_COUNTwhere the count fieldis the number of registered payloads
PAYLOAD_LIST_ITEM_REQUESTfor the first list item(
list position = 0)PAYLOAD_LIST_ITEM_REQUESTresponse from the AutoPilot. If noresponse is received during that time, Autonomy will re-send the
PAYLOAD_LIST_ITEM_REQUESTmessage.PAYLOAD_LIST_ITEM_REQUESTmessage andresponds with the requested Payload details via a
PAYLOAD_LIST_ITEMmessagedeatils of all Payloads
with a single
PAYLOAD_LIST_ACKwith thetype field set to
PAYLOAD_LIST_ACCEPTED(see thePAYLOAD_LIST_RESULTenum) toindicate Payload discovery success
PAYLOAD_LIST_ACKcontainingPAYLOAD_LIST_ACCEPTEDand understands the operation to becomplete.
List request errors are considered unrecoverable see
payload-microservice-errorsfor details on how to handle errors.payload_status_updatebelow shows the communication sequence that autonomy follows to request the status of a registered Payload.Autonomy sends a MAV_CMD_REQUEST_MESSAGE message with the ID of the Payload it wants the status of (
IDX) to the AutoPilot. The AutoPilot then responds with aPAYLOAD_STATUSmessage containing the Type, Health and Status of the Payload.The diagram below shows the communication sequence Autonomy uses to change the State of a registered Payload. The
MAV_CMD_PAYLOAD_SET_STATEmessage is used to set the State of a Payload using a bitmask corresponding to thePAYLOAD_STATEenum, where the bits are set to1to indicate the state istrueand0to indicate the state isfalse.As part of the Payload discovery process, the valid States that can be set for each Payload are specified in the
PAYLOAD_LIST_ITEMmessage. If trying to set a unsupported state on a payload (e.g. settingArmed=trueon a Payload that only has Powered as a valid state), then theMAV_CMD_PAYLOAD_SET_STATEcommand should be rejected by the AutoPilot with a COMMAND_ACK with a MAV_RESULT ofMAV_RESULT_DENIED.If the id field in the
MAV_CMD_PAYLOAD_SET_STATEcommand sent by Autonomy is0, the AutoPilot must apply the state change to all registered Payloads for which it is a valid option. If a Payload ID that is not registered is specified in the command, it should be rejected by sending aCOMMAND_ACK with a MAV_RESULT of
MAV_RESULT_DENIED.If AutoPilot successfully handles the command, a COMMAND_ACK message should be sent back to Autonomy with a MAV_RESULT of
MAV_RESULT_ACCEPTEDfollowed by AutoPilot sending aPAYLOAD_STATUSmessage containing the updated Payload State.When a Payload is added or removed to the AutoPilot's Payload list, a
PAYLOAD_CHANGEmessage needs to be broadcast in order to inform all connected Companion Computers (including Autonomy) of the change, specifying the Payload ID that was affected and whether it has been added or removed.If the update was about a new Payload, the Companion Computers may subsequently sent a
MAV_CMD_REQUEST_MESSAGE
message to obtain details of the new Payload.
Autonomy Microservice
The Autonomy Microservice allows the
AutoPilotto set Autonomy execution state, for control handover, and to monitor the state of Autonomy. The variousMAVLink Messagesused in the Autonomy Microservice are detailed inAutonomy Message Table.MAV_CMD_AUTONOMY_DEMANDAUTONOMY_STATUSMessages used by the Autonomy Microservice
Central to the Autonomy Microservice is a state machine representing the autonomy system's execution state. numref:autonomy_state_diagram shows this state machine, where:
Each state is defined in the
AUTONOMY_STATEenum (sent fromAutonomy to the AutoPilot via the
AUTONOMY_STATUSmessage).Each transition is defined in the
AUTONOMY_DEMANDenum (sentfrom the AutoPilot to Autonomy via the
MAV_CMD_AUTONOMY_DEMANDmessage).
The Autopilot MUST send a
StoporPausecommand to Autonomy whenever it takes forcible control of the system.In addition to providing an overall system state, the
AUTONOMY_STATUSmessage also provides status bitmaps for a number of sub-components. Autonomy provides status for the following components:AUTONOMY_COMPONENT_COREAUTONOMY_COMPONENT_ATRAUTONOMY_COMPONENT_MISSIONStatus bits for all other components are not used by Autonomy and set to zero.
Health Reporting
Component Health Descriptionsdescribes in further detail what eachhealth flag reported by
AUTONOMY_STATUSindicates.AUTONOMY_COMPONENT_COREAUTONOMY_COMPONENT_ATRAUTONOMY_COMPONENT_MISSIONAutonomy Component Health
Workflows
autonomy_request_statebelow shows the communication sequenceAutoPilot needs to follow to get the status of Autonomy.
Autonomy responds to the
MAV_CMD_REQUEST_MESSAGE
command (see
Command Types Table) sent by the AutoPilot with aAUTONOMY_STATUSmessagecontaining the presence, state and health of the components present on
the Autonomy system, and the current execution state of the system.
autonomy_send_autonomy_demandbelow shows the communication sequencewhen the AutoPilot sends a
MAV_CMD_AUTONOMY_DEMANDcontaining anAUTONOMY_DEMANDto Autonomy.Autonomy responds with a
COMMAND_ACK
and a
AUTONOMY_STATUSmessage as stated inpayload-microservice-errors.Additional MAVLink Messages
In addition to the messages already detailed in the various
MAVLink Microservices,MESSAGE_CONTACT_SNIPPETMESSAGE_AUTONOMY_MISSION_STATUSSQUAD_VEHICLE_STATEAlso see the following enumerations:
MAV_SYS_STATUS_SENSOR_EXTENDEDExternal Communications
For integrations where all external communications (acoustic, RF etc.) for a vehicle are handled entirely by the
AutoPilot:the AutoPilot
MESSAGE_CONTACT_SNIPPETMESSAGE_AUTONOMY_MISSION_STATUSAutonomy will provide the following metadata to define how the AutoPilot should handle the communication:
external target of the communication, or
0for broadcastcommunication to be sent (32 bytes by default)
MESSAGE_CONTACT_SNIPPET, to be used by the AutoPilot to sort theorder of the stored snippets
AutoPilot must provide the following metadata when sending a
MESSAGE_AUTONOMY_MISSION_STATUSto Autonomy:external source of the communication
ALL
MESSAGE_CONTACT_SNIPPETmessages received by the AutoPilot should be sent in the order they were received; while only the most recentMESSAGE_AUTONOMY_MISSION_STATUSshould be sent.It is expected that the AutoPilot will forward any incoming Mission Status communications to Autonomy using the
MESSAGE_AUTONOMY_MISSION_STATUSmessage.If sending outgoing communications over a limited medium, such as acoustic, it is expected that the AutoPilot uses the type of outgoing communication to determine the sending priority.
If the AutoPilot receives telemetry, vehicle state or fault updates from other vehicles in the squad via acoustic of other communications, these must be forwarded to Autonomy using the
SQUAD_VEHICLE_STATEmessage.APPENDIX 1
Messages (Marine)
MAVLink Type Enumerations
AUTONOMY_DEMAND
[Enum] Demands
that can be made to an autonomy system.
AUTONOMY_COMPONENT
[Enum] These
encode the components that can be present in an autonomy system.
AUTONOMY_STATE
[Enum] Describes
the states an autonomy system can be in.
MAV_SYS_STATUS_SENSOR_EXTENDED
[Enum] Extends the
common
MAV_SYS_STATUS_SENSOR
enum.
PAYLOAD_TYPE
[Enum] Defines the
various types of
Payloads<Payload>that can be available on theMAVLink system.
PAYLOAD_HEALTH
[Enum] These
define the operational health a
Payloadcan be in.PAYLOAD_STATE
[Enum] These
define the states a
Payloadcan be in.PAYLOAD_LIST_RESULT
[Enum] These
define the results of a payload discovery operation.
VEHICLE_MODE
[Enum] Vehicle
Mode.
MAVLink Commands (MAV_CMD)
Note
MAVLink commands
(MAV_CMD)
and messages are different! These commands define the values of up to 7
parameters that are packaged INSIDE specific messages used in the
Mission Protocol and Command Protocol. Use commands for actions in
missions or if you need acknowledgment and/or retry logic from a
request. Otherwise use messages.
MAV_CMD_AUTONOMY_DEMAND (#44000)
[Command]
Send autonomy demands to a
Companion Computer.MAV_CMD_PAYLOAD_SET_STATE (#44002)
[Command]
Set the states for the payload with the specified ID.
Command will be rejected if trying to set a state not supported by the
payload.
Note
This command should always be sent as
COMMAND_INT which has:
float
MAVLink Messages
AUTONOMY_STATUS (#44001)
[Message] Used
AUTONOMY_STATUSto specify the state of an autonomy system.
AUTONOMY_COMPONENT<AUTONOMY_COMPONENT>enum, showing which autonomy components are present on theMAVLinksystem:0if not present1if presentAUTONOMY_COMPONENT<AUTONOMY_COMPONENT>enum, showing which autonomy components are enabled:0if not enabled1if enabledAUTONOMY_COMPONENT<AUTONOMY_COMPONENT>enum, showing which autonomy components have an error (or are operational):0if it has an error1if it is healthyAUTONOMY_STATE<AUTONOMY_COMPONENT>MESSAGE_CONTACT_SNIPPET (#44100)
[Message]
Contact snippet message to be sent to specified destination with message
priority, as well as confidence of contact snippet.
MESSAGE_AUTONOMY_MISSION_STATUS (#44101)
[Message]
Autonomy System Mission Status message to be sent to specified
destination.
SQUAD_VEHICLE_STATE (#44102)
[Message]
Vehicle state data recieved from another vehicle in the squad.
VEHICLE_MODEfor allowed values.PAYLOAD_REQUEST_LIST (#44200)
[Message] Used
to request the list of registered Payloads from the AutoPilot.
PAYLOAD_COUNT (#44201)
[Message] This
message is emitted as response to
PAYLOAD_REQUEST_LIST. TheCompanion Computercan then request the individual payload item based on theknowledge of the total number of
Payloads<Payload>.PAYLOAD_LIST_ITEM_REQUEST (#44202)
[Message]
Request the status of the Payload with the specified Payload ID.
PAYLOAD_LIST_ITEM (#44203)
[Message]
Description of a single
Payload.PAYLOAD_LIST_ACK (#44204)
[Message]
Acknowledgment Message from a
Companion Computerupon receiving allPAYLOAD_LIST_ITEM Messages.
PAYLOAD_STATUS (#44206)
[Message] This
message is emitted as response to
MAV_CMD_REQUEST_MESSAGE messages with
Param 1 specifying the message ID
(
44206) and Param 2 specifying thepayload ID. This message contains the details (ID and Type) and health
and status (as bitmasks) of the specified payload.
PAYLOAD_CHANGE (#44207)
[Message]
Notification that a
Payloadhas been added or removed from the MAVLinksystem.
WATER_CURRENT (#44300)
[Message] Water
current values.
Note
Calculated Water current = (Vehicle Velocity - Measured Current).
DVL (#44301)
[Message]
Doppler Velocity Log Sensor. Used to provide vehicle velocity.
APPENDIX 2
Autonomy Protocol
The autonomy microservice is used by
AutoPilots<AutoPilot>to getinformation about
Companion Computers<Companion Computer>that provideautonomy. It also allows AutoPilots to configure the state of autonomy
systems.
Full definitions can be found in
marine.xml.
Message/Enum Summary
MAV_CMD_AUTONOMY_DEMANDAUTONOMY_STATUSAutonomy Protocol Message Summary
AUTONOMY_DEMANDAUTONOMY_COMPONENTAUTONOMY_STATEAutonomy Protocol Enum Summary
Operation Exceptions
Timeouts and Retires
A timeout should be set for all messages that require a response. If the
expected response is not received before the timeout then the message
must be resent. If no response is received after a number of retries
then the client must cancel the operation and return to a clean state as
it was before the initial message was sent.
The recommended timeout values before resending, and the number of
retries are:
1500 ms5Errors/Completion
On acceptance of a
MAV_CMD_AUTONOMY_DEMAND, the autonomy componentmust respond with a COMMAND_ACK message
with a
MAV_RESULT_ACCEPTEDMAV_RESULTand then it must publish a
AUTONOMY_STATUSmessage containing theupdated execution state.
If a command is received by the autonomy component when it is not in a
state to carry it out, the resulting
COMMAND_ACK must contain the
MAV_RESULT_TEMPORARILY_REJECTEDMAV_RESULT.
If a command is received by the autonomy component when would constitute
an invalid state transition (for example if the autonomy sytem does not
allow a
AUTONOMY_DEMAND_STARTwhen the current execution mode isAUTONOMY_STATE_EXECUTING), then the autonomy system should continue inits current state, rejecting the new one, and send a resulting
COMMAND_ACK containing the
MAV_RESULT_DENIEDMAV_RESULT.APPENDIX 3
Payload Protocol
The payload microservice is used by
Companion Computers<Companion Computer>to get information aboutPayloads<Payload>from theAutoPilot. It also allows Companion Computers to configure Payloads via
the AutoPilot.
Full definitions can be found in
marine.xml.
Message/Enum Summary
Payload Protocol Message SummaryPAYLOAD_REQUEST_LISTPAYLOAD_COUNTPAYLOAD_LIST_ITEM_REQUESTPAYLOAD_LIST_ITEMPAYLOAD_LIST_ITEM_REQUEST.PAYLOAD_LIST_ACKPAYLOAD_LIST_ITEMMessages.PAYLOAD_STATUSEmitted as response to MAV_CMD_REQUEST_MESSAGE with Param 1 set to
44206.It contains the ID, Type, Health and State of the Payload specified in Param 2 of the MAV_CMD_REQUEST_MESSAGE Message.
PAYLOAD_CHANGEMAV_CMD_PAYLOAD_SET_STATEPAYLOAD_TYPEPAYLOAD_HEALTHPAYLOAD_STATEPAYLOAD_LIST_RESULTPayload Protocol Enum Summary
Operation Exceptions
Timeouts and Retires
A timeout should be set for all messages that require a response. If the
expected response is not received before the timeout then the message
must be resent. If no response is received after a number of retries
then the client must cancel the operation and return to a clean state as
it was before the initial message was sent.
The recommended timeout values before resending, and the number of
retries are:
1500 ms250 ms5Errors/Completion
On successful completion of a payload list exchange the participant that
made the initial
PAYLOAD_REQUEST_LIST, will send aPAYLOAD_LIST_ACKmessages with a result value of
PAYLOAD_LIST_ACCEPTED(see thePAYLOAD_LIST_RESULTenum).Any other value in the result field of
the
PAYLOAD_LIST_ACKmessage indicates an error. Both participants inthe exchange must reset themselves to the state they were in before the
initial
PAYLOAD_REQUEST_LISTmessage, and the initiator must ignoreany information it received during the erronious list exchange.