Add telemetry to flight information

[BenjaminPelletier]

This PR is intended as a concrete design proposal and comments are welcome, including to the effect of taking an entirely different approach.

Motivation

The main purpose of the flight_planning interface is to allow uss_qualifier to "puppeteer" a user of a USS's system to attempt to perform various flight planning activities. To effectively puppeteer a user of a USS's system, InterUSS must provide all the information that would be provided by a real user for that flight. One piece of information we are currently missing is the telemetry that would be generated for the flight. This could be important for some USS implementations if, for instance, they decided to disallow activation until the aircraft associated with the flight reported telemetry inside the flight volumes. Furthermore, a number of ASTM F3548-21 requirements relate to telemetry (specifically: ACM0005, CMSA0100-CMSA0115, LOG0050), so we will likely need to specify it to test those requirements effectively.

Bulk-injection approach (this PR)

This PR approximately follows the approach used by the existing RID injection API by adding one additional field to FlightPlan (the main body of the request to perform a flight planning activity) potentially containing telemetry for that flight. This field would only be populated by InterUSS when appropriate -- for instance, planning activities that never intend to activate the flight would likely not specify telemetry. The USS receiving a planning request with telemetry is expected to queue that telemetry to be delivered to the system under test (by the USS's automated testing adapter), clear that queue if the flight is canceled/closed, and update that queue if the flight is updated with different telemetry.

The data structures used in this PR are modeled after the data structures in the RID injection API and attempt to capture equivalent information, but are generally made more flexible to accommodate more general telemetry needs that don't necessarily match RID telemetry.

One additional feature in this PR (adopted from the RID injection API) is having a response field where the USS can indicate the information that was actually used. This is necessary in the RID injection API as it is expected that USSs will need to change certain features of the injected data (e.g., serial number), and providing USSs the ability to do this in the flight_planning API will provide them more flexibility as well. For backwards compatibility, a USS can omit this field to indicate that the all information was used as provided.

Advantages of the bulk-injection approach in this PR include:

• Similar approach as the RID injection API, so USSs already implementing that API should be able to handle injection of this telemetry easily
• Low network traffic: uss_qualifier often only sends one message containing tens (or perhaps hundreds) of telemetry reports, and the USS queues these locally to be delivered to its system under test. This means uss_qualifier does not need to send a query per telemetry report, which would increase network traffic (and therefore the test report size) substantially.
• Low latency: Since the latency between uss_qualifier and the USS, and between the USS's automated testing handlers and delivery to its system under test, is eliminated, there is less of a risk of telemetry not being delivered in a timely way due to automated testing difficulties.

Disadvantages of this approach include:

• USSs must implement a background/asynchronous telemetry queuing and delivery system. This is substantially more complex than synchronously responding to a request by immediately injecting a single telemetry report.
• The possibility of incorrect queue management is introduced, and this is a source of potential errors that would be due solely to the automated testing adapter rather than the system under test.

Synchronous individual telemetry injection alternative

One alternative to the approach in this PR is not to add any data to the FlightPlan structure, and instead create a new operation named something like PUT /telemetry/{telemetry_id} that would be implemented by a USS under test. uss_qualifier would be responsible for calling this endpoint each time a single telemetry report should be injected into the system under test. The request body of this operation would correspond with the AircraftState data structure in this PR.

Advantages of this approach include:

• Minimum complexity for implementing USSs
• Minimum opportunity for logic errors due solely to implementation of the automated testing adapter
• Conceptual simplicity: developers and users don't need to consider what will need to happen in the future (outside the completion of the handler) to satisfy an automated testing request
• Clear cause/effect relationship: the effect of an injected telemetry report can be more easily linked to the injection of that particular telemetry report

Disadvantages of this approach include:

• Increased susceptibility to network connectivity issues: while failed network requests are rare proportionally, this approach will increase number of requests by 1-3 orders of magnitude (though this could potentially be mitigated by keeping the TCP socket open between requests, and total bandwidth required is likely to increase by only a factor of ~2)
• Increased susceptibility to performance challenges with USSs: if injection of telemetry to one USS took a long time in the middle of a test, the test could be broken for all USSs
• Increased complexity of uss_qualifier: Currently, all uss_qualifier testing is purely synchronous. With this approach, it is likely we would be forced to send telemetry injections to different USSs simultaneously to keep up with typical ~1 Hz expected telemetry rates. This increases conceptual complexity as well as risk of underperformance (relative to what is necessary to successfully conduct the test).

Self-generated telemetry alternative

Another alternative approach leverages that simulation capabilities of some USSs. Some USSs have sophisticated real-time aircraft behavior simulations available that could be incorporated into these tests if, instead of uss_qualifier injecting telemetry into USSs, USSs could report the telemetry that was actually generated by their simulations.

For this approach, there would need to be some mechanism for USSs to effectively push telemetry reports to uss_qualifier. Exposing and endpoint to receive these reports would be an enormous architectural change, so this is not a recommended approach. Push notifications may be able to use a similar approach as browser-based web apps, or uss_qualifier could continually poll for pseudo-push-based updates.

Advantages of this approach include:

• Leveraging sophisticated simulation engines from USSs increases testing realism and likely exercises a larger portion of the system under test

Disadvantages of this approach include:

• We would need to figure out a way to describe desired aircraft behavior in a way that is general enough to apply to all USS simulation implementations, but specific enough to achieve test objectives (e.g., "make sure the aircraft leaves the volume at 15-18 seconds after start"). This would likely be a substantial challenge.
• USSs would be forced to implement their own simulation engine if they didn't have one already
• Figuring out how to conceptually enable USS push notifications to uss_qualifier would require additional development and likely require higher bandwidth and have more opportunity for problems than even the synchronous individual telemetry approach

[mickmis]

I agree with the bulk-injection approach, that does seem to be the most reasonable one to me. I think that synchronous injection would introduce too many potential issues in the process of the testing itself that would be difficult to discriminate from issues with an USS not acting as it should.
The fact that this is similar to the RID injection is strong point, as the USSes having to implement a different type of approaches to do basically the same thing does not seem very reasonable.

There is one point that I feel might be problematic, which is the more fundamental difference between the flight plan itself and the (telemetry of the) flight.
As far I as understand the flight planning interface now, it is mostly declarative, as in an injection request may update a state and return the updated state. And save for cleanup of expired data or other action initiated on the side of the USS qualifier, this state is not expected to change with time (or is it?). Now, with the telemetry, this state that is returned by the USS may change with time. Could this cause issues?
Phrased differently, what is the scope of validity of the response to the injection request? Is this only at the time of the response and then the state can change anytime? In that case shouldn't we need a way to poll the USS for this information?

USSs must implement a background/asynchronous telemetry queuing and delivery system. This is substantially more complex than synchronously responding to a request by immediately injecting a single telemetry report.

If that is really an issue for some USSes, maybe that could be alleviated by implementing an optional and separate tool that would do that for them? For which the execution would be completely independent from the execution of the test scenario.

[punamverma]

Thanks Ben for adding telemetry.

Its not clear which uas_state and usage_state would the telemetry be sent in the flight_plan? How would USS need to handle telemetry info sent in various UpsertFlightPlan requests for a single flight_plan_id? What would be a trigger for a USS to start the telemetry? What would the uas_state and the utm state be, when the telemetry is started? Could we have a sequence diagram for the test of a flight plan that has telemetry and goes from Planned to OkToFly and OffNominal state (NonConforming and Contingent due to position-report)?

I agree with Mike's comment that some USSes may have issues implementing telemetry queuing and delivery system. So, it would be good if the test driver can provide some means of sending telemetry for such USSes. And I think test driver will hav e more control on the test sequences if the telemetry is sent synchronously by the test driver.

[nasajoey]

Is there a reason that ONLY the timestamp is required? With no other elements, do we really have any state information that is reportable? Only viewing in GitHub review editor... if I missed something in regards to required elements, just LMK.

[BenjaminPelletier]

This interface is essentially puppeteering a user of the USS's system. It seems likely that we may want to simulate loss of one or more telemetry components at some point, but it seems like we would always know when the injected telemetry should be injected into the system under test.

For instance, we don't want to make position mandatory because then we would be precluded from testing a scenario where an aircraft lost position information but was still able to report operational status. Do you have suggestions on what other fields should be mandatory? Remember, mandatory here means that uss_qualifier must always inform the USS's virtual user being puppeteered of the value of that field in every test ever conducted.

[nasajoey]

Should we have a min/max elements? I think generally good to upper bound the array size when possible/reasonable.

[BenjaminPelletier]

Since this is data we're going to have appear to originate from a user of the USS's system, it seems like we wouldn't want to set min/max because that would preclude testing how the USS would respond if the user provided values that were outside those reasonable bounds in their telemetry.

[nasajoey]

I wouldn't have made the assumption that these are data that are just sent raw to the USS... the API between the client and USS may very well not take ANY array of position data, let alone one with a limit. I'm just thinking from a data security aspect... if we want more predicability in payloads or not. If it isn't a security or resource concern, then fine not to have the upper limit. But, again, I wouldn't assume the API between a client and a USS at all... the USS under test needs to deal with this array as needed.

[nasajoey]

Should we re-use/reference the utm.yaml for VehicleTelemetry? https://github.com/astm-utm/Protocol/blob/cb7cf962d3a0c01b5ab12502f5f54789624977bf/utm.yaml#L1232

[BenjaminPelletier]

No, certainly not -- this interface is providing inputs to the USS-under-test's user interface. There is no reason those inputs would exactly match the format of ASTM F3548-21, and we need to be able to iterate on this interface independent of standards development. We actually already made the mistake of tying the original RID injection interface to F3411-19 data objects and we were forced to fix that mistake when we needed to test some things that were different in F3411-22a.

Additionally, some standard-defined data elements may not be feasible to obtain from the user (instead must be calculated or inferred from user-provided data) and in those cases we would not want to inject something that could not actually be provided to the true system via the normal user interface.

[nasajoey]

I would suggest making the min, example, and default be more "floaty" for clarity. 0.0, 1.2, 0.0 for example.

[BenjaminPelletier]

Sure; sounds good.

[nasajoey]

Thanks for the initial pro/con breakdown of the options!

I understand the reasoning in the bulk upload and why it is the easiest to get going from the uss_qualifier perspective. I like the option proposed by @mickmis to provide a separate tool to help USS's properly inject telemetry into their own systems, using the bulk data provided via this interface. The major drawback I think is the increased effort on service implementers to implement that functionality if they don't have it. The logical errors there seem more problematic to me than the synchronous approach's potential networking problems. So if we solved that for service implementers and provided a level of consistency for them in that implementation, it makes bulk upload a great choice.

This separate tool could be a future feature and should not hold up this PR.

[BenjaminPelletier]

As far I as understand the flight planning interface now, it is mostly declarative, as in an injection request may update a state and return the updated state.

@mickmis We're basically actuating ("puppeteering") user interactions with the system under test. Our synchronous requests now are something like: "The user does X. What is the content the user observes in the app in response to doing X?"

And save for cleanup of expired data or other action initiated on the side of the USS qualifier, this state is not expected to change with time (or is it?).

The state of the system may change in response to stimuli other than a user interaction with the system. Telemetry is the easy example: USSs are required to change F3548 state in response to telemetry under certain circumstances. Even if we consider telemetry to be a special type of user input, other stimuli could change the state of the system as well. F3548 takes great pains to avoid invalidating deconfliction accomplished strategically based on tactical situations that a USS/operator may not be able to respond effectively/correctly to -- in other words, a system not performing well after planning has very little effect on the safety benefit of strategic deconfliction that F3548 provides. This is not fundamental, however -- for instance, a jurisdiction may establish a requirement for an operator to resolve conflicts in particular ways within particular time limits (F3548 does not generally required the resolution of strategically-allowed conflicts). In this case, the system would change state at that deadline without any user input.

Phrased differently, what is the scope of validity of the response to the injection request?

Merely instantaneous. Hopefully we can set up tests that verify what we need to verify by controlling the environment as needed to avoid race conditions and/or stimuli outside the test parameters, but unless we have done that setup, we can't necessarily declare a system under test as non-compliant since the behavior observed could be explained by compliant responses to stimuli outside the test parameters happening at the same time as the activity of interest.

Is this only at the time of the response and then the state can change anytime? In that case shouldn't we need a way to poll the USS for this information?

No and maybe, respectively. A number of requirements relate to information presented to the operator/user and we do want some way of verifying these requirements. One way to determine what information is provided to the user could be to have a polling endpoint that just provides the state of the system as the user sees it, and commanding/puppeteering user actions would be a separate endpoint. In that paradigm, we would "plan a flight" by commanding the action (which would simply acknowledge that the action has been initiated) and then polling periodically until we determined that the action had resulted in the expected state change (or not). Instead, we happen to have chosen a system where the action is commanded, but that request does not return until the action has been completed, and then it is convenient to also return the state of the system (from the user perspective) at the same time. Our current system will not be sufficient to test requirements where users are required to get "push" notifications (e.g., SCD0090, SCD0095, GEN0400, GEN0405), but there are multiple ways to fix this. One way could be to have a "system state from user perspective" endpoint that we poll, and then detect the equivalent of push notifications by observing a change in that state. Another way could be to have a list of push notifications that were sent to the user, and we poll that list (I happen to have a predisposition toward the latter). But, I don't think we need either of these things to do many useful things with telemetry, especially (in the F3548 context) since we can poll the DSS and/or a mock_uss for HTTP interactions that it observed from other USSs.

If [implementing a background/asynchronous telemetry queuing and delivery system] is really an issue for some USSes, maybe that could be alleviated by implementing an optional and separate tool that would do that for them? For which the execution would be completely independent from the execution of the test scenario.

Makes sense to me, and I think that optional-helper-tool approach would be more flexible since USSs could choose to use it or implement something better themselves. Sounds like @nasajoey agrees, assuming the separate tool.

Its not clear which uas_state and usage_state would the telemetry be sent in the flight_plan?

@punamverma We're just describing what the virtual user is doing, so your question is equivalent to "it's not clear whether a user might send telemetry while indicating a particular UAS state and/or a particular airspace/flight plan usage state". I would expect that telemetry would nominally be sent whenever airspace is in use, but there are many off-nominal situations to consider and potentially produce during testing. What do you find unclear?

How would USS need to handle telemetry info sent in various UpsertFlightPlan requests for a single flight_plan_id?

I've added documentation that essentially treats telemetry as a RESTful sub-resource.

What would be a trigger for a USS to start the telemetry?

Obtaining telemetry in a FlightPlan. And remember, this is the virtual user's UAS that is reporting telemetry. Hopefully the core USS system is doing nothing with telemetry apart from receiving it -- instead, a separate adapter system is receiving this message from uss_qualifier and doing whatever is necessary to translate it into a USS-native telemetry report that can be sent to the core USS system at the appropriate time. We're talking about the red box on slide 7.

What would the uas_state and the utm state be, when the telemetry is started?

The uas_state and usage_state would be whatever the virtual user indicated them as. The F3548 op intent state would be whatever the USS set it as. Perhaps I'm not clear on what "telemetry is started" means; we're telling the virtual UAS that it's going to report these telemetry points at the specified times and expecting that the virtual UAS will do that. There is no "starting" and "stopping" per se from uss_qualifier's perspective.

Could we have a sequence diagram for the test of a flight plan that has telemetry and goes from Planned to OkToFly and OffNominal state (NonConforming and Contingent due to position-report)?

This is one of many possible diagrams involving the elements you mentioned. Coloring is designed to align with the slide 7 diagram I mentioned earlier.