Agency/telemetry: add unregistered error

[Retzoh]

Is your feature request related to a problem? Please describe.

As an agency, when receiving a batch of telemetry data including unregistered devices, there is currently no official way to let the provider know why the corresponding telemetries were rejected. This makes it complicated for the provider to automate the detection and resolution of this problem.

Describe the solution you'd like

Adapt the unregistered error existing in the Agency/event endpoint to Agency/telemetry.

Is this a breaking change

No, not breaking : we are just adding a new error type.

Impacted Spec

For which spec is this feature being requested?

• agency

Describe alternatives you've considered

The current workaround we are using is to return a bad_param error on parameter device_id, Listing the unregistered ids in the error description: A validation error occured. Unregistered devices: UUID1, UUID2, etc..

Additional context

Add any other context or screenshots about the feature request here.

[viestat]

Really looking forward for this!
It makes everything much simpler. As the consumer of the Agency endpoints it helps enormously to not only know what made a batch fail but specifically which vehicles are not registered. Then it is possible to trigger a register request for any vehicle present in that error as soon as this error is received.

[marie-x]

This is already in the mds-core implementation. Will add to spec.

[schnuerle]

We will be talking about this on the Working Group call tomorrow: https://github.com/openmobilityfoundation/mobility-data-specification/wiki/Web-conference-notes,-2020.10.01-(Joint-Working-Group-City-Services)

[schnuerle]

From the call today @karcass will double check but it looks good and can be merged to dev.

[marie-x]

Ah, I see where this diverges from the mds-core implementation. unregistered shows up in the failures field and does not necessarily force a 400. What will show up in that case is the failures array will contain one entry per failure of the form

device_id ${telemetry.device_id}: not found

So I would change the PR to describe this as a possible failure entry. A bad device should not prevent the receipt of valid telemetry data.

What do you think about this counter-proposal?

[schnuerle]

@karcass are you waiting on @Retzoh to reply here on your counter proposal? @Retzoh what do you think so we can merge this PR?

[marie-x]

@schnuerle Yes, I'm interested in what @Retzoh thinks about my counter.

[Retzoh]

Hi @karcass , sorry, I missed the initial ping.

I think it is dangerous to return a success code and rely on text-processing of the response payload to handle any errors.
To me, any error should be really straightforward.

Additionally, it seems more robust to me to just reject a invalid batch of telemetries until it is valid and pick-up the ingestion from there, rather than moving on and rely on tracking specific errors until resolved. With the second solution, it seems too easy both for the provider and the city to forget the errors and never handle them.

It would be interesting to have some feedback from providers on this. @viestat do you have an opinion ?

[schnuerle]

Tagging some more providers who may have a technical opinion on this: @dirkdk @bhandzo.

[schnuerle]

Do you think you can reach some consensus on this here in the GitHub PR in the next week so these changes can be added to the 1.1.0 release?

[dirkdk]

so the 2 alternatives are:

1. return 400 HTTP error code and error with key failures that contains an array of unregistered vehicle ids
2. return 200 HTTP success code but still a failures key that contains an array of unregistered vehicle ids.

For 1), for me it is implied that no data, even from already registered vehicles is processed
For 2), for me it is implied that correct data from registered vehicles is indeed processed but not for unregistered vehicles

IMHO, the solutions are very similar. I would opt for 1), as it is more explicit and its is clear that no data at all is processed and the client will need to correct their mistake and then retry.

[viestat]

In my opinion as the consumer of the Agency endpoints, I agree with what @Retzoh says about having to rely on the text response to identify errors.

Regarding @karcass 's counter proposal, if the endpoint takes batches as a whole it really does not make any sense to me to then treat individual items in the batch selectively. Either all the items that make up a batch are correct (and registered) in orderer to be processed or the batch is rejected and the reasons why it was rejected are made evident via the appropriate error (essentially what @dirkdk proposed above as option #1).

Doing this relieves the load on both sides (the consumer and the provider of the API) as it enables for a simpler retry flow and error handling.

[marie-x]

Sorry, I misread this diff originally. Is fine!

[schnuerle]

Looks good thanks for the conversations and concensus.