AuthorisationMetricsV2 abandonmentsByStage property descriptions and CDS Guide

[anzbankau]

Description

There have been numerous questions (and DSB answers) about properties in the abandonmentsByStage object in Get Metrics. The following proposes minor refinements to property descriptions and a new 'Authorisations' section in the relevant CDS Guide page.

Area Affected

Get Metrics > AuthorisationMetricsV2

Change Proposed

1. Description fields in abandonmentsByStage object:
   Note: Bold italics are only used to highlight the proposed changes
   Before: The number of authorisations where the customer actively rejected the authorisation rather than abandoning the process
   After: The number of authorisations where the customer actively rejected/denied the authorisation at the final stage of the consent process rather than abandoning ed the process or experienced an interruption at an earlier stage
2. Additional 'Authorisations' section in CDS Guide page CDR metrics and reporting by Data Holders to clarify the terms 'abandon' and 'reject'. The final screen in the consent flow (allowing consumers to 'confirm' or 'deny' data sharing) has explicit actions, whereas a 'Cancel' on earlier screens should not be considered a 'rejection' but an 'abandonment' - along with time-out and technical interruptions.

Reference

1. Tickets #2125.2 and #2170 in Q&A section of CDR Implementation call minutes of 7/12/23

DSB Proposed Solution

The proposed solution can be found in this comment.

[nils-work]

Hi @anzbankau

For reference, this guidance was recently published and I think aligns to your proposal - Get Metrics V5 abandonment by stages.

[anzbankau]

@nils-work,
Thanks Nils. That's very useful. At first I was concerned about introducing a new term 'exit', but I can see why that was necessary for the structuring of the article.

While that covers part 2 of the change proposal, we do want to keep part 1 as the data standards are obviously the pre-eminent resource for the community.

[nils-work]

To summarise, this issue proposes the following change to the description of the rejected property of AuthorisationMetrics -

Current

The number of authorisations where the customer actively rejected the authorisation rather than abandoning the process

Proposed

The number of authorisations where the customer actively rejected/denied the authorisation at the final stage of the consent process rather than abandoned the process or experienced an interruption at an earlier stage

[anzbankau]

Thanks Nils.

That's the appropriate property and the revised description provides additional information that will be helpful to participants.

[nils-work]

There was a discussion in the Maintenance Iteration 18 call on 6th March as to whether an updated description for the rejected field in Get Metrics v5 should be considered a breaking change, requiring a Future Dated Obligation (FDO) due to a potential change in interpretation -

• Current wording: "...actively rejected the authorisation rather than abandoning"
• Proposed: "...actively rejected/denied the authorisation at the final stage of the consent process rather than abandoned"

The intent is to clarify the Standards and align to previous guidance as stated in the opening comment -

Tickets #2125.2 and #2170 in Q&A section of CDR Implementation call minutes of 7/12/23

and the Get Metrics V5 abandonment by stages guidance.

As the proposed change would not affect the v5 response schema, a new endpoint version is not anticipated.

The DSB would be interested in comments from participants as to whether this change should be introduced with an FDO for the updated description of the rejected field, if the proposed change would cause existing or in-progress implementations to provide incorrect abandonmentsByStage metrics.

[perlboy]

As discussed in the call we don't believe the proposal as it stands is really in the interests of the Consumer and we raised this during the definition of the endpoint in the first place.

In addition, given this endpoint has already shipped we do not agree this is a minor clarification and as such request, if the DSB makes this change (which we don't agree with), that there be a suitable FDO that gives sufficient time to implement.

[nils-work]

This issue was discussed in the 20th March Maintenance Iteration call.

• One attendee requested that the proposed change to the description of the rejected field be treated as a breaking change and an FDO provided to cater for differences in previous interpretations (but not to accommodate a new endpoint version). It was suggested that it may be possible to meet an FDO within six months, but feedback from participants on a potential milestone would be valuable. Options could be Y24 #4 (09/09/2024) or Y24 #5 (11/11/2024).
• There was a concern raised in relation to the benefit to consumers from making this change.
• There was discussion around the granularity of potential reasons and terminology for abandonment, exiting, cancelling, and rejecting and whether these are active or passive, and whether they could be better distinguished in future versions if necessary.
• There was a question as to whether there was also a potential gap in the consistency of delivered CX in this flow (related to the availability and terminology used in navigation options provided by different Data Holders) and if there is potential to better define and pass more standardised detail back to ADRs via the oAuth error response where possible.
• It was noted that there could be a relationship between the issues described above and the intent of #628 - Addition of a DH-side endpoint for querying the status of a consent establishment flow.

[biza-io]

As stated by @perlboy we believe the proposed description change is a breaking change requiring an FDO and request this be Y24 #5 to align with other FDOs proposed in this MI.

At a more holistic level Biza.io has formed the opinion that the proposed change is in fact detrimental to the value of the data being collected. Resolution of this issue may be best served through a more detailed consultation on another new version of Get Metrics that encapsulates abandonment throughout the stages of the consent flow.

[nils-work]

This issue will be reviewed again in Maintenance Iteration 19.

[nils-work]

Update: Based on discussion and feedback in MI18, it is proposed that this change not be made.

The DSB has provided guidance in relation to abandonment by stages, which can be referred to by Data Holders, however it is accepted that different interpretations of abandonment and rejection metrics have been inferred which will result in differing metrics being reported via Get Metrics v5.

This issue may continue to be considered in the context of issue #628 - Addition of a DH-side endpoint for querying the status of a consent establishment flow, or other potential changes to authentication process requirements in future.

[perlboy]

Can the guidance be updated to omit the suggestion of Cancel as well please. The description in the Standards is:

The number of authorisations where the customer actively rejected the authorisation rather than abandoning the process

I don't think anyone would dispute clicking a Cancel button could reasonable be seen as an active rejection so omit'ing the suggestion ensures guidance is not seen as conflicting with the Standard while leaving the alternate interpretation (as per the original poster) open.

[nils-work]

Hi @perlboy
As we will be proposing that the Standards do not change as a result of this request, we consider the guidance should remain unchanged for the time being as well.

[anzbankau]

@perlboy

The "clicking 'Cancel'" item in the list under the preIdentification, preAuthentication, preAccountSelection and preAuthentication count classifications relates to properties within the abandonmentsByStage object in GetMetrics.AuthorisationMetricsV2 (with Description 'Customer abandonment count per stage of the consent flow'), rather than the rejected object for which you provided the Description.

We agree that clicking a Cancel button at any of these points (i.e., at any step prior to the final accept/confirm/deny stage) is an abandonment, not an active rejection, which is why it's listed in the actions/events that could cause the abandonment sub-count to be incremented in the diagram in the CDS Guide article. We believe this is consistent with the data standards so do not agree with your proposed change.

Our position (as stated in the MI18 call when this was last discussed) is that anything that results in a user not proceeding to the next step in a multi-step process (prior to the last) should be considered an 'abandonment' for the purposes of the metrics. While the term could be interpreted as an active choice made by the consumer (e.g., click the 'Cancel' button), an agreed purpose of the metrics is what's necessary for consistency and metrics aggregation accuracy. The list of causes for incrementing the sub-counts in the current diagram in the CDS Guide article provides this. Whether the consumer pressed cancel because they can't be bothered at that particular time or they have to catch a bus, or their session timed out or they experienced a technical interruption is not pertinent to the required metrics (as they are currently formulated).

[nils-work]

This issue was discussed in MI18, but no changes to the Standards were made. The closing position is described in this comment and this comment.