16. KPIs

16. Identify performance indicators for the service, including the 4 mandatory key performance indicators (KPIs) defined in the manual. Establish a benchmark for each metric and make a plan to enable improvements.

Link to corresponding item in JIRA: https://jira.informed.com:8443/browse/FCOLOI-169

Questions

• Have you identified the behaviour, characteristics and dependencies of the new service and the factors that influence the choice of metrics and data sources for those metrics?
• Have these metrics changed since alpha review, and if so why?
• How are you measuring the performance of the existing (if applicable) and beta service across digital, non-digital and assisted-digital channels?
• How are you capturing user journey data for the new service during beta?
• How are you collecting cost per transaction (or equivalent*) data during beta?
• How are you collecting usage volume or digital take up data during beta?
• How are you collecting completion rate (or equivalent*) data during beta?
• How are you collecting other identified metrics data during beta?
• How are data analytics and user research being used to identify % of users who could channel shift and who will need assisted digital services?
• What are their plans for engaging with service users, delivery partners or other stakeholders during the beta phase?

Evidence

Service Manager able to:

• explain changes to the metrics collected since alpha and the reasons for change (if any).
• show how they currently measure the performance of the existing service (if applicable) and which measures will provide a baseline for the new service.
• show how they are collecting users journey data for the new service during beta (using a chosen analytics package).
• show how they are collecting and calculating cost per transaction (or equivalent*) during beta.
• show how they are collecting user satisfaction data during beta (GOV.UK 'done page' questionnaire or other).
• show how they are collecting completion rate (or equivalent*) data during beta.
• show how they are collecting other metrics data (including journey stage information) during beta.
• explain how they plan to increase digital take up during beta.
• explain how they are assessing potential for channel shift and level of assisted digital services required.
• explain how they are engaging with beta users, delivery partners and other stakeholders to manage transition to live service.
• explain how they will track migration from online to offline.

---

Have you identified the behaviour, characteristics and dependencies of the new service and the factors that influence the choice of metrics and data sources for those metrics? Have these metrics changed since alpha review, and if so why?

Metrics continue to be considered under two main dimensions:

• The choice of service - standard, premium, drop-off
• The type of user - first timer, notary/specialist repeat user, business user

There is not necessarily a one-to-one mapping between these e.g. a notary may use all 3 services and offer the choice to clients.

Metrics have evolved since alpha review in tandem with the development of the product e.g.

• Increased focus on accounts and repeat users, once the foundational standard 'guest' application was built
• Incremental addition of features to accounts, and monitoring of feedback. Save a card -> save personal details -> save addresses

How are you measuring the performance of the existing (if applicable) and beta service across digital, non-digital and assisted-digital channels?

Much of this is covered in detail in point 15 Data analysis, as far as the private beta and AD performance - piwik, surveys, direct feedback to dedicated mail address, usability testing, etc

Non-digital performance is tracked using the business KPIs, detail below

How are you capturing user journey data for the new service during beta? How are you collecting cost per transaction (or equivalent) data during beta? How are you collecting usage volume or digital take up data during beta? How are you collecting completion rate (or equivalent) data during beta?

KPI	Current Data source	Future Data source	Benchmark
Cost per transaction	Internal cost model against number of apostilles issued	Internal cost model against number of apostilles issued	Cost: £2,156,721 Volume 471,716 Cost per apostille: £4.57
User satisfaction	GOV.UK done page survey; private beta survey	GOV.UK done page/feedback survey	Legalisation and Apostille certificate application: 84.2%; Document legalisation: Payment for drop off service: 87.5%; PRivate beta feedback survey so far: 86%
Completion rate	Piwik analytics	Piwik analytics	Legalisation and Apostille certificate application: 68.4% Document legalisation: Payment for drop off service: 84.3%
Digital takeup	-	Iizuka casebook - tracking different sources of applications	Patent renewals 90%, Planning applications 94.5%, internal benchmarks for online payment 99%+

How are you collecting other identified metrics data during beta?

Piwik (now) & Casebook (April)

How are data analytics and user research being used to identify % of users who could channel shift and who will need assisted digital services?

For AD see all the information on point 1 User needs about identifying AD users. Our ongoing transactional research now includes the digital inclusion scale.

Our channel shift plans are covered in point 14 Digital takeup - we have a very high target, driven by past success with moving to online payment.

What are their plans for engaging with service users, delivery partners or other stakeholders during the beta phase?

See point 14 Digital takeup

---

Further background from alpha, including offline service measurement

How do we currently measure the existing service?

The overall service is measured by the following KPIs:

Ministerial targets – All applications are processed in line with a defined Ministerial target. They are:

Service	Target	YTD performance
Post	100% in 48 hours	87.6%
MK drop off	100% in 24 hours	100%
Premium Service	100% Same day	99.8%

Cost per transaction – The cost of the service is calculated on a ‘cost per apostille’. The current cost is £4.57. If calculated on an application basis it is c. £18

Average output – Individual performance is measured against pre-defined targets. This allows us to calculate resourcing requirements to deal with the volume or work received (55 envelopes per day). The benefits of the new service will allow us to offer a far more efficient back-end service, with output increased by over 50%.

Customer satisfaction – Customer satisfaction is measured against a baseline figure of 80% (rating the service as either good or very good). Current satisfaction sits at 82% (IFF research).

Non-straightforward applications – Just under 10% of all applications received are considered non-straightforward (either incorrect or a signature query). With the benefit of the new application service, we have a set a target for a reduction of 2.5%.

Have you identified (in addition to the 4 mandatory KPI's) other measures for the new service and where the data will be provided from?

Initially we will focus on the four core KPIs plus:

i. Non-straightforward applications.

ii. Individual productivity.

The previous measures for Digital Take Up and Completion rate were misleading or not measurable, covering only a very small part of the process. The new service will be a fairer reflection of these KPIs. The key benefits of moving to the new service is to offer customers a simpler, more efficient service. The two additional KPIs are key to evaluating the success of this project. We will use the benchmarks provided by a time and motion study completed in 2014 to measure individual output and service efficiency.

Have you investigated how the service is expected to perform in comparison with similar services?

Measuring against other similar services is challenging. Using the www.gov.uk/performance/services information, the following services have a similar volume of transactions and may offer an indication of the digital take up KPI that we should be expecting:

Service	Volume of transactions	Digital take up
Land registry: Searches for title numbers	486,762	87.5%
Planning applications	483,257	94.5%
Patent renewals	407,436	90.7%