15. Data analysis

15. Use tools for analysis that collect performance data. Use this data to analyse the success of the service and to translate this into features and tasks for the next phase of development

Link to corresponding item in JIRA: https://jira.informed.com:8443/browse/FCOLOI-168

Questions

• What have you instrumented and why?
• Have you modelled user journeys, and are you able to track progression through your service so you can identify completions and areas of poor performance?
• Where appropriate, have you built a funnel, exit paths?
• What tools are you using to collect data?
• Has the SIRO signed these off?
• Where appropriate, have you anonymised the user IP address, have you opted out of data sharing with 3rd parties?
• What analysis has been carried out on the service so far and how has this impacted on the backlog?
• What is the ongoing roadmap for performance analysis, including performance of assisted digital support?
• Who in the team is responsible for identifying actionable data insights, including for assisted digital support?
• What is the next performance analysis user story?
• Have you started discussions with GOV.UK about start and end pages?

Evidence

Service Manager able to:

• explain what data sources and analysis have been undertaken in the alpha stage.
• explain how the shape of the service has influenced the choice of metrics, data points and data sources.
• explain the choice of analysis tools used in the beta (and alpha if appropriate).
• show that appropriate information security and privacy issues have been addressed.
• explain how they have modelled user journeys and will track progression through the service so they can identify completions and areas of poor performance in the beta.
• talk clearly about evidence from qualitative and quantitative data, what they learned from these sources and what changes to user needs/improvements they identified.
• talk through how these were prioritised and what features were changed or implemented.
• talk about the ongoing roadmap for performance analysis, and explain who in the team is responsible for identifying actionable data insights during the beta, including for assisted digital support.
• explain the next performance analysis user story.
• show they have discussed and agreed start and end pages with GOV.UK and these are optimised

---

What have you instrumented and why?

Have you modelled user journeys, and are you able to track progression through your service so you can identify completions and areas of poor performance?

Where appropriate, have you built a funnel, exit paths?

What tools are you using to collect data?

Has the SIRO signed these off?

Yes

Where appropriate, have you anonymised the user IP address, have you opted out of data sharing with 3rd parties?

What analysis has been carried out on the service so far and how has this impacted on the backlog?

What is the ongoing roadmap for performance analysis, including performance of assisted digital support?

Who in the team is responsible for identifying actionable data insights, including for assisted digital support?

What is the next performance analysis user story?

Improved instrumentation of accounts section
Eligibility checker improved tracking including search

Have you started discussions with GOV.UK about start and end pages?

---

Alpha answers

What data and analysis have you done during discovery from existing or legacy systems or the wider digital landscape?

• Anonymous GOV.UK feedback for all the legalisation pages: analysed and summarised (see point 1)
• Google Analytics on GOV.UK pages: browser info and general metrics e.g. 92k pvs, 36k users (see point 1)
• Peter Jordan's report on Google Analytics, Trends, Hitwise: how users find the legalisation pages, highlighting:
  • Confusion about navigation
  • "apostille" is the #1 search term on all the key legalisation pages -> added in alpha
  • rank well for legalisation searches, but results also cover drugs (especially news results)
• Content: request outstanding with Gwen / GOV.UK content team for refreshed path and on-page search analysis for business users, wrt conversations about start page (due in August, currently on content team todo still)
• Analysis of audience numbers, sampling
• Detailed IFF qual user research per point 1
• Internal process report: data on process timings, re-typing and numbers of errors/un-legalisable docs (10%) from commissioned report

What analysis have you made on the alpha?

• Usability testing per point 1, on both existing pages as baseline and our prototype. Remote and pop up in person.
• Detailed IFF quant user research per point 1
• Data collection and initial interviews for assisted digital needs

Who in the team is responsible for analysis during the alpha, including for assisted digital support?

• Prototype: led by Mark and Mike
• Assisted digital: Alison Diaz, lead for AD and digital takeup; IFF research & Consular Insights team