Digital Services Veterans Affairs (DSVA), VSA, Facility Locator
Leyda Hughes (leyda.hughes@adhocteam.us), 06/23/2020
View the Research Plan.
VA.gov’s Facility Locator tool displays one of three possible operating statuses when a search is performed, if a facility is not operating under normal hours and services. These statuses are intended to categorize the type of operating deviation to alert and communicate to a user changes that they should expect about the facilities that they may want to visit.
The goals of this study were to:
- Validate or invalidate the usefulness and understandability of operating statuses
- Identify gaps between an operating statuses’ application and how they are interpreted
- Determine how Veterans categorize events that affect a facility's operating status
The research questions we wanted to address were:
- How do people currently get information on facility operating hours/status?
- How do Veterans understand the current messaging/categorization of statuses?
- What information about events impacting a facility's status are important for Veterans to know?
- What terminology is helpful for Veterans to understand a facility's operating status?
- How do Veterans categorize various events that affect a facility's operating status?
- How might we create consistent messaging to Veterans across VA (VHA, Vet Centers, etc.)?
- How do virtual services fit into facility operating statuses?
- How much do people trust the operating statuses?
Prior to the study, we hypothesized that:
- Veterans will not understand what service/access they can receive through the current status messaging
- Veterans want a clear understanding of a facility's operating status
- Veterans will only want to know when a facility's normal operation is impacted, regardless of facility type
- Veterans want to know what service/access they can receive upon viewing a facility's operating status
Remote, moderated research sessions were conducted via Zoom that consisted of a combination of both generative questioning and card sorting exercises.
- 6 Veterans
- 1 Caretaker
- 1 Adult Child of a Veteran
- Are Veterans familiar with the Facility Locator tool?
- How and when do they use it?
- When performing a search, what information is important for them to know at the different stages of a search?
- When shown in isolation of other statuses or messaging, how would a Veteran define and interpret each of the three statuses?
- How would a Veteran categorize and name the different types of events that affect a facility’s operating status?
- When presented with ‘Facility Closed’ participants believed that no services would be available to them.
- 5 out of 8 participants classified a facility as ‘Limited Services’ if virtual/telehealth services were available.
- ‘Facility Notice’ was generally understood as information that a user might need to know about a facility but interpretations varied widely from mask-wearing requirements to permanent closure.
- Users understood that the terms 'virtual services' and 'telehealth' are sometimes used interchangeably and that they mean that some level of service might still be available to them.
- Veterans want to know from the website what services will be available to them.
- Participants expected to be contacted by phone, text or email if something about their appointment changed. Unprompted, they would not think to go to the website to assume something might have changed.
- Participants trust information on the website more when they see a recent date associated with the update.
- Participants don't want to 'dig' or 'click too many times' for important information.
1. When presented with ‘Facility Closed’ all participants believed that services would be completely unavailable to them. When seeing a status of a 'Facility Closed' on search results locations, participants interpreted the status to mean that the physical building was closed and no services, neither in person nor virtual, would be available to them. Some participants interpreted the status to mean that the facility had been permanently closed. When asked about their next step in their search, some users expressed bypassing the 'closed' facility altogether to seek out an open facility as they assumed all services would be completely unavailable.
"Because the first facility was closed, I would call the next one to see if it was open."
"not expecting veterans for any type of treatment, does not have emergency facility, red means stop, not offering anything. Not in operation."
Towards the end of the study, following the card sort and definition exercises, participants were shown a live location that used ‘Facility Closed’ as a status but offered virtual services. This 'mismatch' elicited strong reactions in some participants,
“Facility Closed is garbage! That is like, nope, we're closed. But they are not!”
and
“See, I don't like that. To me that's a limited service.”
2. 5 out of 8 participants classified a facility as ‘Limited Services’ if virtual/telehealth services were available. When given the following COVID 19 scenario, "The Evanston clinic has temporarily suspended face-to-face visits. Enrolled Veterans are still able to receive care during this temporary period with telehealth services available for appointments. Visit the website to learn more about hours and services." and the option to assign from the three current statuses, 62.5% chose "Limited Services", 37.5% chose "Facility Notice", and 0% chose "Facility Closed".
3. ‘Facility Notice’ was generally understood as information that a user might need to know about a facility but interpretations varied widely from mask-wearing requirements to permanent closure. Most participants understood, “something is going on” and that they would need to click or call for more information.
“I would interpret that as limited hours, limited services, or temporarily closed due to an emergency, or cleaning, construction, or changes to their normal activity. That one just has an “i” for information.”
"May have been some changes to hours and services so need to look at website, with covid, I'd call ahead to see if I need to take any extra precautions, like wearing a mask"
"I would make assumption it would give me special instruction at the facility or something out of the ordinary going on at the facility."
4. Users understood that the terms 'virtual services' and 'telehealth' are sometimes used interchangeably and that they mean that some level of service might still be available to them. Additionally, participants want clear details about what services will be available to them and how.
“Telehealth lets me know I will talk to someone, and virtual. I guess they are the same. When I see virtual I think being able to use video. I can see them and they can see me. I would think virtual services would be like getting a doctor to see an injury so I do not need to show up to the site. Have a nurse or doctor check out the injury to see if I need an appointment.”
"My doctor was offering Telehealth, but how do you do labs? Why would I want to just talk to you if I still have to come in to do labs?"
"If there is a list of services, it would say, these are the services and we can do through Telehealth. I guess it could just be a dropdown from limited services with a list."
5. Veterans want to know from the website what services will be available to them and they expect it to be accurate. Participants were given a scenario of planning a visit to a facility in the area where they would be travelling. When asked what information they would need, participants mentioned proximity, hours and services.
“Proximity to me and hours open. And I guess also the service I was looking for was important. One of the facilities that I reached out to didn’t have the services that the website was saying. At least, what I was interpreting the services the website to be saying. They said, that they didn’t have the service the website was saying that they had.”
Upon seeing a listing with “Limited Hours and Services” another participant mentioned,
“I would attempt to click on it and see what services they mean, by what’s limited and hours as well.”
6. Participants expected to be contacted by phone, text or email if something about their appointment changed. Unprompted, users would not think to go to the website to assume something might have changed. During the sorting exercises, participants would often group scenarios that might impact their appointment into a unified category but then go on to say that they would like to be notified. One participant even paused early into the exercise, a little confused and said,
"Are these messages that I’m going to get on my phone?"
This was especially important for Veterans that may need to travel long distances,
“because if you drive two hours to a VA and there is a chemical spill and they are only taking emergencies, but you have an appointment, that’s a long distance to drive only to find out that your appointment has been canceled.”
One participant said,
“I don't like the fact that when it's something that impacts the facility, it's on the Veteran to reschedule.”
7. Participants trust information on the website more when they see a recent date associated with the update. When information is of a temporary nature, specifying a time range for which the change would be in effect, was suggested. When there are changes to normal operating conditions, users like seeing an explicit date of when the website was last updated/reviewed.
"Oh, it hasn't been updated in a year so it might not be as accurate vs. being updated yesterday so it should be accurate."
Even when a date was present but was missing the year, it was still mistrusted,
"Missing the year would make me wonder if this is an old notice. If you see it in that manner I do not know if this is new or old, not having the year associated would make me question it."
8. Participants don't want to 'dig' or 'click too many times' for important information. While most participants expected to be able to click into each listing to get more information about each status, users did not want information "buried" in many clicks and expected accurate hours and services information surfaced at the search results screen. Participants made suggestions like, 'hover over icon for more information', 'add a bullet', and 'add a drop down' to get specific explanations about each status.
I would think I would have to click on the blue hyperlink to find it due to it not being noted or call the contact number to find that out.
"See it still says here to visit the website to learn more. Why should I have to go anywhere else? Why should I have to click on something else?"
"bullet with extra information without the person having to go in"
- Some participants expressed bypassing the facility locator altogether, using Google to get to the facility's website directly.
- Most participants had used the facility locator tool but only on a couple of occasions explaining that they only used it initially to find their VA location or when travelling or moving.
- Some participants felt strongly that the pink status design and placement of 'Facility Closed' was too subtle to communicate its intended severity.
Short Term / Lower Lift
- Reserve ’Facility Closed’ for temporary or permanent facility closures where no services are available to Veterans.
- Use ‘Limited Hours and Services’ as the status for COVID related operating changes if virtual and telehealth services are available.
- Do not use ‘Facility Notice’ as a status for use in search results. It is too broad to communicate anything effectively. See 'Long Term / Heavier Lift' recommendations.
Long Term / Heavier Lift
- Replace or extend the ‘Facility Notice’ category with more specific categories. Participants used words like 'Health Notice', 'Weather Notice', 'Emergency Notice', 'Holiday Schedule', 'Emergency and construction operations', 'Rules', 'Patient Information' to categorize the types of scenarios that affect a facilities operating status.
- Specify which services at facilities are available on detail pages, where available.
- Add timestamps or timeranges wherever temporary changes affect a facility.
- Consider the red styling from detail page instead of pink styling to communicate the severity of ‘Facility Closed’. Some users missed the alert entirely and pointed out that the red styling matched their expectations.
- Communicate changes in appointments and visiting requirements to Veterans via phone, text or email. Let them choose how they get notified.
Present findings to the Facilities team
- Share out insights with team. Get input and insights on recommendations.
Update statuses in search results
- Implement design recommendations
Google Doc Format (https://docs.google.com/spreadsheets/d/1Y91ZNJ_6RN4o9BvGUc0DBqpxnrxg7638-hyVbLrJfAs/edit?usp=sharing)