Site flow findings and recommendations
- Searching for "part 433" or "42 CFR 433" should return a reasonable result on the search results page, ideally the part table of contents page as the first(ish) result.
- Explore hypothetical designs for multi-source search, incorporating the ideas people have shared with us (such as supplemental content link text and the preambles for relevant rules).
- Update "About Medicaid & CHIP Regulations" link text to “About This Tool” for clarity, but also update the “Medicaid & CHIP Regulations” title of the site to “Medicaid & CHIP eRegulations” to reflect that this website is a unique compilation of information and tools, not just raw regulation text.
- Move forward with a combination of both — helping people to get to the eRegs homepage ToC and improving search
- Can implement some of the lower lift homepage improvements quickly
- Might not have time to implement a completely different homepage before September, but can at least explore design options of a complete redesign
- Include some sort of effective date on the reg text page
To date, we have used Figma prototypes to test features, primarily comparing two design approaches to see which performed better.
With this study, we wanted to test the site's flow with the working draft to get a better understanding of how well the eRegulations pilot flows in a real web browser (in a holistic sense), since we have refined numerous features based on feedback from users.
Key findings include:
- All participants expected to use the search bar as a navigation technique to access citations
- Participants who used the Table of Contents as a navigation technique were less confused by the search results than those that used the search bar
- The search feature will need to accommodate navigating to parts/citations in addition to keyword searching
- The Jump-to feature performed well for those that used it to get directly to citations and to switch parts
People continue to be enthusiastic about using eRegulations and have high expectations for it:
- P1 about the live working draft: "I didn't really get a chance to see any, like what the sub-regulatory [guidance] would look like along the way here -- that just might not be ready yet -- because that stuff is incredibly helpful, especially for the stuff I do. Because as I said, the regulatory text itself hasn't been incredibly hard to find with the existing tools. It's having a tool that would give me a “look forward and back, and then who's talked about it in what way?” that's in one spot, which would be seriously helpful because that does require going to 24 different places right now."
- P2 about the Figma prototype: "I really like this, having the NPRM and the final version tied together. That's exactly what I was hoping for. That's great."
- P3 about the Figma prototype: "If everything's in one place that would be very helpful for me instead of trying to piece it together... and then having the tools to go along with it, that just makes it a little bit easier too -- all the state state letters to the Medicaid directors -- anything to make it more efficient I find it very helpful."
- P4 about the live working draft: "Oh gosh. Wow. This is unbelievable. Really. I mean, when you did the training at the leadership meeting, I was like, oh my God, I've been waiting so long for something like this. Like I thought, I mean, to be honest with you, I thought medicaid.gov, once it was up and running, would be a bit more like this. This is, this is just so amazing."
- P5 about the live working draft: "I like what I'm seeing so far. It's very clean and you know, I like how it's broken down by each Subpart and I can go directly to the numbers. So it's extremely user-friendly from my initial glance and overview of it. I like how I like how it's broken down by subpart C, D. This is much easier than what I'm used to looking at."
We wanted to know whether the current navigation was sufficient for helping people find the information we asked them to find, as well as finding their way back to earlier starting points in the flow.
We also wanted to know whether the updated design for "jump to" part/section makes sense to people.
Participants were more likely to use the homepage ToC when it was visible "above the fold" or when they were motivated to scroll
The majority of participants used the search bar to navigate because the ToC wasn’t visible in their viewport and they didn’t scroll down before deciding on an action. Since the majority of participants didn’t see anything below the features row, the most prominent action in their viewport was “Search Regulations” in the search bar.
- The homepage ToC was partially visible on P2’s screen only
- Nothing below the features row was visible on P1, P3 and P5’s screens
- Nothing below the homepage hero was visible on P4’s screen
- P5 was the only participant to immediately start scrolling
- P2 and P5, the participants who saw the ToC, successfully used the ToC to navigate to Part 433
- P1, P3 and P4 used search to navigate to Part 433 and did not scroll down without prompting
The three participants who used search to navigate were unable to quickly and easily navigate to Part 433. Because the existing search feature uses keyword search, which isn't as precise as navigating via the homepage ToC, it's not currently a viable navigation solution. Since the participants are accustomed to using Google to search for specific citations, we should eventually work on refining the search feature, but the higher priority is helping more people use the ToC to navigate.
Since the two participants who saw the ToC decided to use it, we speculate that more people would use the ToC if were visible in their viewport or if they were aware it was on the page. A variety of solutions can be explored to mitigate these issues, including decreasing the height of the content above the ToC or introducing UI elements that motivate users to scroll. See the "Recommendations" section below for details.
The same amount of content was visible on P1 and P5’s screens. Below is a screenshot of P3’s screen:
Even less of the homepage was visible on P4’s screen:
Nothing below the features row was initially visible on P5’s screen, but they immediately started scrolling. While their first thought was to use the search feature to navigate to Part 433, they ended up using the ToC because they had previously scrolled down and saw the ToC.
P5: So what I would do is I would just search under that quite frankly and see if it takes me there. So if you'd like, I can try that or I can scroll down to Part 433 since it’s right here.
Part of the homepage ToC was initially visible on P2’s screen only. Because they could immediately see more content below the features row, they only had to scroll down slightly to understand that the Part numbers were listed on the homepage.
P2: I would scroll right down here because I could see immediately, right when I opened up the screen, I could initially see just from scrolling down a little bit, I could see that the part numbers started to be listed. So I scrolled down to see Part 433. If I were maybe on a smaller screen and couldn't see that they were listed down here, I might've clicked “using this tool” to see how it was intended to be laid out.
Recommendations
As a low lift, interim improvement, we could explore ways to hint that there’s more “below the fold”
- Add a prominent “Jump to ToC” button in the hero
Additional improvements could be made by exploring ways to make the existing hero and features row shorter
- Make the illustration smaller
- Reduce the amount of copy in the hero and features row
- Introduce the ability to collapse the content above the ToC
- Remove the features row entirely
The most effective improvements could be made by exploring a complete redesign of everything above the ToC
- If using the ToC is the primary action we want people to take on the homepage, part of the ToC should be visible on a wider range of screen sizes
- Reduce the number of links to the About page
- There are currently six links to the About page above the ToC, but perhaps the link in the flash banner is sufficient
- Add visual interest without occupying as much screen real estate as the current illustration
- Consider using background textures / graphics / colors
- Explore ways to make the search bar less prominent
- If we used the standard header on the homepage, there would be less contrast between the search bar and the header background
- Add a version of the left sidebar nav’s “Jump to Regulation Section” feature to the homepage, grouped with the full ToC
Regardless of which paths we explore, we should do further usability testing to validate whether or not the revisions actually help more people use the ToC.
Three participants used jump-to with ease. One participant didn't notice it, but when asked about it, was able to intuit its function.
P3 was unable to use the Jump-to feature due to browser-rendering issues:
We wanted to get a sense of whether the search capability we are offering is adequate for early adopters.
As stated previously, we found that participants were more likely to use the homepage ToC when it was visible "above the fold" or when they were motivated to scroll, and we detailed how the hero image eclipses the primary action we want people to take on the page—using the Table of Contents (ToC).
Because the ToC wasn’t visible to most participants, four out of five participants expected to be able to use search to navigate directly to a regulation citation (e.g., Part 433).
- P1, P3, P4, and P5 all had the initial thought to search to navigate to a part when asked to find one
- P1, P3, and P4 went forward with using search to navigate to that part
- While P5 initially thought they would use search to navigate, they decided to use the ToC instead, likely due to scrolling down the homepage more than the participants above who used search
- P2 is the sole participant who used the ToC for navigation, but the ToC was more visible to them upon page load than every other participant. However, when asked to directly interact with the search feature, the first thing P2 thought of to search was an exact citation (435.917).
At this point in time, the eRegs search feature only supports keyword searching, and does not support exact citation search in a way that aligns with user expectations. Relying on search as a way to navigate directly to parts (or exact) citations unfortunately resulted in confusion among some participants.
- P1 and P4 searched for “Part 433” and couldn’t understand why they saw 435 first in the results and not 433
- P1 thought it strange that “the first result wasn’t a generic link to 433,” which led them to assume the external link to Beta eCFR’s part 433 would be the place they’d have to get used to using in order to get to the needed part.
- P4 was confused to see 435 first, and that they felt they were being directed to 433.15. “I would expect to be able to go to the whole part 433.”
They probably expected to be able to use search for navigation because they’ve learned to use other search engines like that and expected they could search for anything and the engine would infer their intent. What they saw in their viewport suggested to them that this might be possible because the search bar was, in effect, the only available action to them because the ToC was not visible.
We believe people aren’t able to search effectively or at all with classic eCFR (other than ctrl-f on a page), so they use Google to find relevant information. Because eRegs looks modern in comparison to existing online regulations tools (and of a similar quality to Google, according to one of the participants in this study) people likely have high expectations of eRegs without realizing it — they are used to searching citations in Google. This is an area of further study.
P1’s experience — they started to scroll down after the facilitator invited them to scroll around and get acclimated to the page, but they mentioned they would typically use the search bar:
P1: And so if I just — if you want me to think out loud — if I was just getting here and didn't realize that I was just exploring the page and scrolled down and saw it right there, State Fiscal Administration. I mean, I would probably pop up here — and you didn't give me a part, but I'm assuming it's 42. Part 433. 42 CFR 433. The title is 42. Yes. Yeah. So I'll just plug in the 42 CFR 433 into the search part here. And now let's see what would pop up.
As a result of using search to navigate to Part 433, P1 found it strange that the first result wasn’t a generic link to Part 433, and assumed that they would have to go to the right sidebar and click on 42 CFR 433 in Beta eCFR to get to 433 directly:
P1: Well, it was a little bit strange to under the, that the very first search result isn't just a generic link to 433. You got to come over here to do that. But if that's by design, I would get used to that pretty quickly to be like, okay, well, this is where the stuff is. Like, I don't know how it popped up with 435.4 as the top hit. Yeah. I will look at the reason why that's there and then why .15. But again, if, if the idea was I knew that I wanted to sort of 33 broadly, 433 broadly now I knew that the way it worked was it was going to pop up over on the right-hand side. Like it wouldn't be the end of the world. -
P1 also mentioned that the search results page felt and looked like Google:
P1: It feels like Google, it looks like Google looks like, all right, huh?
P4’s experience — They was drawn to that part of the page because it said, “search regulations.”
P4: I would search for the… I would search here. Do you want me to type it in? Yup. And that's Part 433 State Fiscal Administration. I'm curious what drew you to that part of the page? It said, “search regs, search regulations.”
Similar to P1, P4 was confused to see 435 first — “I would expect to be able to go to the whole part 433”. P2 interpreted this as being directed to 433.15 instead of Part 433:
P2: Well, first of all, I'm not sure why 435.4 popped up and then this one, 433.15, I guess like I would expect to be able to go to the whole part 433. So if this is just one part for the entire 433, and I could see the whole 433, it would be fine, but I guess I wasn't… I wouldn't be sure, like why was I directed to .15?
P3’s experience — when asked to find Part 433, P3 did not scroll down the page. They immediately went to the search bar to navigate to 433.
I would try to go up to your search engine. Yeah, go ahead and give that a try. Okay. Hold on. Good. - [P3]
P3 saw the search results page, but the facilitator redirected P3 to find another way to get to Part 433 (using eRegs homepage ToC) to complete that task instead of pausing to focus on search result page questions. Due to technical issues with the browser, the facilitator was not able to return to evaluate search with P3.
The two participants who used the ToC to find a part were not as confused by the results they saw, and both focused on other aspects of the search results page for feedback. (However, one of those participants still wanted to use search to navigate to a citation.)
P2’s experience — After they used the ToC (instead of search) to find part 433, we directed them to the search feature to observe how they used it. When asked about the most recent thing they tried looking for, they said:
P2: I can try searching for 435.917. That's the regulation I had to go to recently. And a question that pops up immediately is whether or not I need to type 42 CFR, or if I just need to type the, the last part of the regulation.
After informing them that eRegs does not fully support using search as a way to navigate, they offered the phrase “beneficiary eligibility notices”, but was a little surprised by the results — they were definitely related citations, but not the ones they were thinking of:
P2: There are several regulations that are related to that. 42 CFR 435.917 is related, 431 — I'm forgetting the citation — but 433.110 I believe is also related. So this one, this regulation is related to the requirement for the Medicaid agency to coordinate with the marketplace and with the CHIP program to provide a coordinated eligibility process for the beneficiary. And this citation is related to a requirement for states to allow applicants to select an authorized representative when they're going through the eligibility process. But there are other citations that are related to the requirements for the notices themselves.
P2 had feedback on the search results they saw after searching for beneficiary eligibility notices. The second result’s keyword highlight occurred in the middle of the snippet (quoted piece of the page), but the first result was more confusing. The highlighted keyword occurred at the beginning of the snippet, making the result feel out of context:
Facilitator: Is it displayed in a way that makes sense to you?
P2: Yes and no. I really liked the way that it says Part 435 and then it provides the, the title of that section. And then I like the way that it shows the citation itself and the subject of the citation. But for this one, for example, it is a little out of context, the way that it just starts off with the eligibility. If I were unfamiliar with the citations, that should be… that should be pulling up based on this type of search, I would have to go into the citation to see exactly what it's, what it's referring to. And that's not unfamiliar, that happens with other search features as well and on other sites, so… something that could be worked around, but I like the way that this citation explanation like this detailed information is pulling up a little bit more. It starts off a very clear, I think this is the beginning of the regulation. I describes the content a little bit better.
P5’s experience — When asked to search for something they recently had to look up, they searched for HCBS (Home and community based services). When asked if the results were displayed in a way that made sense to them, they read the result headings aloud and said they were pretty clear. They were interested in being able to refine their search. We explained that we didn’t support filtering yet and we asked if it would be helpful, and they concurred:
P5: So, okay. Here's a question. What if I, now that I'm in this section and I want to know specifically about 1915 C, can I do a search within this? So now that I've I've, I'm connected to HCBS, can I refine that search? That's a question.
Facilitator: We don't have filters available at the moment. Is, is that, is that something that would be helpful?
P5: Yeah, absolutely.
Recommendations
Based on this research, most people expect to use search for both regulation text keyword queries and direct navigation to citations. In previous studies, participants wanted to search for keywords among supplemental content as well. We don’t have substantial supplemental content in the application yet, so this study did not provide evidence related to that, but it may also still be true.
Consider the following:
- Explore ways to support both keyword searching and using search to navigate to specific parts and full citations.
- Make the ToC more visible for navigation to offer people an immediately viable way to find a particular part or section.
Discerning what body of content eRegs is searching is difficult, despite textual cues on the homepage ToC and search results page
We wanted to know if participants had an idea of what content the search bar was searching. Everyone had difficulty answering this -- partially because it's an awkward question to ask, and partially because no one had yet formed a mental model of the website that matched what eRegs offers. Only two participants scrolled down to see the ToC below, which is meant to give a hint at the website's content, but that did not help participants answer the question.
Two participants wished to search the whole site for a topic, not just the reg text, but all of the available material. Those that were able to see the supplemental content prototype wished to search within in it as well.
- P1 wished to see subreg guidance on the search results page, as well as search within the supplemental sidebar.
- P4 also wanted to search within the supplemental content to find subreg guidance.
- P2 commented on the ordering of subreg guidance by date making sense, unless you could filter by topic, which is another kind of search.
We wanted to know if participants could find information about when the content was last updated from the eCFR, as well as how the site was updated.
- 3 out of 5 participants were able to find when eRegs is updated (in the footer).
- Conversely, P1 said they would just trust that it was an updated source, and P3 found information about the cadence of updates on the About page
- Three participants found the About page in the footer and followed the link to that page
- Another participant found the About page with the home page's "Learn more and contact us"
- P3 had difficulties with site functionality due to screen resolution settings, so the facilitator screenshared the page. P3 asked whether the product would eventually be under oversight and control.
P5 wanted to know how they would be able to tell they're looking at the most up-to-date version of the regs. This UI detail was present in a previous wireframe, and this inclusion would benefit users greatly.
"How would I be able to tell if I'm looking at the most recent version (as opposed to an older version) if it's been updated or not? How would one know? Cause like, if you have someone that's in this... this happened to me when I was a new employee and I didn't realize it had changed and that it wasn't up-to-date (because it wasn't an area that I was familiar with) that I needed to reference. How can a user know that you're looking at the most recent?"
We gathered additional feedback from participants to help in revising content on eRegulations.
- P2: “I really like this — having the NPRM and the Final Rule tied together — this is exactly what I was looking for.”
- P2 said that ordering subreg guidance by date makes sense, unless you can filter by topic, and Prefers supplemental content related to specific reg section, not subpart, but knows it’s hard. Subpart resources need to remain visible as a user scrolls down
- P4 noticed the share button and pointed out that we may want to consider making it known whether this is an internal versus external site, so people know whether they can share things to people.
- P5 said “I don’t know what this is for” about the anchor link icon and wondered whether it would take her someplace.
"About Medicaid & CHIP Regulations" does not accurately convey to participants what the page is about
The link text in the footer for the About page did not give enough information for people to understand what this page is about.
- P4 expected to see general information about how anything that had an impact on the site, maybe something about the regulatory process or a very general description of Medicaid/CHIP
- P5 thought it would take them back to Medicaid.gov to see more specific information about Medicaid and CHIP.
- Both P1 and P2 talked about needing more information around the Product Owner.
- P2 specifically shared that "from experience, I know that sometimes contact information gets put on a website and sometimes it doesn't get switched out. And sometimes when you're looking at a contact page and it's not clear what you're intended to be contacting them about, it feels a little uncomfortable. I don't want to spam people. I don't want to waste anyone's time. So with this, with this message, I would probably have to spend a little bit more time searching around, making sure that there is no other contact information for a website issue or something like that before I would actually reach out."
- P4 asked "What does “coming soon” mean? 2 months? 2 years? A nice to have, I’m excited by it, not confused."
- P4 also thought we could be more descriptive: "People are going to be surprised that you are going to have the regs with the preambles, and you don’t really get to that until you get down.. perhaps the one thing you could do is be exhaustive in this row… for sub regulatory guidance you say “learn more” but say more… like mention toolkits."
- P5 gave food for thought: "Is this a resource or a tool? I would till consider it a resource for me because it’s here to help/assist me. Is it necessary to distinguish or are they interchangeable?"
- Before we share the link with CMCS for asynchronous and synchronous feedback going forward, we want to catch any major interface issues in time to make improvements to the working draft.
Wayfinding
- Is the current navigation sufficient to help people find the information they are asked to find? Are people able to find their way back?
- Does the updated design for "jump to" part/section make sense to people?
Search adequacy
- Is our current search system sufficient for early adopters in July?
Timeliness of Content
- If people are curious about how the site is updated, can they find that information?
- If people want to know when the content was last updated from the eCFR, can they find that info?
We recruited 6 people all involved heavily in policy. We wound up with 5 participants total.
We will conduct a moderated formative evaluation that will generate qualitative data that will inform iterative improvements to the working draft.