Medicaid and CHIP regulations user experience

To inform and prioritize what CMCS eRegulations should provide, we've learned from a variety of people at CMCS about how they use regulations at work, including their methods, tools, ideas, and feelings.

Our focus for this pilot year is making eRegulations available to CMCS employees. We also talked with some CMCS contractors and people who used to work at CMCS to gain a broad perspective. Eventually, the goal is to have eRegulations available to State agencies, so interviewing State staff will happen at a future stage of this project.

Usability testing sessions (January - June 2021)

We have conducted usability testing sessions with many CMCS staff members in a range of roles. During these sessions, our primary goal is to gather specific feedback to improve the eRegulations tool, but people also share useful background information about their work, interests, and needs. Here are a few highlights from these sessions.

People want the information in this tool for their work.

Most of the people who tried the prototypes in usability sessions expressed excitement about using CMCS eRegulations!  There is pent-up demand to use a tool that gathers contextual information about Medicaid and CHIP policy in one place. We can expect that people will tolerate minor UI issues if they can access and use the content (either regulations or related supplemental content). Our team should focus first on the findability and usability of content, then a beautiful user interface. (Note that this is different than most private sector design where most consumers have come to expect and require a slick UI.)

People develop unique personal research strategies and help each other with research.

The audience for this pilot are CMCS staff who interact with Medicaid/CHIP regulations as part of their work. Some use regulations on a daily basis and are very proficient with policy research, while others use regulations less frequently and are not as familiar with policy research. We have learned that people don't always have good tools, so they often help each other out when looking for and consulting resources. Each person develops idiosyncratic tricks over time to help streamline their research.

People mostly use Google and the original eCFR.

Even if they have go-to regulation site resources, people often use Google to search for citations and phrases because it is often the fastest way to find something. People may prefer one cross-reference resource because they trust their own vetting process, but will often suggest a different resource be used because it is more reliably current. Interestingly, no one we talked to was aware of beta eCFR (https://ecfr.federalregister.gov/). Although many people we talked to used "the original" eCFR, the text that links to the beta eCFR is not obvious that it is a link to the beta site.

People research policy using multiple tabs and windows on multiple monitors.

Most people we have met with use laptops and external monitors. A few still use CFR books for research, and a few other people mentioned that they previously used the books but left them at their offices when they shifted to remote work because of the pandemic. In usability study sessions, many people commented on having multiple tabs and windows open on multiple monitors to look at multiple things. We tried to help people stay on the page by designing cross-reference and definition pop-ups so they wouldn't lose their place, but this solution often blocked the main (reg) content on the page. Our findings indicated that our audience considered multiple tabs/windows a successful and practical (though sometimes frustrating) experience, rather than being seen negatively or as an unfortunate reality.

People rely on underlined links and labeled buttons for navigation.

There are a lot of different preferred navigation styles, so we designed eRegulations to accommodate most of them. People rely on links being underlined, otherwise text will not stand out as being a web link. Additionally, icons do not stand out to people, but when asked, they prefer labeled icons. What people do notice is the regulation text itself, down to the detail of paragraph indentation levels.

Semi-structured interviews (October - December 2020)

We sourced interview participants via recommendations from our CMCS project partners (including our Product Owner) and team domain SME, as well as through snowball sampling (asking participants to recommend people to talk to). We also learned from informal discussions with people about their personal experiences, including our Product Owner, other DSG team members, and a few other people we know use Medicaid/CHIP regulations.

People using regulations default to using Google when existing tools aren’t sufficient or easy to use.

Everyone needs to look for things: While policy experts had a firm grasp on where to find the regulations they were looking for, especially within their area of expertise, other frequent regulation users were, at times, less confident about knowing where to look or how to find specific information. Policy experts also need to search around when working with regulation parts outside of their specific area of expertise.

eCFR is the most popular regulation reading tool: Respondents described using a wide range of tools to find specific regulations and supplemental content. eCFR (classic, not beta) is the most commonly used tool, and some people also use printed CFR books. People frequently save links/bookmarks and sometimes print specific regulations themselves.

Google is the most popular regulation search tool: People we spoke with, especially those with less experience or working with a wider range of regulations, commonly mentioned Google as a place to start. One respondent mentioned googling “IMD Exclusion, because if I search the CFR I can’t find it.” Another respondent mentioned “if I encounter [the need to look up reg references]... I'm looking to Medicaid.gov and just doing a search or just Googling it, which shows Medicaid.gov references usually anyway.”

People frequently need to confirm what information isn’t in the regulation, and there is not an easy way to do that.

Searching, whether through Google or using a different tool, doesn’t always yield the needed information. Several people we spoke with discussed the challenges of searching for information that is not in the regulations. One common barrier mentioned in our discussions is that the “terms people used to refer to certain things aren't necessarily the terms that are literally in the regulation.” One person we spoke with discussed this challenge when working with state officers: “State officers have a deep need for getting answers to questions about ambiguous policy, [but] the answer may not exist in any written form.” They went on to mention that “State Officers will try and look up information with terms that don’t get them the information they need.”

People may use terms that don’t mirror the terms used in regulation. People also need to understand when a term has a specific regulations-defined meaning and what that meaning is.

Some terms have definitions in the regulations that may or may not align with vernacular usage. These differences are crucial to understand when reading regulations but may be missed if they are subtle. Readers may not not know to look for them if they are less familiar with regulations. One user noted “in the context of our work... we have specific meanings for enrollment versus eligibility, and a lot of people think those terms mean the same thing.”

A different user found value in a tool that helped users see definitions in the context of reading a regulation. They noted, “Cornell's system... has identified various terms that you can either mouse over click on... and pull up kind of the regulatory definition. I kind of like that ability to kind of see, well, what exactly do you mean by ‘insurance affordability program’ and that kind of thing.”

Through our conversations it also became clear that even when the users felt confident about the definitions themselves, they wanted to ensure the state implementers were able to connect the information just as readily through the sharing of specific parts of regulation and other policy.

To overcome websites with poor navigation, Search Engine Optimization (SEO), or general user experience, people bookmark frequently-used parts, print out what they need, rely on the printed manual, or create other solutions (like creating a spreadsheet).

People have to find specific pages: When sharing and linking (as well as bookmarking), users favored tools that would allow a greater degree of specificity when needed. When tools give users more generic links, the relevant information can be buried in the page or document. A State Officer told us “I don't just give them the reference. I actually tell them where in that twenty-five page doc, they can find that answer.” Another respondent told us that “people rely heavily on experience [to find regs]... it took me a long time to figure out how to access the regs on the internet, I think I needed someone to send me a link that I [then] bookmarked.”

People annotate the regs: While bookmarking was the most common way users mentioned finding their way back to specific regulations or guidance, we also spoke with a respondent who had copied and pasted regulations into a spreadsheet to more easily access, search, and find information again. The spreadsheet also allowed this user to annotate or add contextual information (such as search terms) so that it was accessible the next time it was needed. This kind of personalization may be a reason some people prefer analog versions of the regulations. Two people we spoke with mentioned “many [people] that have dog eared copies of their CFR” and the other echoed the sentiment saying “highlighting is something people will like. Highlighting is what they want and then tagging.” This type of documentation may also help people using regulations and guidance documents feel more confident they have found all of the relevant information.

Many people had anxiety around feeling confident a tool had all related policy guidance.

You have to figure out if you've found enough: Many respondents mentioned having some anxiety about finding the full scope of information. With Medicaid/CHIP policy guidance being found in a variety of documents and being updated in irregular intervals, even expert users find themselves wondering if they’ve found all the applicable information. In early prototype testing where supplemental information related to the regulations was listed by date, one user said “I just don't want to have to question myself and be like, ‘this is a really cool start, but do I need to use it as a start and then go somewhere else?’”