This sprint, we did two main types of research: user interviews, in which we asked people to explore the site while talking through their responses, and content testing, which pulled content out-of-context and asked users to highlight confusing (or clarifying!) phrases. See the full research plan, interview script, and content testing script for details and methodology.
Our hope in this sprint was to get more clarity about which parts of the website were confusing users, and which — if any — were helping them understand 18F.
For overall usability testing, we talked to:
- 3 U.S. government employees, all military (2 Air Force, 1 Coast Guard)
- 1 Canadian government employee
- All had some level of familiarity with either 18F or technology and acquisitions innovation work
For content testing: We did 3 content-testing sessions with GSA PBS employees in San Francisco. All 3 had heard of 18F, but were otherwise unfamiliar with what we do.
The featured project on the homepage (Every Kid in a Park) caused some confusion.
I see pictures with kids — great! But...then I have to scroll down to what we deliver, and even then I’m not entirely sure [what 18F does].
Now that I know that it’s basically websites, web systems, and so forth — it makes the kids in National Parks thing more confusing.
I feel like “what we deliver” is the main point of the homepage — I want there to be more information and less reading.
People are confused about what 18F does.
I guess I don’t know what level you guys are trying to get your customers or your projects at.
You guys do so much, it seems, and you’re open to so many ideas I don’t know if there’s certain things you specialize in, that you could highlight on here, i.e. here’s where we do things related to acquisition, or database construction.
You’re definitely gonna need background information about 18F before coming to this website.
Why not just put ‘what is 18F’ or ‘who we are’? You’re trying to go too many directions!
People don't know how to start figuring out whether their agency could work with 18F.
I don’t know what point you start at — if you have to be a manager, or if you need funding, or anything.
I like the part about becoming a digitally-powered organization, where you could embed a team — but there were no links to learn more about that. Is is done under a contract? Are there certain times you’re available for? How would you do that?
The big disconnect is “how do you work with you guys?”
Is [the intake email] a generic email that kind of gets filled out on your end, or does it go to a specific person?
People aren't sure how to envision an 18F-shaped project, and are eager to learn more about projects we’ve worked on.
Would be nice to have a section where people could go and see what you’ve done for other folks, and what problem you’ve solved.
I like the fact that you have the agencies highlighted, like you can click on the agencies, but the real interesting point is what you’re doing with those agencies. The projects are kind of the pull that will get us clicking, more so than the agency.
I don’t think people have a good idea of how big or how small their projects could be...maybe some kind of link, like...here’s the kinds of projects we’ve done for agencies. A lot of the big problems facing our agencies might be a series of small projects — helping them build a plan would be good. A lot of folks from agencies, like myself, would not have any idea what I need unless I had a consultant walk me through it.
Links to guides can be confusing, especially as the primary way to explain crucial elements of how to work with us.
When I look at your guides section...which is how we work...If you want to talk to a department, they might not know what these guides mean. I would present the guides differently: your guides are awesome content, but they’re for an intermediate to expert audience.
I branched off into the guides, and I want an easier link back.
“Our DOD system won’t let me go to your Partnership Playbook. At the Pentagon there’s a lot of websites we’re not allowed to go to...there’s a problem connecting securely to this website.
In all 3 sessions, participants used only the orange highlighter. (This may be because it is much less instictive to meta-cognate about things that do make sense; future test designs should adjust for this possibility.)
These sessions focused on testing 2 pages of content, including the current homepage copy and several new paragraphs being considered to describe 18F.
Another said “I had to look up the word ‘technologist’ that you use a lot on the website, because I didn’t know what that meant — I had to look that up in a couple different places.”
The terms participants highlighted as confusing included:
platforms (3)
- “I’m not sure what a platform is, unless you can build it with steel and wood.”
cultural change (2)
- What is “cultural change”?
- “What are you getting at here?”
modern (2)
- “Modern...that’s unclear to me”
legacy modernization
- “I don’t know what legacy modernization means”
IT spend (2)
- “I’ve never heard this term.”
often guarantee that their projects are delayed, over-budget, and/or unable to meet their customers’ needs
- “I had to read that, like, ten times.”
API (came up a couple times in both kinds of testing)
- “I didn’t know what API was, and when I clicked through to learn more, it didn’t explain that either.”
digitally-powered, digital service, digital service techniques (2)
These terms were also each highlighted as confusing by at least one person:
Delivery is the strategy,What we delivercustom productsstandard government procedurescloud hostingbringing your data and services into the 21st centurypartner agencies
One participant, who had visited the 18F website before, said the current site is “doing a better job of explaining in simpler terms what 18F does.”
In response to the paragraph that began ”while the federal IT spend is approximately $90 billion...,” one participant said:
This catches my attention! This is nice. Put it up front — how will 18F help deliver that value? I want to know that.
One participant talked about initially seeing 18F as “dot-com whiz kids,” but said that “reading about where people came from, and their backgrounds, I realized that people came from a wider range.”
One participant really like seeing examples of agencies and products we deliver (on “What we deliver” page):
You can click on those examples — there’s definitely a clickable thread: what are the products, who did you make them for, what are the products you delivered?
One participant expressed concern about their agency’s budget process: it can be hard to plan for rapidly changing technology when budgeting is always 5 years behind.
Two participants asked about how to “nominate” or “suggest” projects that they wanted to see happen, but didn’t have funding for.
People are interested in acquisitions:
The acquisition part is very interesting, because that’s something that most government agencies are struggling with.
Anything dealing with Federal Acquisition Regulations, we’d be more likely to click on that.
Miscellaneous quotes:
Within the armed forces, we have “joint duty assignments”, where we send people off to work in another agency for a while...Is there a program to let folks in other agencies go work in 18F, learn how you work, then return to their agencies with that knowledge?
One thing that I didn’t see that would be interesting — I don’t know if you guys go to the technology expos, where companies come in and show products...18F should come to those and talk with our senior folks.
Already implemented: (after these research sessions, but before this report) We changed the featured project and included key words about our work on the project (“data,” “easier to use”). This feature also shifts the main verb from “building” to “helping” in order to foreground how we partner with agencies to support their missions, which also adresses feedback from last sprint.
Partially implemented: Our next round of content testing will focus on whether these new paragraphs are doing better:
18F partners with agencies to approach technology projects in new ways. We help agencies improve processes and deliver efficient, easy-to-use digital services to the public.
As an office within the General Services Administration, we know how to work with government. All our projects support agencies in transforming how they deliver digital services and technology products.
Remaining: We should also try new language under the “What we deliver” header.
Begun: Our first version of this will use existing blog content but add an introduction to help potential partners understand the problems we were trying to solve and the kinds of work we have done on the project.