Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

System.Text.Json usability study results #117

Open
terrajobst opened this issue Jul 8, 2019 · 0 comments

Comments

@terrajobst
Copy link
Member

commented Jul 8, 2019

We've performed a usability study on the new System.Text.Json APIs which will replace JSON.NET as the default JSON experience in .NET Core 3.0:

  • We focused on the primary "getting started" scenarios to validate that we picked the right type- and method names.
  • We simulated what we believe to be common error cases to validate that our exception messages are understandable and actionable.
  • Since we were short on time, we couldn't use the recruiting provided by the UX lab but instead brought in our own developers, which we recruited from the .NET engineering team

This has worked out well for us as we were able to get short term feedback which resulted in significant improvements to the overall API shape and usability. In the future, we should do more API focused UX studies 😊

Tasks

# Description Code
1 Serialize and pretty print Link
2 Deserialize with trailing comma Link
3 Deserialize with custom property names Link
4 Read with DOM Link
5 Pretty print an existing JSON file Link

Hypotheses

We expect developers to...

# Description Result
1 ...assume the APIs are similar/identical to JSON.NET Validated
2 ...complete tasks 1-4 successfully Validated
3 ...struggle/unable to complete the task 5, because it requires working with two different components which is very different from JSON.NET Invalidated
4 ...not look at documentation for most of their coding as IntelliSense is good enough Invalidated
5 ...be happy with the experience of using the API Validated
6 ...plan on using System.Text.Json for their next JSON task Partially Validated

Key Observations

  • Everyone completed scenarios #1-#3, 2 didn't complete #4, 6 didn't complete #5
    • The reason was always "out of time". Every person who started a scenario successfully finished it.
    • We feel this validates that we picked the right scope for a 60 minute study of people from various backgrounds
  • Overall people liked the new API
    • Key value prop for most people was the fact that "it's now built in"
    • Very few mentioned perf and the ones that did thought that it probably wouldn't make a difference in their scenarios.
    • Most people said they would prefer System.Text.Json for new APIs, but would stick to JSON.NET in code that already uses it.
    • [Action] None, this is in line with what we expected.
  • Virtually everyone tried to google for sample code in the documentation
    • Everyone found our blog post and online documentation
    • People new to .NET were confused that JSON.NET and System.Text.Json weren't the same thing
    • Everyone complained that the current docs are lacking, especially sample code.
    • [Action] Docs are work in progress. We plan on shipping with a fully documented API surface and sample code.
  • Virtually everyone found the key type relatively quickly
    • Some people were confused when types were static, rather than something that one can new up
    • [Action] None. This is by-design as we try to reduce the number of allocations.
  • Virtually everyone had trouble selecting the correct method on
    JsonSerializer
    • We chose a very complicated naming convention
    • [Action] We simplified it by using names that people in the study told us were looking for.
  • Most of our error messages were well received because they told people what
    the problem is & how they can fix them.
    • One error message was unclear because we reported an unexpected character when we could have reported that we didn't expect a comment.
    • [Action] We've reworded the error message.
    • We had one error message that was generated by a lower layer and provided an action item that doesn't make sense when people receive it when using the higher layer component.
    • [Action] We've reworded the error message to be more generic so it makes sense for both kind of consumers.
  • Virtually everyone used the new Roslyn completion feature which will also show
    types from namespaces not imported yet.
    • However, virtually nobody realized that they were doing it.
    • This caused confusion when people found other APIs that happen to include Json in them, specifically the old data contract serialization.
    • [Action] Talk to the Roslyn IDE team to see if this something more people will run into. This could prove to be especially problematic for technologies that share type names (e.g. UserControl in WinForms and WPF) and both are referenced by the project.
  • Optional parameters and async/await are usability cliffs
    • Many were confused by optional parameters and tried to pass them in, even though they didn't have to.
    • [Action] Talk to the Roslyn IDE team to see if we can visualize them better in VS.
    • [Action] Try to avoid heavy use of optional parameters for mainline APIs.
    • Async is a very steep cliff. People have no idea what to do in their code to make async work. While IntelliSense shows the usage by prefixing it with await, people didn't notice that.
    • [Action] Unclear; this is a broader issue.
  • Virtually nobody used F1 to get help for APIs
    • Surprise. I expected more people to use F1.
  • Virtually everyone used F12 to explore the API ("metadata view")
    • Surprise. I expected most people to explore the type via the
      IntelliSense completion
  • Nobody used Object Browser to explorer the APIs
    • Surprise. I expected at least some people to.

Discussion

For a discussion, please go to dotnet/corefx#39304.

@dotnet dotnet locked and limited conversation to collaborators Jul 8, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
1 participant
You can’t perform that action at this time.