Question Format: Checkbox + Sentiment Chips

[SachaG]

We've discussed the Radio Button + Sentiment Chips question format extensively (#183) but I don't think we've had a thread to discuss the Checkbox + Sentiment Chips format yet.

Prototype

https://lea.verou.me/files/stateof/mocks/sentiment/

("Which of the following elements have you used?")

Status

Not implemented as a functioning question template yet

Potential Issues

I think that even more than with the Radio + Sentiment UI, this is a case where adding sentiment chips drastically increases the cognitive load required to get through the question. We're essentially asking them to consider three prompts at once:

• Do you use X (the original question)
• Do you want to use X again (sentiment chip 1)
• Do you not want to use X again (sentiment chip 2)

Which becomes quite mentally taxing when multiplied by the number of items in the list.

[LeaVerou]

I think it's essential to start from the use cases for this template:

• Features that have been around for very long, so we don't really care about awareness, only usage + sentiment
• Features that have not been implemented yet, so we don't really care about usage, only awareness + sentiment
• Features that are multi-faceted, and it's tells us little to ask about them as a whole (e.g. Intl API, web app manifest fields, resource hints, form validation)

I think that even more than with the Radio + Sentiment UI, this is a case where adding sentiment chips drastically increases the cognitive load required to get through the question. We're essentially asking them to consider three prompts at once:

You are comparing a checkbox question to a feature question. That is not a valid comparison; a checkbox question replaces several feature questions. Do you think a checkbox question with 5 options, each with sentiment, is more taxing than 5 separate feature questions, each with sentiment?

[SachaG]

That's very true, and if we think of these "sentiment checkbox" questions as closer to regular feature questions in terms of mental cost it becomes clear that there's way too many of them and/or that they have too many options.

Actually, I think the closest equivalent in terms of traditional form controls would be a Likert scale question. But it's not quite a 100% match.

[LeaVerou]

That's very true, and if we think of these "sentiment checkbox" questions as closer to regular feature questions in terms of mental cost it becomes clear that there's way too many of them and/or that they have too many options.

Mental load is not a function of the data being asked — it depends very much on the UI too, that's the whole point of these changes. So, it does not follow that because these would be very taxing if expanded to full feature questions, they are equally taxing as checkbox questions and thus need to be cut. It's hard to come up with a precise factor, especially in the absence of data, but my gut feeling is that a checkbox question with N options is about as taxing as N/3 feature questions.

[SachaG]

I agree, I meant that e.g. if a checkbox takes N energy, then a checkbox with sentiment takes 2N and a main feature question with sentiment chips takes 3N. So if we're ok with a regular checkbox question having 10 items, then we should limit the checkbox+sentiment questions to e.g. 5 items.