Meta: Asking about developers' experience with each CSS Feature

[LeaVerou]

There is currently a lot of discussion about this scattered in a variety of places, and I wanted to have a specific issue about this.

The problem is that the current survey tracks primarily awareness (Haven't heard about it, Heard about it, Used it), but it does not track what developers think about each CSS feature.

• "Heard about it" could mean that they just haven't had a chance to use it, or they have no use cases, or they don't see the point because browser support is so bad, or the feature seems confusing and hard to learn, or several other things.
• "Used it" could mean they used it and loved it and plan to use it in every project now, or they found it very confusing and plan to only use it if absolutely necessary, or they loved it but don't plan to use it again for a while due to browser support etc etc.

There are several thoughts about capturing some or all of that.

Option 1: Ask a followup question for every "Heard about it" and "Used it" answers.

There is a lot more brainstorming about this in Devographics/Monorepo#99 but I believe the latest mockup looks like this:

Pros: Captures pretty much all of the nuance described above
Cons: Increases cognitive overhead for each feature question quite significantly.

Option 2: "Would use it again" / "Would not use it again" distinction

There are two ways to implement this:

• a) Either break the "I have used it" answer into two: "I have used it, would use again" and "I have used it, would not use again"
• b) Or have a popup similar to the above, but with a simple radio: "Would you use it again? 🔘 Yes 🔘 No". Note that we cannot use a checkbox here, as the default will affect the results, both Yes and No need to be a deliberate choice.

Pros: Much lower cognitive overhead than Option 1, especially with Option 2a.
Cons:

• The obvious con is that we get much less information, and none about the cases where people have only heard about a feature (but may still have opinions!). The idea being there can always be followup surveys about specific features that score poorly.
• Option 2a breaks historical comparison. We could still group them together for that, but the mere act of having two answers instead of one affects what respondents do. While this is a downside, it's not necessarily a dealbreaker.
• Option 2b is still in essence a separate question, so there's still some of the cognitive overhead of that, while getting a lot less info.

Option 3

Emoji rating widget

Other ideas

Perhaps the best way forwards is a combination of 1 and 2. E.g.: a followup popup for "Would you use it [again]?" ("again" only for "I have used it" answers) with an "Add details" UI.

[SachaG]

I may have mentioned this somewhere, but my initial rationale for not having the same "would use again/would not use again" distinction for features as exists for tools is that using a feature or not seems a lot less subjective. Usually the main factor is simply whether you need the feature or not. To pick an example from the distant past, I hated using float for layout but for years there was no alternative. So I would've picked "would use again" even though the experience was horrible.

So for that reason I'm partial to Option 1, and it also lets us be consistent and reuse the same "follow-up" pattern elsewhere as well.

(cc @stubbornella)

[SachaG]

• Add follow-up question widget

[LeaVerou]

Here’s another idea: instead of a popup with a whole followup question, an inline emoji rating widget about their experience.
This would only appear when they select the relevant option (2nd or 3rd choice).

Something like this perhaps:

Or even making it part of the answer:

Tooltips could also provide more context (perhaps allowing us to do away with the "…and?" label). E.g. "I hated it", "Meh", "I loved it".
A similar widget would appear if the participant selects the second option.

possibly with a discreet "💬" or "+" button to add more context for those really invested in the survey?

[stubbornella]

I’m also interested in sentiment from folks who have heard of a feature but not yet tried it. Waiting to understand sentiment only from those that have tried a feature could be a very lagging signal that our education efforts have been incomplete.

[LeaVerou]

I’m also interested in sentiment from folks who have heard of a feature but not yet tried it. Waiting to understand sentiment only from those that have tried a feature could be a very lagging signal that our education efforts have been incomplete.

Absolutely. Whatever UI we show, it (or a suitable variation of it) should be shown for both Option 2 and Option 3. IIRC this was one of our disagreements with @SachaG, I think he believed it should only be shown when Option 3 is selected.

[SachaG]

Coming back at this a couple weeks later, I'm thinking we should make it much simpler and just have an optional freeform textfield to go along each feature/tool question, with the prompt being something like "tell us more about your choice".

The textfield would appear when you click a little speech bubble icon, which would always be visible no matter which option you select (or even if you don't select anything).

The resulting data won't be that "clean" since it'll be freeform text, but we can still get some valuable insight through things like word clouds, or even just counting the number of freeform comments each feature gets as a proxy for interest. And I think a lot of our features might be niche enough that even just reading through the raw comments might still be very manageable for motivated people :)

[LeaVerou]

I'm a bit worried that even if optional, some people feel like they need to complete every field, which would make the survey extremely fatiguing. A little user testing can answer whether this is a valid concern.
I assume this would only show for options 2 & 3, right?