Demographics: Using numbers instead of brackets?

[LeaVerou]

I posted this as a blog post, but I’m also including a modified version here for discussion

This spawned out of the discussion about changing "Salary" to "Income" as @ShaineRosewel proposed using a number input instead of brackets for salary.

There are actually four demographics questions in State of X surveys where the answer is essentially a number, yet we ask respondents to select a bracket:
age, years of experience, company size, and income.

The arguments for brackets are:

1. They are more privacy preserving for sensitive questions (e.g. people may feel more comfortable sharing an income bracket than their actual income)
2. They are more efficient to input (one click vs homing to keyboard and hitting several keystrokes).
3. In some cases respondents may not know the precise number offhand (e.g. company size)

The arguments for numerical input are:

1. Depending on the specifics, these can actually be faster to answer overall since they involve lower cognitive overhead (for known numbers).
2. The brackets are applied at the analysis stage, so they can be designed to provide a better overview of the dataset
3. More elaborate statistics can be computed (e.g. averages, medians, stdevs, the sky is the limit)

Which one is faster?

We can actually calculate this!
Average reading speed for non-fiction is around 240 wpm (= 250ms/word) ¹
Therefore, we can approximate reading time for each question by multiplying number of brackets × average words per bracket (wpb) × 250ms.

However, this assumes the respondent reads all brackets from top to bottom, but this is a rare worst case scenario.
Usually they stop reading once they find the bracket that matches their answer, and they may even skip some brackets, performing a sort of manual binary search. We should probably halve these times to get a more realistic estimate.

Average typing speed is 200 cpm ² (≈ 300ms/character). This means we can approximate typing time for each question by multiplying the number of digits on average × 300ms.

Let’s see how this works out for each question:

Question	Brackets	WPB	Reading time	Avg Digits	Typing time
Age	8	4	4s	2	0.6s
Years of Experience	6	4	3s	2	0.6s
Company size	9	4	4.5s	3	0.9s
Income	7	2	1.75s	5	1.5s

As you can see, despite our initial intuition that brackets are faster, the time it takes to read each bracketed question vastly outweighs typing time for all questions!

Of course, this is a simplification. There are models in HCI, such as KLM that can more accurately estimate the time it takes for certain UI flows.

For example, here are some of the variables we left out in our analysis above:

• When answering with numerical input, most users need to home from mouse to keyboard, which takes time (estimated as 0.4s in KLM)
  and then focus the input so they can write in it, which takes an additional click (estimated as 0.2s in KLM)
• When answering with brackets, users need to move the mouse to the correct bracket, which takes time (KLM estimates all pointing tasks as a flat 1.1s, but this can be more accurately estimated using Fitts’ Law)
• We are assuming that the decision is instantaneous, but doing the mental math of comparing the number in your head to the bracket numbers also takes time.

However, given the vast difference in times, I don't think a more accurate model would change the conclusion much.

Note
Note that this analysis is based on a desktop interface, primarily because it’s easier (most of these models were developed before mobile was widespread, e.g. KLM was invented in 1978!) Mobile would require a separate calculation taking into account the specifics of mobile interaction (e.g. the time it takes for the keyboard to pop up), though the same logic applies.

What about sliders?

Sliders are uncommon in surveys, and for good reason.
They offer the most benefit in UIs where changes to the value provide feedback, and allow users to iteratively approach the desired value by reacting to this feedback.
For example:

• In a color picker, the user can zero in to the desired coordinates iteratively, by seeing the color change in real time
• In a video player, the user can drag the slider to the right time by getting feedback about video frames.
• In searches (e.g. for flights), dragging the slider updates the results in real time, allowing the user to gradually refine their search with suitable tradeoffs

In surveys, there is usually no feedback, which eliminates this core benefit.

When the number is known in advance, sliders are usually a poor choice, except when we have very few numbers to choose among (e.g. a 1-5 rating)
and the slider UI makes it very clear where to click to select each of them, or we don't much care about the number we select (e.g. search flights by departure time).³
None of our demographics questions falls in this category (unless bracketed, in which case why not use regular brackets?).

There are several reasons for this:

• It is hard to predict where exactly to click to select the desired number. The denser the range, the harder it is.
• Even if you know where to click, it’s hard to do so on mobile
• Dragging a slider on desktop is generally slower than typing the number outright.⁴

<input type=number> all the things?

Efficiency is not the only consideration here.
Privacy is a big one. These surveys are anonoymous, but respondents are still often concerned about entering data they consider sensitive.
Also, for the efficiency argument to hold true, the numerical answer needs to be top of mind, which is not always the case.

I summarize my recommendations below.

Age

This is a two digit number, that is always top of mind. Number input.

Years of experience

This is a 1-2 digit number, and it is either top of mind, or very close to it. Number input.

Company size

While most people know their rough company size, they very rarely would be able to provide an exact number without searching.
This is a good candidate for brackets.
However, the number of brackets should be reduced from the current 9 (does the difference between 2-5 and 6-10 employees really matter?),
and their labels should be copyedited for scannability.

We should also take existing data into account.
Looking at the State of CSS 2022 results for this question,
it appears that about one third of respondents work at companies with 2-100 people,
so we should probably not combine these 5 brackets into one, like I was planning to propose.
101 to 1000 employees is also the existing bracket with the most responses (15.1%), so we could narrow it a little,
shifting some of its respondents to the previous bracket.

Taking all these factors into consideration,
I proposed the following brackets:

• Just me!
• Small (2 - 50)
• Medium (51 - 200)
• Large (201 - 1000)
• Very Large (1000+)

Income

The question tht started it all is unfortunately the hardest.

Income is a number that people know (or can approximate).
It is faster to type, but only marginally (1.75s vs 1.5s).
We can however reduce the keystrokes further (from 1.5s to 0.6s on average) by asking people to enter thousands.

The biggest concern here is privacy.
Would people be comfortable sharing a more precise number?
We could mitigate this somewhat by explicitly instructing respondents to round it further, e.g. to the nearest multiple of 10:

What is your approximate yearly income (before taxes)? Feel free to round to the nearest multiple of 10 if you are not comfortable sharing an exact number. If it varies by year, please enter an average.
$⬜️⬜️⬜️ ,000

However, this assumes that the privacy issues are about granularity, or about the number being too low (and rounding to 10s could help with both).
However, David Karger made an excellent point in the comments,
that people at the higher income brackets may also be reluctant to share their income:

I don't think that rounding off accomplishes anything. It's not the least significant digit that people care about, but the most significant digit. This all depends on who they imagine will read the data of course. But consider some techy earnings, say 350k. That's quite a generous salary and some people might be embarrassed to reveal their good fortune. Rounding it to 300k would still be embarrassing. On the other hand, a bracket like 150 to 500 would give them wiggle room to say that they're earning a decent salary without revealing that they're earning a fantastic one. I don't have immediate insight into what brackets should be chosen to give people the cover they need to be truthful, but I think they will work better for this question.

Another idea was to offer UI that lets users indicate that the number they have entered is actually an upper or lower bound.

Of course, a dropdown PLUS a number input is much slower than using brackets, but if only a tiny fraction of respondents uses it, it does not affect the analysis of the average case.

However, after careful consideration and input, both qualitative and quantitative, it appears that privacy is a much bigger factor than I had previously realized.
Even though I was aware that people see income level as sensitive data (more so in certain cultures than others),
I had not fully realized the extent of this.
In the end, I think the additional privacy afforded by brackets far outweighs any argument for efficiency or data analysis convenience.

---

@SachaG what do you think?

Footnotes

1. https://www.sciencedirect.com/science/article/abs/pii/S0749596X19300786 ↩

2. https://www.typingpal.com/en/blog/good-typing-speed ↩

3. Slider Design: Rules of Thumb, NNGroup, 2015 ↩

4. KLM is a poor model for dragging tasks for two reasons:
   First, it regards dragging as simply a combination of three actions: button press, mouse move, button release.
   But we all know from experience that dragging is much harder than simply pointing, as managing two tasks simultaneously (holding down the mouse button and moving the pointer) is almost always harder than doing them sequentially.
   Second, it assumes that all pointing tasks have a fixed cost (1.1s), which may be acceptable for actual pointing tasks, but the inaccuracy is magnified for dragging tasks.
   A lot of HCI literature (and even NNGroup) refers to the Steering Law to estimate the time it takes to use a slider,
   however modern sliders (and scrollbars) do not require steering, as they are not constrained to a single axis:
   once dragging is initiated, moving the pointer in any direction adjusts the slider, until the mouse button is released.
   Fitts Law actually appears to be a better model here, and indeed there are many papers extending it to dragging.
   However, evaluating this research is out of scope for this post. ↩

[SachaG]

Great overview! I think this makes a lot of sense, and I especially like the idea of limiting income to thousands. The only thing is that if we reduce the number of "Company Size" options we also lose some granularity, but since we're gaining granularity for the other three questions maybe that's fine.

Would people be comfortable sharing a more precise number?

We could also use a slider and have the slider increase in $5,000 or $10,000 increments?

[LeaVerou]

The only thing is that if we reduce the number of "Company Size" options we also lose some granularity, but since we're gaining granularity

What additional insight does this granularity give us about developers? What is different between a company with 2-5 people and one with 6-10?

We could also use a slider and have the slider increase in $5,000 or $10,000 increments?

I just updated the post with both some discussion on sliders, as well as more work on the income input.

[SachaG]

More granularity opens up more ways to present the data. For example I tried doing boxplots for our current data but because it's not very granular, e.g. the 50th percentile and 75th percentile end up overlapping sometimes and it kinda breaks the chart. But it's a detail, and anyway we don't need to apply the same data visualizations to every question.

[Arkkimaagi]

What is the wanted/needed precision? I think the survey should not ask personal questions on a higher specificity level than absolutely needed.

With brackets I feel like there is some anonymity in the vagueness of the brackets and I'm a little more relaxed in answering. I think it's also faster to answer brackets as I do not need to consider the proper precision to answer in a semi-anonymous way.

Also if I'm asked a bracketed question, it's way faster for me to answer as I probably know the ballpark easily. For example, in Finland we do not talk about our salaries on a yearly level in USD. We generally talk about our pay on a monthly level and in EUR. So to answer a survey like this I already need to do some math in my head. Brackets help me know the useful precision and may speed up the math. So, no, I do not have a number for my yearly salary on the top of my mind.

Another thing to consider about the input type number is the accessibility side of it. Creating truly accessible forms is surprisingly difficult and input number might be problematic. Here are some pointers: https://technology.blog.gov.uk/2020/02/24/why-the-gov-uk-design-system-team-changed-the-input-type-for-numbers/

I might stop answering the survey if I'm asked too precise private question as it feels more insecure and targeted. Why does the survey maker need my exact salary? The precision requirement reasoning (or lax requirements) would need to be written to the survey in a convincing way. This text would further slow down filling the form.

And even if the intentions asking precise private questions are pure, leaks happen, so criminals might their hands on personal information due to a breach or such.

Regarding form filling speed, I feel that manual typing/clicking is not the part we should optimise solely. Instead privacy, mental burden and ease of answering should also be considered in addition to mechanical speed.

Thank you for discussing these things in the public! I was motivated to try to help share my thoughts on the matter.

[joyously]

I second all of these points, especially the concern about speed of input.
In addition, people who have multiple streams of income don't have the numbers in mind.
Also, those of us who have retired don't have years of experience top of mind. In past surveys, even though I answered "no employer", the questions about company size and responsibilities are still there to be answered.

[SachaG]

I do think that regarding the privacy angle, brackets feel less intrusive to me. There's a risk that if we use number inputs, we get fewer responses to the question just because they scare people off.

[LeaVerou]

Question to all three: What do you think about the second idea, with the dropdown? In theory that is even more private than brackets if you just give an upper or lower bound.

@Arkkimaagi: I thought in Scandinavia salaries are public data even published somewhere, is that not the case for Finland?

@SachaG

There's a risk that if we use number inputs, we get fewer responses to the question just because they scare people off.

We may be able to test this out during the user testing phase.

[joyously]

What do you think about the second idea, with the dropdown?

I don't see any advantage worth the added complexity.

[Arkkimaagi]

The dropdown example seems a bit complex to understand and to fill. Also, I do not understand what benefit do the "above" and "below"-options have. Feels like this would lead to four kinds of data that could be harder to combine than is resonable.

Personally I'd avoid a dropdown with so few options. Separate question with radiobuttons would probably be better. With radiobuttons, the options could be explained furher if needed.

Why not just ask for the data with the precision people are comfortable entering? I mean, if the previous data is bracketed, having freeform data and deriving brackets from it after the survey could work.

Anyways, when designing forms and surveys we should only ask the information that is absolutely needed on a precision level that is appropriate.

While the information is public in Finland, it has not been made easily available online. Here is an explanation from our IRS(?): https://www.vero.fi/en/About-us/finnish-tax-administration/data-security-and-information-access/public-information-on-taxes/public-information-on-individual-income-taxes/

The public information on individual income taxes falls under the category of personal data by definition of law. Present law lays down a number of restrictions on how personal data can be released, and this means that it is not possible to put up the public income-tax data on the web. Accordingly, we only provide electronic access to the information at tax offices, not over the Internet.

Public information can be requested by telephone and browsed at tax offices

So in practise basically the top earners and some public figures have their data requested by the media, but others have some level of anonymity. Talking about the exact salary openly is not too common in my experience.

[riidom]

About the company size:

For very small companies, it is a difference whether you ask for employees or the total count. "Just me!" in your proposal probably refers to a freelancer? I'd suggest to start the next bracket with 1 and not with 2. From my experience it is not overly uncommon to be the only employee, but have 1 to 3 bosses, for example.

Looking forward to the survey!:)

[LeaVerou]

That’s why I also proposed a rewording :) #178

[SachaG]

Quick informal poll: https://twitter.com/SachaGreif/status/1688447006459805696

[LeaVerou]

Yeah, I was already reaching the conclusion that brackets is preferable here in the last couple of days. It’s not about efficiency of input; privacy seems to be a much bigger factor than I had previously realized (do note this is also culturally dependent).

[LeaVerou]

I also updated the post to reflect my current view on this.

[SachaG]

Since the concensus is that we're keeping brackets, should we try and make the brackets more granular? For reference, the current ones are (in thousands of USD):

• 0 - 10k
• 10 - 30k
• 30 - 50k
• 50 - 100k
• 100 - 200k
• 200k+

Maybe we could do this instead?

• 0 - 20k
• 20 - 40k
• 40 - 60k
• 60 - 80k
• 80 - 100k
• 100 - 150k
• 150 - 200k
• 200k+

The reason for the change would be that in my early attempts to generate box plot charts off the ranges (by using the average of each range) I got quite a few charts where the various percentiles overlap, which I assume is because there are too many respondents concentrated in too few datapoints?

[SachaG]

Other UI idea I just had to adapt to the user's privacy comfort level:

[SachaG]

Or maybe even simpler, something like:

[LeaVerou]

I quite like the last idea! We can deploy it and see how user study participants feel about it — though there’s a bit of a self selection there (since these are people who know they will fill out the survey with a researcher watching, they may also not belong to the group of people who feels this information is very private).