Randomize order of response options and matrix rows

[soerendamsbo]

Description

Hi! I'm a big believer in the surveydown project, and I'm currently spending time testing the possibilities and limitations.

I would like to request a feature, which is common in my field (political science): order randomization (with/without recording the order).

To mitigate order/anchoring effects, it's in some cases standard practice in the social sciences to randomize the order of (discrete) responses options as well as questions (especially in compressed matrix style questions). As an example, we might consider the following question:

Which fruit do you prefer most from this list? Options: Apple, Banana, Pear, Strawberry.

First, the question code could ideally include something like (a) "randomize = TRUE" or, perhaps, (b) "randomize = X", where X is one of "no", "silent", "record" (determining whether the order is recorded in the data), or possibly (c) where X is a integer vector saying which options to randomize and which to hold fixed, e.g., "randomize = 2:4". This feature should be there for all "mc" type questions.

Secondly, it also would be great to extend the same feature to (especially) matrix type questions, to enable randomization of the order of (sub)questions (matrix rows).

FWIW I've experimented with implementing list experiments in surveydown (presenting a list of items and asking respondents to select a number of these, e.g., 'five'). Randomizing the order of the list items is straightforward to implement with javascript, which makes me think that implementing the feature is highly feasible.

[jhelvy]

You read my mind! I posted #248 just last week. And I do believe javascript is the solution here. I was thinking of supporting it at the yaml level though, so you could have some yaml keys like we have with other survey settings to determine if you want all questions randomized (e.g. randomize_options: true), or select ones, something like:

randomize_options: 
  question1
  question7

This imo would make it much easier to quickly see and control which questions are set to be randomized and which are not, otherwise you'd have to search through each call to sd_question(), which is tedious.

However, that doesn't provide the level of control that you're suggesting, like randomize = 2:4. I think that level of control makes sense as an argument to sd_question(), for one because I don't otherwise know how to set that as a yaml key, and also because if you're using indexing to refer to the question options, then it would make sense that you'd want to set those numbers very close to where you are defining the options, otherwise you might choose the wrong indexes.

It could be set up to support both though. You could have yaml keys for the general case of "all or nothing" randomization, and then a randomize argument in sd_question() that overrides any yaml setting, where you could provide greater control over what exactly is randomized.

What's the reason for not wanting to record the randomized order? Like, if it's set to just always record it, then you can later as the analyst choose to use that piece of data or not. Not recording it at all though means you don't have the choice. I'm not 100% sure how best to record it either. I guess we could have some auto-created data entries that are appended to the end of the data to capture the order, like if the question_id is foo, you could have foo_order created automatically and it would store the order shown like "2,3,1". That's how we handle time stamps - they're all auto-generated.

I don't think implementation will be that difficult. I think it's just a matter of setting things up so that there are little items in the html that trigger a javascript action to randomize the option set. Is that how you did it in your experiment?

[soerendamsbo]

That's great news, John! To me the yaml level setting is a nice addition, not a need-to-have. But I support the idea of being able to set more granular and default-overriding behavior at the individual question in addition to the general setting of randomization defaults. I think that's the way to go. Do you think that implementing it for mc options and matrix rows would be similar (and similarly feasible)? That would be amazing. You are right, it would be fine to just always record the order in the data. For instance in a sample comma-separated text field ("Apple, Banana, Orange" or "1, 4, 3"). Like you're suggesting. That would be the simplest solution that allows anyone to parse it how they want, when they need it. My only concern is that recording everything could, in extreme cases, be a lot of extra information written to the database – and I am struggling a bit with low DB read/write speeds with Supabase (I'm experimenting with Supabase Pro, but *sd_store_value()* still significantly slows everything down. If you have thoughts on this, I would be extremely grateful to hear them – I was considering opening a separate performance discussion on this question). In the list experiment, I tried implementing, I just created a named unordered list ('myList') in html and had chatgpt write a short script that immediately reorders the list items: <script> functionrandomizeList() { constlist = document.getElementById('myList'); constitems = Array.from(list.getElementsByTagName('li')); // Shuffle the items using Fisher-Yates shuffle algorithm for (leti = items.length - 1; i > 0; i--) { constj = Math.floor(Math.random() * (i + 1)); [items[i], items[j]] = [items[j], items[i]]; // Swap } // Clear the list and append the shuffled items list.innerHTML = ''; // Clear existing list items items.forEach(item=>list.appendChild(item)); // Append items in new order } // Randomize the list immediately randomizeList(); // Call the randomize function immediately</script>

…

On Monday, January 5th, 2026 at 12:14, John Helveston ***@***.***> wrote: You read my mind! I posted [#248](#248) just last week. And I do believe javascript is the solution here. I was thinking of supporting it at the yaml level though, so you could have some yaml keys like we have with other [survey settings](https://surveydown.org/docs/survey-settings#default-settings) to determine if you want all questions randomized (e.g. randomize_options = true), or select ones, something like: randomize_options : question1 question7 This imo would make it much easier to quickly see and control which questions are set to be randomized and which are not, otherwise you'd have to search through each call to sd_question(), which is tedious. However, that doesn't provide the level of control that you're suggesting, like randomize = 2:4. I think that level of control makes sense as an argument to sd_question(), for one because I don't otherwise know how to set that as a yaml key, and also because if you're using indexing to refer to the question options, then it would make sense that you'd want to set those numbers very close to where you are defining the options, otherwise you might choose the wrong indexes. It could be set up to support both though. You could have yaml keys for the general case of "all or nothing" randomization, and then a randomize argument in sd_question() that overrides any yaml setting, where you could provide greater control over what exactly is randomized. What's the reason for not wanting to record the randomized order? Like, if it's set to just always record it, then you can later as the analyst choose to use that piece of data or not. Not recording it at all though means you don't have the choice. I'm not 100% sure how best to record it either. I guess we could have some auto-created data entries that are appended to the end of the data to capture the order, like if the question_id is foo, you could have foo_order created automatically and it would store the order shown like "2,3,1". That's how we handle time stamps - they're all auto-generated. I don't think implementation will be that difficult. I think it's just a matter of setting things up so that there are little items in the html that trigger a javascript action to randomize the option set. Is that how you did it in your experiment? — Reply to this email directly, [view it on GitHub](#250 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AIXW4FARLZPVYPPNAZZJAMT4FJBRVAVCNFSM6AAAAACQWTJIJ6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTKNBRGM3DAOI). You are receiving this because you authored the thread.Message ID: ***@***.***>

[jhelvy]

I'd probably want to implement it such that any question with options can be randomized, so both mc and matrix (among others) would work. With matrix though, there are two possible levels of randomization: the rows and the columns. I'm assuming people would mostly want only the rows to be randomized, because the columns are typically used for things that often have a natural order to them, like likert scales. So for now I'll probably only randomize the rows and leave the columns alone, unless you have any thoughts on that?

As for recording the order, I agree that it might get to be a little too much if we're recording the order always. In fact, I kind of feel like the time stamp recording we currently do is already too much. We record the time stamps for starting each page and each question, which can double the number of columns in the data. I'm already thinking of adding another set of yaml settings like record-time-questions: false and record-time-pages: false to allow that to be turned off is desired. I'll still always record the start and end times of the entire survey though. So we could just another record-random-order: false if we want to turn that off too. I'll add another issue to keep that item on the to-do list as well.

Your example is exactly what I was thinking. Once you have a list in html, you can use javascript to randomize it, so it shouldn't be a big challenge to implement this. The most tedious bits are adding support for the different levels of control, like both yaml settings and sd_question() support, etc.

[soerendamsbo]

I agree with you on everything here, John. To me, it is most important to implement the question-level randomization option. As a first step, I think it is fine to (a) always record the order and (b) only randomize row order in matrices (imo, it is typically bad survey design – too hard on the user – to use matrix type questions with anything but a simple, familiar ordinal scale in the columns, such as Likert). My personal prioritized ranking of useful additions (just fyi) – now or down the line – would be: - Ability to define, at question level, which items are randomized and which are fixed using indices (e.g., to always have things like "Don't know" come last) - Ability to control recording of order at question level - Ability to control recording of order at yaml level (overridden by question-lvl settings) - Ability to control order randomization at yaml level (overridden by question-lvl settings) - (Not for me, but possibly others) Ability to also/instead randomize column order in matrix questions I think it's a great idea to always record start and end time and let everything else be optional. Ideally with yaml-lvl and q-lvl settings, but yaml-lvl disabling of q-timing would be a nice, simpler improvement. Thank you so much for all your great work.

…

On Tuesday, January 6th, 2026 at 17:01, John Helveston ***@***.***> wrote: I'd probably want to implement it such that any question with options can be randomized, so both mc and matrix (among others) would work. With matrix though, there are two possible levels of randomization: the rows and the columns. I'm assuming people would mostly want only the rows to be randomized, because the columns are typically used for things that often have a natural order to them, like likert scales. So for now I'll probably only randomize the rows and leave the columns alone, unless you have any thoughts on that? As for recording the order, I agree that it might get to be a little too much if we're recording the order always. In fact, I kind of feel like the time stamp recording we currently do is already too much. We record the time stamps for starting each page and each question, which can double the number of columns in the data. I'm already thinking of adding another set of yaml settings like record-time-questions: false and record-time-pages: false to allow that to be turned off is desired. I'll still always record the start and end times of the entire survey though. So we could just another record-random-order: false if we want to turn that off too. I'll add another issue to keep that item on the to-do list as well. Your example is exactly what I was thinking. Once you have a list in html, you can use javascript to randomize it, so it shouldn't be a big challenge to implement this. The most tedious bits are adding support for the different levels of control, like both yaml settings and sd_question() support, etc. — Reply to this email directly, [view it on GitHub](#250 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AIXW4FFREXDZRE2KC7YARRT4FPL37AVCNFSM6AAAAACQWTJIJ6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTKNBSGU4TKNI). You are receiving this because you authored the thread.Message ID: ***@***.***>

[jhelvy]

This is now supported as of v1.1.0. We have this documented as well here, and a new template showcasing this feature here.

[soerendamsbo]

Fantastic! I can confirm that it works like a charm. As far as I can tell, the order is not recorded in v1.1.0? Nevertheless, this is a really elegant solution that solves 95% of common use cases, imo. Thanks a lot for your great work.

[jhelvy]

Yes, we didn't yet choose to record it. We're going to next do a revision of what gets recorded so users have more control over it. See #265.

[soerendamsbo]

Hello again, John. After struggling to make indexed response option shuffling work, I've come to the conclusion that there may be a bug. I worked with Claude Code to try and fix it and we ultimately gave up. I have Claude write up a description of the issue:

Potential Bug: Indexed Option Shuffling in surveydown v1.1.0

Summary

The shuffled YAML setting in survey-settings works correctly when shuffling all options (no indices), but fails silently when using indexed shuffling to pin certain options in place.

What works

survey-settings:
  shuffled:
    - quiz_rubin
    - quiz_skaarup

This shuffles all options for each question. Confirmed working.

What doesn't work

All of these index formats fail (options appear in original order, no shuffling):

# Bare range
- quiz_rubin: 1-3

# Bracketed range
- quiz_rubin: [1-3]

# Explicit list
- quiz_rubin: [1, 2, 3]

Root cause

The issue appears to be in how the YAML front matter from survey.qmd is serialized to _survey/settings.yml. When using indexed shuffling like:

- quiz_rubin: [1, 2, 3]
- quiz_skaarup: [1, 2, 3]

The generated _survey/settings.yml produces a malformed structure:

shuffled:
  quiz_rubin:
  - - 1
    - 2
    - 3
  - ~
  - ~
  - ~
  quiz_skaarup:
  - ~
  - - 1
    - 2
    - 3
  - ~
  - ~

Instead of the expected list-of-single-key-maps, the YAML list items get merged into a single map with null (~) padding. This means parse_shuffled_yaml() never sees the {question_id: indices} format it expects.

Relevant source functions (surveydown v1.1.0)

• parse_shuffled_yaml() — expects list items that are either single-key lists or plain strings
• parse_shuffle_indices() → parse_single_index_spec() — correctly handles numeric vectors and "N-M" range strings
• read_settings_yaml() — reads from _survey/settings.yml, not directly from survey.qmd

The parsing functions themselves look correct. The bug is in whatever serializes the YAML front matter to _survey/settings.yml (likely during Quarto rendering).

Environment

• surveydown v1.1.0
• R 4.5
• macOS (aarch64-apple-darwin20)

[jhelvy]

@pingfan-hu this looks like a relatively small issue. Seems like the yaml indices aren't getting parsed properly into the _survey/settings.yml file. Could you take a look?

[pingfan-hu]

Hi @soerendamsbo , the problem is now solved. You can install our new GitHub version and give it a try:

pak::pak("surveydown-dev/surveydown", ask = FALSE)

The reason is that if all entries under shuffled are indexed (aka single-key maps), quarto::quarto_inspect() silently converts them into data.frame, not a list. But surveydown wants a list.

If it's a mix of indexed and non-indexed entries, this auto-conversion doesn't happen. Therefore, you only encounter this problem if all your entries are indexed.

# This triggers the problem
survey-settings:
  shuffled:
    - question_id_1: [1, 2, 3]
    - question_id_2: [4, 5, 6]

# This doesn't trigger
survey-settings:
  shuffled:
    - question_id_1
    - question_id_2: [4, 5, 6]

Now this auto-conversion is gone, so your survey should work fine.
Please let me know if it works!

[soerendamsbo]

Fantastic, thanks!

…

-------- Original Message --------

On Wednesday, 02/18/26 at 17:06 Pingfan Hu ***@***.***> wrote: Hi ***@***.***(https://github.com/soerendamsbo) , the problem is now solved. You can install our new GitHub version and give it a try: pak :: pak( " surveydown-dev/surveydown " , ask = FALSE ) The reason is that if all entries under shuffled are indexed (aka single-key maps), quarto::quarto_inspect() silently converts them into data.frame, not a list. But surveydown wants a list. If it's a mix of indexed and non-indexed entries, this auto-conversion doesn't happen. Therefore, you only encounter this problem if all your entries are indexed. # This triggers the problem survey-settings : shuffled : - question_id_1 : [1, 2, 3] - question_id_2 : [4, 5, 6] # This doesn't trigger survey-settings : shuffled : - question_id_1 - question_id_2 : [4, 5, 6] Now this auto-conversion is gone, so your survey should work fine. Please let me know if it works! — Reply to this email directly, [view it on GitHub](#250 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AIXW4FCA4KCNGPRETKHQMN34MSEYJAVCNFSM6AAAAACQWTJIJ6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTKOBUHE4DSOI). You are receiving this because you were mentioned.Message ID: ***@***.***>

[Hergie]

Hello,
Related to this, I'm using the shuffled YAML setting and it works correctly for mc questions (with and without index notation), but matrix question rows are not shuffled at all — they always appear in the order defined in sd_question(). I have installed the latest github version of the package.

[pingfan-hu]

Hi @Hergie ! I'll take a look and see what's going on. While the YAML setting is in the same format for both matrix and options, they are separately defined in our package. There might be some glitches in the matrix definition.
In the meantime, could you provide details of your YAML setup?

[jhelvy]

Posting a simple example that shows the issue will really help us address it. If you could, could you share a simple survey with just two questions that reproduce this issue?

[Hergie]

In R 4.5.2, surveydown 1.1.0, RStudio 2025.9.2.418. Windows.

Note: I added # before the ticks for the r chunks in survey.qmd because of the code layout conflict on github.

app.R

# Load packages
library(surveydown)

db <- sd_db_connect(ignore = TRUE)

# UI setup --------------------------------------------------------------------

ui <- sd_ui()

# Server setup ----------------------------------------------------------------

server <- function(input, output, session) {

  # Run surveydown server and define database
  sd_server(db = db)
}

# Launch the app
shiny::shinyApp(ui = ui, server = server)

survey.qmd

---
# The YAML header specifies several settings for the survey but is not required.
# A full list of settings can be found under _survey/settings.yml directory.

theme-settings:
  theme: default
  barposition: top
  
survey-settings:
  show-previous: false
  use-cookies: false
  auto-scroll: false
  rate-survey: false
  system-language: en
  highlight-unanswered: true
  highlight-color: blue
  capture-metadata: true
  all-questions-required: false
  all-shuffled: false
  shuffled:
    - q_money_happiness
    - q_happiness

---

#```{r prelims}
library(surveydown)

#```

--- active_students

#```{r}
sd_question(
  id     = 'q_happiness',
  type   = 'mc',
  label  = "Are you happy",
  option = c(
    "Yes" = "yes",
    "No" = "no"
  )
)
#```

--- comfort

#```{r}
sd_question(
  id    = 'q_money_happiness',
  type  = 'matrix',
  label = "What kind of money would make you happy?",
  row = c(
    "Salary" = "salary",
    "Payday loan" = "paydayLoan",
    "Money from parents" = "parents"
  ),
  option = c("No" = "wouldNot", "Neutral" = "neutral", "Yes" = "yes")
)
#```

--- end

#```{r}
sd_close()
#```

[pingfan-hu]

Hi @Hergie ! I tested on my Mac today and it was working fine. You can see I opened 2 sessions and they have matrix questions shuffled randomly. I'll test on my Windows machine later today and see if there's problem.

[pingfan-hu]

Hi @Hergie ! I just tried on my PC and it's working fine as well:

[soerendamsbo]

Is the issue 1.1.0 vs. 1.1.1?

…

-------- Original Message --------

On Thursday, 02/26/26 at 17:56 Pingfan Hu ***@***.***> wrote: Hi ***@***.***(https://github.com/Hergie) ! I just tried on my PC and it's working fine as well: [windows-working-fine.png (view on web)](https://github.com/user-attachments/assets/cd8196bb-d58b-452c-b65c-981f9b5d7dae) — Reply to this email directly, [view it on GitHub](#250 (reply in thread)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AIXW4FDQGS7QOHPXCN35PFD4N4QSLAVCNFSM6AAAAACQWTJIJ6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTKOJTG4YDKOA). You are receiving this because you were mentioned.Message ID: ***@***.***>

[Hergie]

Thanks! I had claude code debug it and issue a pull request: #266. As said at the top of pull request: I am not a developer so cannot critically evaluate whether the solution of claude is appropriate (though it works in my case).

[jhelvy]

I just saw the PR, could you do one more thing before we merge it - could you install the latest version of Quarto, then try your example again without the fix in the PR? That is, just install the current latest version of surveydown? I just want to try and confirm if the core issue has to do with the version of Quarto you have installed.

[Hergie]

I installed Quarto 1.8.27 and re-installed surveydown using pak::pak("surveydown-dev/surveydown", ask = FALSE).

Still encounter bug that mc shuffling works but matrix-shuffling doesn't.

[pingfan-hu]

Great @Hergie ! Now I've merged your PR (#266) into the main branch, bumping it to v1.1.2.
Now you can install using pak again:

pak::pak("surveydown-dev/surveydown", ask = FALSE)

This new update is the same as what you tested, so it should work without error.

[Hergie]

Wonderful, 1.1.2 works on my end for matrix questions too! Thanks a lot for being so responsive.