After review, relabel to 'reviewTwo'. After second review, relabel to 'EditorsComment'.
Avoiding Survey Design Traps: A Successful Process for Sailing Past the Temptations of Persephone's Sirens
Effort put into the design and testing of a survey will pay off. The authors propose to a six stage process including interviews and piloting.
The paper is written for a generalist audience. The authors use the legend of Odysseus and the Sirens as an analogy to motivate why planning is important to stay on track. Although the authors refer to this analogy throught the chapter, they succeed in not overstretching it. I REALLY ENJOYED READING THIS CHAPTER. I would actually recommend this chapter to my students as it presents a lot of good advice while being an easy read and the analogy might help to better memorize the suggested best practice.
The length of the chapter is about right. Maybe the authors could add a concluding sentence which reprises the Odysseus theme.
The authors contrast "psychometric evaluation" and "triangulation". I think that this is like comparing apple and oranges. I would also argue that triangulation is a widely used method in social sciences. I would suggest to totally rephrase this paragraph or to remove it.
How to survive your own survey: test, interview, and pilot
The Sirens analogy: Planning helps to not be lured by the low-hanging data fruit and drown in the sea of knowledge
Typo: They always wants --> They always want
What is the chapter's clear and approachable take away message?
The chapter makes it clear that like software, survey design should be iterative.
Is the chapters written for a generalist audience (no excessive use of technical terminology) with a minimum of diagrams and references?
How can it be made more accessible to generalist?
It is for software engineering researchers, not practitioners. Nothing clearly addresses the software engineer. Improves could be made in suggesting uses for surveys for practitioners.
Is the chapter the right length?
Should anything missing be added?
Can anything superfluous be removed (e.g. by deleting some section that does not work so well or by using less jargon, less formulae, lees diagrams, less references).?
What are the aspects of the chapter that authors SHOULD change?
Use a public domain image, maybe from archive.org, of the same public domain work.
We encouraged (but did not require) the chapter title to be a mantra or something cute/catchy, i.e., some slogan reflecting best practice for data science for SE? If you have suggestion for a better title, please put them here.
The title isn't really a mantra and the chapter doesn't really need it. It is laid out well enough that the parts are already bullet points to proper survey design.
What are the best points of the chapter that the authors should NOT change?
This has a good story and a kind of mantra. The metaphor of the story helps the rest of the chapter. I'm very happy with this chapter and have thus lost a lot of objectivity. My immediate thought was to give this chapter to a student of mine to avoid issues with the survey in progress.
Avoiding Survey Design Traps: A Successful Process for Sailing Past the
Temptations of Persephone's Sirens
Designing surveys is not an easy task and should be done with care
The chapter is definitely accessible. Nevertheless, this sometimes results in
some suggestions/indications for better survey design that are not well
grounded and motivated.
The chapter is definitely the right length.
In general, it seems that some suggestions are not well motivated, see the
"How did we know which statements to ask? Our statements came directly from
the stories told by participants during our semi-structured interviews." This
seems a good idea, but relying only on interviews for designing survey
questions may lead to other biases (e.g., what happens if all the interviewed
people are not a representative sample of important matters that should also be
investigated in the survey). I think a good survey has to be also grounded on a
through literature review. I'd encourage you to, at least, mention it.
"We recommend that the survey not take more than 15 minutes." That's a
sensible advice, can you also mention on what you ground it on?
"providing a strong initial recruitment letter is perhaps just as important as
the survey" From my experience, most people do not read this kind of
introductions. What is your statement based on? Then, I would also mention
something about how to write such a letter.
In short, I would encourage to point the reader to why you suggest these (and
I like the metaphor used in the chapter: It's a very convincing one. Yet, I
have the impression that takes too much of the text (15% of the words,
excluding references, are for introducing the metaphor). The title is also very
long and could be much sharper, if the reference to the metaphor was removed.
So, in summary, I know it would feel a shame for the authors, but I think the
summary would be better without the metaphor (although I like it).
A few parts are less rigorous than others. Despite the importance of
practitioners understanding us, it's also important to keep a certain rigor in
what we say/suggest. See the following examples:
"A final reason is that surveys give us a certain sense of quantitative comfort
because the survey responses can be downloaded into a spreadsheet or other
statistical analysis software. A popular form of question-style, Likert
statements (Strongly Disagree to Strongly Agree), can then be quantitatively
analyzed through various descriptive and statistical techniques that are
already familiar to us."
In this case, I would shorten this part and not talk that much about "comfort"
and "familiar to us", because that may be definitely not true (probably, some
types of researchers are more comfortable with other types of analyses). I'd
just mention that they allow to draw statistical conclusions from the
"these lures are only surface benefits." Well, no :) These are real
benefits, the point is that the survey has to be carefully designed to get them
It seems that Stages 1 and 2 are about the same topic: Getting more data to
compose a more well thought survey (btw, doing interviews is not the only way
to do it well, maybe you want to mention other methods that can be used to
reach the same goal). I would encourage you to merge the two stages into one.
I would recommend the title to be something along this:
Make Your Online Survey Tell you The Truth
Overall, the chapter is interesting and captivating. So the structure and
story is very good. But the metaphor and the parts that lack in rigor should be
fixed. Thanks for writing it!
I'm getting to reading your chapter late! As Tim sent out a while ago, we were hoping for revised chapters by January 13, which happens to have passed! You have three reviews -- please revise your chapter as soon as you can.
In addition to the comments from the reviewers, I will add my own ... Stage 3, survey design. You need to make sure all the questions directly tie to the research question(s). When starting with surveys I remember asking many "interesting" questions but then realized later these questions were wasted because they were interesting but not part of the research I was really trying to do. Since then I started having students use Basili's GQM -- laying out their goals and research questions and then mapping every survey question to a research question.
I think good survey design has piloting not being "ask a few friends " but rather more pointedly representative of the population.
Put the references at the end somehow in the body of the chapter as well.
@lauriew @timm @tzimmermsr What's the reference format going to be across all chapters? Author-year (Barik 2012)?
@lauriew @sback @abramhindle @sdeal. Thanks for all your reviews. There's still one more editing pass I want to do before changing the label on this issue, but otherwise, I've tried to incorporate all your suggestions. You can find them in commit 0b6bc48, with individual fixes for each bullet point in #134.
Thanks for the new version ... I'd still stick with "ask a few friends to take the survey" may sound like nice, informal advice like the tone of the chapter, but may not be the best advice for an expert to give in a book. The friends really do need to be part of the target population.
@lauriew Using a friend, whether in the target population or not, is intended to avoid the "paralysis by analysis" outlined in the prior paragraph. Further, our experience has been that most initial survey bugs aren't domain specific (and thus requiring a reader from the target population), but instead are bonehead mistakes that you just need a fresh set of eyes for.
what do i need to tell publisher? good to go?