./ermurph/survey-chapter.md #35

Closed
timm opened this Issue Oct 16, 2015 · 9 comments

Projects

None yet

7 participants

@timm
Contributor
timm commented Oct 16, 2015

After review, relabel to 'reviewTwo'. After second review, relabel to 'EditorsComment'.

@timm timm added the reviewOne label Oct 16, 2015
@timm timm removed the 1.reviewOne label Oct 26, 2015
@sdeal
Contributor
sdeal commented Nov 30, 2015

Title of chapter

Avoiding Survey Design Traps: A Successful Process for Sailing Past the Temptations of Persephone's Sirens

URL to the chapter

https://github.com/ds4se/chapters/blob/master/ermurph/survey-chapter.md

Message?

Effort put into the design and testing of a survey will pay off. The authors propose to a six stage process including interviews and piloting.

Accessible?

The paper is written for a generalist audience. The authors use the legend of Odysseus and the Sirens as an analogy to motivate why planning is important to stay on track. Although the authors refer to this analogy throught the chapter, they succeed in not overstretching it. I REALLY ENJOYED READING THIS CHAPTER. I would actually recommend this chapter to my students as it presents a lot of good advice while being an easy read and the analogy might help to better memorize the suggested best practice.

Size?

The length of the chapter is about right. Maybe the authors could add a concluding sentence which reprises the Odysseus theme.

The authors contrast "psychometric evaluation" and "triangulation". I think that this is like comparing apple and oranges. I would also argue that triangulation is a widely used method in social sciences. I would suggest to totally rephrase this paragraph or to remove it.

Gotta Mantra?

How to survive your own survey: test, interview, and pilot

Best Points

The Sirens analogy: Planning helps to not be lured by the low-hanging data fruit and drown in the sea of knowledge

Minor Issues

Typo: They always wants --> They always want

@abramhindle
Contributor

Title of chapter

Avoiding Survey Design Traps: A Successful Process for Sailing Past the Temptations of Persephone's Sirens

URL to the chapter

https://github.com/ds4se/chapters/blob/master/ermurph/survey-chapter.md

Message?

What is the chapter's clear and approachable take away message?

The chapter makes it clear that like software, survey design should be iterative.

Accessible?

Is the chapters written for a generalist audience (no excessive use of technical terminology) with a minimum of diagrams and references?
How can it be made more accessible to generalist?

It is for software engineering researchers, not practitioners. Nothing clearly addresses the software engineer. Improves could be made in suggesting uses for surveys for practitioners.

Size?

Is the chapter the right length?

Yes

Should anything missing be added?
Can anything superfluous be removed (e.g. by deleting some section that does not work so well or by using less jargon, less formulae, lees diagrams, less references).?
What are the aspects of the chapter that authors SHOULD change?

Use a public domain image, maybe from archive.org, of the same public domain work.

Gotta Mantra?

We encouraged (but did not require) the chapter title to be a mantra or something cute/catchy, i.e., some slogan reflecting best practice for data science for SE? If you have suggestion for a better title, please put them here.

The title isn't really a mantra and the chapter doesn't really need it. It is laid out well enough that the parts are already bullet points to proper survey design.

Best Points

What are the best points of the chapter that the authors should NOT change?

This has a good story and a kind of mantra. The metaphor of the story helps the rest of the chapter. I'm very happy with this chapter and have thus lost a lot of objectivity. My immediate thought was to give this chapter to a student of mine to avoid issues with the survey in progress.

@sback
Contributor
sback commented Dec 5, 2015

Title of chapter

Avoiding Survey Design Traps: A Successful Process for Sailing Past the
Temptations of Persephone's Sirens

URL to the chapter

https://github.com/ds4se/chapters/blob/master/ermurph/survey-chapter.md

Message?

Designing surveys is not an easy task and should be done with care

Accessible?

The chapter is definitely accessible. Nevertheless, this sometimes results in
some suggestions/indications for better survey design that are not well
grounded and motivated.

Size?

The chapter is definitely the right length.

To be added

In general, it seems that some suggestions are not well motivated, see the
following examples:

"How did we know which statements to ask? Our statements came directly from
the stories told by participants during our semi-structured interviews.
" This
seems a good idea, but relying only on interviews for designing survey
questions may lead to other biases (e.g., what happens if all the interviewed
people are not a representative sample of important matters that should also be
investigated in the survey). I think a good survey has to be also grounded on a
through literature review. I'd encourage you to, at least, mention it.

"We recommend that the survey not take more than 15 minutes." That's a
sensible advice, can you also mention on what you ground it on?

"providing a strong initial recruitment letter is perhaps just as important as
the survey
" From my experience, most people do not read this kind of
introductions. What is your statement based on? Then, I would also mention
something about how to write such a letter.

In short, I would encourage to point the reader to why you suggest these (and
other) things.

To be shortened/removed

I like the metaphor used in the chapter: It's a very convincing one. Yet, I
have the impression that takes too much of the text (15% of the words,
excluding references, are for introducing the metaphor). The title is also very
long and could be much sharper, if the reference to the metaphor was removed.
So, in summary, I know it would feel a shame for the authors, but I think the
summary would be better without the metaphor (although I like it).

To be changed

A few parts are less rigorous than others. Despite the importance of
practitioners understanding us, it's also important to keep a certain rigor in
what we say/suggest. See the following examples:

"A final reason is that surveys give us a certain sense of quantitative comfort
because the survey responses can be downloaded into a spreadsheet or other
statistical analysis software. A popular form of question-style, Likert
statements (Strongly Disagree to Strongly Agree), can then be quantitatively
analyzed through various descriptive and statistical techniques that are
already familiar to us.
"
In this case, I would shorten this part and not talk that much about "comfort"
and "familiar to us", because that may be definitely not true (probably, some
types of researchers are more comfortable with other types of analyses). I'd
just mention that they allow to draw statistical conclusions from the
aggregated answers.

"these lures are only surface benefits." Well, no :) These are real
benefits, the point is that the survey has to be carefully designed to get them
right.

It seems that Stages 1 and 2 are about the same topic: Getting more data to
compose a more well thought survey (btw, doing interviews is not the only way
to do it well, maybe you want to mention other methods that can be used to
reach the same goal). I would encourage you to merge the two stages into one.

Gotta Mantra?

I would recommend the title to be something along this:
Make Your Online Survey Tell you The Truth

Best Points

Overall, the chapter is interesting and captivating. So the structure and
story is very good. But the metaphor and the parts that lack in rigor should be
fixed. Thanks for writing it!

@lauriew
Contributor
lauriew commented Jan 14, 2016

@CaptainEmerson
I'm getting to reading your chapter late! As Tim sent out a while ago, we were hoping for revised chapters by January 13, which happens to have passed! You have three reviews -- please revise your chapter as soon as you can.

In addition to the comments from the reviewers, I will add my own ... Stage 3, survey design. You need to make sure all the questions directly tie to the research question(s). When starting with surveys I remember asking many "interesting" questions but then realized later these questions were wasted because they were interesting but not part of the research I was really trying to do. Since then I started having students use Basili's GQM -- laying out their goals and research questions and then mapping every survey question to a research question.

I think good survey design has piloting not being "ask a few friends " but rather more pointedly representative of the population.

Put the references at the end somehow in the body of the chapter as well.

@CaptainEmerson
Contributor

@lauriew @timm @tzimmermsr What's the reference format going to be across all chapters? Author-year (Barik 2012)?

@barik barik self-assigned this Jan 31, 2016
@barik
Contributor
barik commented Jan 31, 2016

@lauriew @sback @abramhindle @sdeal. Thanks for all your reviews. There's still one more editing pass I want to do before changing the label on this issue, but otherwise, I've tried to incorporate all your suggestions. You can find them in commit 0b6bc48, with individual fixes for each bullet point in #134.

Titus

@lauriew
Contributor
lauriew commented Feb 8, 2016

@CaptainEmerson @barik
Thanks for the new version ... I'd still stick with "ask a few friends to take the survey" may sound like nice, informal advice like the tone of the chapter, but may not be the best advice for an expert to give in a book. The friends really do need to be part of the target population.

@CaptainEmerson
Contributor

@lauriew Using a friend, whether in the target population or not, is intended to avoid the "paralysis by analysis" outlined in the prior paragraph. Further, our experience has been that most initial survey bugs aren't domain specific (and thus requiring a reader from the target population), but instead are bonehead mistakes that you just need a fresh set of eyes for.

@timm
Contributor
timm commented Feb 14, 2016

@lauriew

what do i need to tell publisher? good to go?

@sback sback closed this Feb 18, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment