Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft the Survey #1

Closed
marcdotson opened this issue Feb 20, 2019 · 51 comments
Closed

Draft the Survey #1

marcdotson opened this issue Feb 20, 2019 · 51 comments
Assignees

Comments

@marcdotson
Copy link
Owner

Hey, @paulstat6. I have three RAs that would be happy to help draft the survey, @embeukes, @morganbale, and @meghop. I'm not sure where you are in the process, but if you have an outline, we could put it together in Qualtrics or just in Word?

@marcdotson
Copy link
Owner Author

Hey @embeukes, @morganbale, and @meghop -- @paulstat6 is going to be on campus tomorrow (Thursday). We're going to talk at 3 pm in my office about the survey. If you're available, please join us!

@marcdotson
Copy link
Owner Author

We had a good meeting with @paulstat6 yesterday. @embeukes, if you'll take the lead, with @morganbale and @meghop helping, let's build out the draft of the survey (attached). This includes fleshing out the attributes and levels grid (don't worry about pictures just yet). If you have any questions, drop on by or comment here.

Auto Intenders Survey Draft 1.docx

@embeukes
Copy link
Contributor

@marcdotson will do! Are we planning on doing this in Qualtrics?

@marcdotson
Copy link
Owner Author

Nope, @embeukes. Just draft it out in Word (including associated logic in all caps and bold, like what you see already). It will be custom-coded by Paul's team.

@embeukes
Copy link
Contributor

@embeukes
Copy link
Contributor

@marcdotson are these survey participants from across the US or a certain location?

@paulstat6
Copy link
Collaborator

paulstat6 commented Feb 27, 2019 via email

@marcdotson
Copy link
Owner Author

Thanks, Paul!

@embeukes
Copy link
Contributor

@embeukes
Copy link
Contributor

embeukes commented Mar 3, 2019

@marcdotson I added demographic questions and built out the attribute & levels grid based on our discussion. https://docs.google.com/document/d/1np-hWovOozXM6BRmaLdVLIUiRuKni241ptp54Ph0EmE/edit?usp=sharing

@marcdotson
Copy link
Owner Author

@embeukes is investigating possible ways to deal with Make and Model/Body Style in the conjoint attributes and levels grid. @marcdotson is going to talk with Jeff and Elea about how this has been deal with in the past. Complete draft of the survey soon!

@embeukes
Copy link
Contributor

@marcdotson here are some of the options I found for exploring make/model.

  1. Sawtooth actually recommends using a menu-based conjoint for things like cars. Have you worked with those before and would it still measure utility the same way? https://www.sawtoothsoftware.com/download/mbcbooklet.pdf
  2. Most other surveys I found were either only testing a handful of brands (which we could do by filtering out anyone not interested in the ~10 that we include) or they used country of origin in place of brand (e.g. German, Japanese, American) Personally I don't think that second option measures what we actually want to measure...
  3. The only way I see to include more of the market would be the ACBC model. And the question with that is does it really work?

Were you able to find anything else from your contacts?

@paulstat6
Copy link
Collaborator

paulstat6 commented Mar 18, 2019 via email

@embeukes
Copy link
Contributor

@marcdotson @meghop

Here's what Megan found as far as Market Shares (the following make up over 80%). More details are found in the google sheet.

  1. Ford (13.84%)
  2. Toyota (12.90%)
  3. Chevrolet (11.70%)
  4. Honda (8.39)
  5. Nissan (7.80%)
  6. Jeep (5.65%)
  7. Subaru (3.95%)
  8. Hyundai (3.94%)
  9. Ram (3.46%)
  10. Kia (3.42%)
  11. GMC (3.23%)
  12. Dodge (2.66%)

@embeukes
Copy link
Contributor

@marcdotson
As far as grouping brands go, its far more common to group by body style than brand, due to the variety of styles that one brand can produce (e.g. Ford Mustang vs. Ford F150 vs. Ford Focus). If we wanted to group brands the only way I can think to do it would be high-end (luxury), mid, and low end segments based off of average model price.

@marcdotson
Copy link
Owner Author

marcdotson commented Mar 25, 2019

@paulstat6 @adam-n-smith @embeukes so I think we may be overcomplicating this. I discussed this with Jeff Dotson, who has done a number car surveys with Elea Feit, and he recommended screening based on type of car rather than brand.

So we ask whether or not the respondent is interesting in purchasing -- let's say -- a crossover/SUV in the next period of time -- say six months. We can use that specificity to simplify not only the brand but the other attributes and levels we include.

We're not producing this for an actual client, so the real focus of the survey revolves around having a straightforward conjoint so we can clearly demonstrate the impact of various covariates and model specifications in the model of heterogeneity.

Does that sound fair? If so, what type of vehicle should we screen based on?

@paulstat6
Copy link
Collaborator

paulstat6 commented Mar 26, 2019 via email

@adam-n-smith
Copy link
Contributor

Sedan or crossover/SUV?

@embeukes
Copy link
Contributor

Crossover/SUV's combined make up close to 50% of the market and shares for Sedans/Cars are on a downward trend - so let's do Crossover/SUV's. :)
https://www.statista.com/statistics/276506/change-in-us-car-demand-by-vehicle-type/

More details about market share of top Crossover/SUV makes and models are found here: https://docs.google.com/spreadsheets/d/1_I1_JG5LJ6Q91T0UG6L0PkkYpT8ebhaS0YZ9wr6PLYM/edit?usp=sharing

@marcdotson
Copy link
Owner Author

@paulstat6 @adam-n-smith @embeukes I think screening based on consideration of purchasing crossovers/SUVs would work best. The price range is broad enough, I don't think we need to worry about building up specific prices based on previous attribute levels -- let's just make sure we cover the expected range with sufficient granularity.

Since we're using GitHub, let's use GitHub. I've moved the survey with a screening based on crossovers/SUVs from Google Docs to R Markdown, knit as a GitHub document here.

If you'd like to make changes, you'll need to create a branch and make changes to the R Markdown, knit, then submit a push request, as detailed here. Or if you don't want to deal with that, post requested changes here and the RAs can make the changes.

@paulstat6
Copy link
Collaborator

paulstat6 commented Mar 28, 2019 via email

@marcdotson
Copy link
Owner Author

@paulstat6 you don't sound excited about this. Am I missing an objective? It's for an internal test about the use of this kind of data and you won't be presenting this to an actual client, right?

Also, to clarify, it sounds like you'll want us to provide the experimental design? How many respondents are we hoping to field?

@paulstat6
Copy link
Collaborator

paulstat6 commented Mar 29, 2019 via email

@marcdotson
Copy link
Owner Author

@paulstat6 I get that. I'll confirm with @adam-n-smith and create the design with my RAs.

@adam-n-smith
Copy link
Contributor

@marcdotson let me know what you need from me or if you want to discuss anything.

@marcdotson
Copy link
Owner Author

If it looks good to you, @adam-n-smith, I'll create the design and have you take a final review before we give @paulstat6 the go-ahead to launch. It's the kids' spring break this week, so I'll try and get it done with a number of other things before we leave Wednesday.

@marcdotson
Copy link
Owner Author

I've merged @adam-n-smith's changes and created the design matrix using the design script in Code. Balance across attribute levels is good:

design_balance

I think we're ready to start coding. I'll email you the design matrix, @paulstat6, since it's a bit too large to share here. Note that I haven't coded an outside option, although I'm assuming we'll want one.

@marcdotson
Copy link
Owner Author

@embeukes @morganbale @meghop let's get images for the brand and safety rating attributes.

  • For brand, we need logos as PNG files with white backgrounds, all the same size.
  • For safety rating, we need to create five images, each with five starts with one or more filled.

Make sense?

@paulstat6
Copy link
Collaborator

paulstat6 commented Apr 10, 2019 via email

@embeukes
Copy link
Contributor

@paulstat6 @marcdotson I have emailed those images to you and pending a pull request they will be in the Figures folder in Github as well.

@marcdotson
Copy link
Owner Author

@embeukes @morganbale @meghop we have the survey! Let's get testing.

Here's the survey.

Here's the survey with logic.

Post any issues here. We'd like to be ready to go to field by Monday.

@morganbale
Copy link

@marcdotson @embeukes @meghop

I didn't find anything wrong. The only thing I noticed is that if you say you rent, it asks the value of your current residence, wasn't sure if that is on purpose. I tested all the number limits (year born, income, etc) as well. I tested to make sure all brand options show up on open end and different combos for the first two questions.

@paulstat6
Copy link
Collaborator

paulstat6 commented Apr 19, 2019 via email

@paulstat6
Copy link
Collaborator

paulstat6 commented Apr 19, 2019 via email

@paulstat6
Copy link
Collaborator

paulstat6 commented Apr 19, 2019 via email

@marcdotson
Copy link
Owner Author

marcdotson commented Apr 19, 2019

@paulstat6 @adam-n-smith is the "Back" option only for testing purposes? We don't want them going back to modify answers based on information they get exposed to later on in the survey.

Also, I'm questioning my wording on the conjoint. Should be "Which of the following vehicles do you prefer most?" or "Which of the following vehicles would you be most likely to purchase?"

Finally, what’s the expected time for appending the cleaned geolocation data once we finish fielding the survey?

@morganbale
Copy link

@marcdotson I think most likely to purchase is better

@paulstat6
Copy link
Collaborator

paulstat6 commented Apr 20, 2019 via email

@embeukes
Copy link
Contributor

@marcdotson @morganbale
I went through and tested all the skip logic, number ranges, and anything else I could. Everything seems to work great, there's just a couple of nit-picky things I wanted to check:

  1. On Q2.7 (the auto-suggest brands) do we want them to be able to enter text other than what we have auto-suggested? As of right now any text is accepted, which is nice if we forgot a brand in our list but also means that people if people are typing faster than our auto-suggest they can misspell brands (e.g. "Honde") and those would be accepted.
  2. When people get "terminated" will they be sent to some kind of "thanks for taking the survey" screen? As of right now it just sends me to an error page.
  3. I completely skimmed the instructions on the first conjoint slide about it being a crossover/SUV. Maybe we could bold that? Or include it in the instructions on the following conjoint questions? I feel like knowing that effects how much people are willing to pay.
  4. Is 20,000 the lowest price in the conjoint? For new SUV/Crossovers that totally makes sense, but since we are including used cars as well should we go lower? Maybe I just represent the super cheap end of the market haha. :)
    Thanks!
    Emily

@marcdotson
Copy link
Owner Author

Thanks, @embeukes. @paulstat6 will need to weigh in as well.

  1. I think whatever they put is fine -- it's only semi-aided recall.
  2. Yes, the will get a termination screen. What you see now is just because we're testing it.
  3. That's a good idea: bold that it's a crossover/SUV for the first conjoint question and change the subsequent conjoint questions to: "Which of the following crossover/SUVs would you be most likely to purchase?"
  4. 20,000 is the lowest. We already have a ton of price levels. What do you think, @paulstat6 and @adam-n-smith?

@paulstat6
Copy link
Collaborator

paulstat6 commented Apr 22, 2019 via email

@paulstat6
Copy link
Collaborator

paulstat6 commented Apr 22, 2019 via email

@paulstat6
Copy link
Collaborator

paulstat6 commented Apr 22, 2019 via email

@marcdotson
Copy link
Owner Author

@paulstat6 while @embeukes's point is a good one, I don't think we want to expand the price any more than we have. If we get into the 1000s of dollars, we're getting into clear dominating alternatives.

I say we're good to proceed with the existing design matrix.

@paulstat6
Copy link
Collaborator

paulstat6 commented Apr 22, 2019 via email

@marcdotson
Copy link
Owner Author

Sounds like a good plan. @embeukes @morganbale @meghop let's do one more round of testing and then we'll soft launch.

@morganbale
Copy link

@marcdotson @embeukes @meghop Worked fine for me, the changes were in place.

@embeukes
Copy link
Contributor

@marcdotson @morganbale I just ran through everything as well and everything seems to be working!

@marcdotson
Copy link
Owner Author

@paulstat6 sounds like we're good for a soft launch. What are you planning -- 100 respondents then we stop and check?

@paulstat6
Copy link
Collaborator

paulstat6 commented Apr 22, 2019 via email

@marcdotson
Copy link
Owner Author

@paulstat6 Excel or just CSV, please.

This issue is complete. I'm closing it out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants