[REVIEW]: Detecting Fraud in Online Surveys by Tracing, Scoring, and Visualizing IP Addresses

[whedon]

Submitting author: @pdwaggoner (Philip D. Waggoner)
Repository: https://github.com/MAHDLab/rIP
Version: 1.1.2
Editor: @yochannah
Reviewer: @alexhanna
Archive: 10.5281/zenodo.3178394

Status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/50ab5b9f833f6c665f7e05d0577b782c"><img src="http://joss.theoj.org/papers/50ab5b9f833f6c665f7e05d0577b782c/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/50ab5b9f833f6c665f7e05d0577b782c/status.svg)](http://joss.theoj.org/papers/50ab5b9f833f6c665f7e05d0577b782c)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@alexhanna, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @yochannah know.

✨ Please try and complete your review in the next two weeks ✨

Review checklist for @alexhanna

Conflict of interest

• As the reviewer I confirm that I have read the JOSS conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Version: 1.1.2
• Authorship: Has the submitting author (@pdwaggoner) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Authors: Does the paper.md file include a list of authors with their affiliations?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @alexhanna it looks like you're currently assigned as the reviewer for this paper 🎉.

⭐ Important ⭐

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

👉 Check article proof 📄 👈

[pdwaggoner]

Looks great! Thanks again. Let me know next steps whenever they happen.

[arfon]

👋 @alexhanna - how are you getting along with your review here?

[yochannah]

quick ping - @alexhanna, do you think you might have a chance to look at this soon?

[alexhanna]

Having a look today.

…

On Sat, Mar 23, 2019 at 3:41 PM Yo Yehudi ***@***.***> wrote: quick ping - @alexhanna <https://github.com/alexhanna>, do you think you might have a chance to look at this soon? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1285 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAwvDQkIHqsAlaHwW066N_2sdkqW7Oxrks5vZq2UgaJpZM4bVjGv> .

-- Alex Hanna alex-hanna.com @alexhanna

[pdwaggoner]

hi @yochannah! Just wanted to follow up on this paper and see what the status is? No rush; just checking. Thanks!

[alexhanna]

Hi @pdwaggoner -- I'm still reviewing it, sorry. Will try to have this completed by tomorrow.

…

On Fri, Apr 19, 2019 at 9:02 AM Philip Waggoner ***@***.***> wrote: hi @yochannah <https://github.com/yochannah>! Just wanted to follow up on this paper and see what the status is? No rush; just checking. Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1285 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGC6DLBVUKL6RITWBSR7U3PRHUJ3ANCNFSM4G2WGGXQ> .

-- Alex Hanna alex-hanna.com @alexhanna

[pdwaggoner]

Hey @alexhanna - no worries at all. Thanks so much for the review!

[alexhanna]

@whedon check references

[whedon]

Attempting to check references...

[whedon]

OK DOIs

- None

MISSING DOIs

- https://doi.org/10.1177/2053168015604648 may be missing for title: Who are these people? Evaluating the demographic characteristics and political preferences of MTurk survey respondents
- https://doi.org/10.1177/2053168015622072 may be missing for title: Are samples drawn from Mechanical Turk valid for research on political ideology?
- https://doi.org/10.2139/ssrn.3233954 may be missing for title: Mturk Workers’ Use of Low-Cost "Virtual Private Servers" to Circumvent Screening Methods: A Research Note
- https://doi.org/10.1016/j.chb.2013.05.009 may be missing for title: Separate but equal? A comparison of participants and data gathered via Amazon’s MTurk, social media, and face-to-face behavioral testing
- https://doi.org/10.1037/e527772014-223 may be missing for title: Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data
- https://doi.org/10.2139/ssrn.3272468 may be missing for title: The Shape of and Solutions to the MTurk Quality Crisis
- https://doi.org/10.1126/science.aag0592 may be missing for title: Psychologists grow increasingly dependent on online research subjects
- https://doi.org/10.1037/pspa0000056 may be missing for title: The pitfall of experimenting on the web: How unattended selective attrition leads to surprising (yet false) research conclusions

INVALID DOIs

- None

[alexhanna]

Hi @pdwaggoner,

This is almost there. There's a few pieces which I think would strengthen this submission.

• Could have an automated test or an example IP address. I'd like to have this to test particular IP addresses
• Needs to have some kind of contributor statement or CONTRIBUTING file
• Needs references DOIs
• It'd be helpful to have example usage and installation on the repo page

[pdwaggoner]

Hi @alexhanna (CC: @yochannah ) - Many thanks for the review! All great points to consider. I have updated everything accordingly:

1. I updated the example in the source code (the rIP file in the R folder in the repo) to include two real IPs, along with a real IP Hub key that any user could run successfully.

2. I added a CONTRIBUTING .txt file in the repo with corresponding responsibilities for each main author.

3. Added reference DOIs for all citations that had them. The following did not have DOIs:

• https://www.wired.com/story/amazon-mechanical-turk-bot-panic/
• https://cran.r-project.org/doc/manuals/r-release/R-lang.html
• https://blog.turkprime.com/after-the-bot-scare-understanding-whats-been-happening-with-data-collection-on-mturk-and-how-to-stop-it
• Ahler et al.: http://www.gsood.com/research/papers/turk.pdf

4. I included the same real example from the source code mentioned above in #1 on the main repo page/README, along with installation instructions per your request.

5. Finally, I updated the acknowledgements in the main paper (paper.md) to thank both you and @yochannah for your review of our paper and package.

Please don't hesitate to let me know anything else that may require attention as well as any next steps you'd like us to take. Thanks again!

[pdwaggoner]

@whedon check references

[whedon]

Attempting to check references...

[whedon]

OK DOIs

- 10.1177/2053168015604648 is OK
- 10.1177/2053168015622072 is OK
- 10.2139/ssrn.3233954 is OK
- 10.1016/j.chb.2013.05.009 is OK
- 10.1177/1745691610393980 is OK
- 10.2139/ssrn.3272468 is OK
- 10.1126/science.aag0592 is OK
- 10.1037/pspa0000056 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[pdwaggoner]

HI @yochannah 0 no rush, just following up to see if we need to do anything else here? Happy to do so, if needed. Thanks!

[alexhanna]

Sorry about the delay, @pdwaggoner. Thanks for the changes in the README.

I'm getting this error when I try to test.

> getIPinfo(ipsample, "IPAddress", iphub_key = ip_hub_key)
  |======================================================================| 100%Getting IP Hub information.
Error in getIPinfo(ipsample, "IPAddress", iphub_key = ip_hub_key) : 
  argument "ipintel_key" is missing, with no default

[pdwaggoner]

No worries @alexhanna . So we left that in there to alert users that they only supplied one (or two) of the three possible ip check keys. However, this shouldn't stop the function from still pinging the provided service based on the information provided. So for the error you provided, it still worked and should have produced a plot of two "clean" IPs (if you are replicating the example from the repo README). But we decided to still throw an error message (though not a stop error) to let the user know they could've supplied additional keys to check additional services. But it still works as intended with any or all of the keys provided.

Still, to address your concern, which highlights a point of confusion in the architecture of the package, I added this line to line 12 in the README:

Note: rIP requires users to have active (free) accounts and/or valid keys at iphub, ipintel, and/or proxycheck. If users opt to use only one or not all three of these keys, then the function will throw a message letting you know as much. However, it will still work properly and check the supplied IPs based on any combination of keys the user supplies.

I hope this answers your question. Thanks!

[alexhanna]

I see now. Would it be preferable to make this a warning rather than an error?

Also, I still believe you need to add a version release, per the reviewer guidelines (Version: Does the release version given match the GitHub release (v1.1.1)?)

[pdwaggoner]

Thanks! We opted for the error for a couple reasons: first, it is slightly more efficient from our perspective, as we didn't "hard code" any error or warnings messages. It's just a default due to lack of supplied arguments, making the function run a bit quicker. And second, we like the idea of an error over a warning (but still not a stop error) as its a bit of a stronger flag to the user, "strongly suggesting" they take advantage of the other IP services supported by the function. We felt that an error message, while still allowing the function to work, would be more likely to get their attention to corroborate the findings from using less than the full 3 IP service options.

Also, thanks for the note! We updated minor things since the original submission (now 1.1.2). I just released the stable version here: https://github.com/MAHDLab/rIP/releases/tag/1.1.2 .

Anything else you need form us? Thanks again!

[alexhanna]

Nope, I think that's it. Thanks so much. Sorry this took so long.

@yochannah I think this is ready for publication.

[pdwaggoner]

No worries at all - thanks so much for the great review! We really appreciate your perspective and feedback here @alexhanna - the paper and software are much stronger as a result.

[pdwaggoner]

Hi @yochannah - anything else we need to do here? Thanks!

[whedon]

OK. 1.1.2 is the version.

[pdwaggoner]

Hi @danielskatz - thanks for the note and this sounds great! The software is archived at Zenodo, and the DOI is: 10.5281/zenodo.3178394 . Let me know whatever else you need. Thanks again!

[danielskatz]

@whedon set 10.5281/zenodo.3178394 as archive

[whedon]

OK. 10.5281/zenodo.3178394 is the archive.

[danielskatz]

@whedon accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

OK DOIs

- 10.1177/2053168015604648 is OK
- 10.1177/2053168015622072 is OK
- 10.2139/ssrn.3233954 is OK
- 10.1016/j.chb.2013.05.009 is OK
- 10.1177/1745691610393980 is OK
- 10.2139/ssrn.3272468 is OK
- 10.1126/science.aag0592 is OK
- 10.1037/pspa0000056 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[whedon]

Check final proof 👉 openjournals/joss-papers#712

If the paper PDF and Crossref deposit XML look good in openjournals/joss-papers#712, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true

[danielskatz]

👋 @pdwaggoner - there are a bunch of little issues in the paper, and I believe MAHDLab/rIP#5 will fix them. Please take a look and merge this, or otherwise resolve the broken references, figure placement, and other small issues.

[pdwaggoner]

@danielskatz merged. Thanks for the fixes! Happy to take a look at another proof if you think it’s necessary.

[danielskatz]

@whedon accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

OK DOIs

- 10.1177/2053168015604648 is OK
- 10.1177/2053168015622072 is OK
- 10.2139/ssrn.3233954 is OK
- 10.1016/j.chb.2013.05.009 is OK
- 10.1177/1745691610393980 is OK
- 10.2139/ssrn.3272468 is OK
- 10.1126/science.aag0592 is OK
- 10.1037/pspa0000056 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[whedon]

Check final proof 👉 openjournals/joss-papers#713

If the paper PDF and Crossref deposit XML look good in openjournals/joss-papers#713, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true

[danielskatz]

@whedon generate pdf

[whedon]

Attempting PDF compilation. Reticulating splines etc...

[whedon]

👉 Check article proof 📄 👈

[danielskatz]

@whedon accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

OK DOIs

- 10.1177/2053168015604648 is OK
- 10.1177/2053168015622072 is OK
- 10.2139/ssrn.3233954 is OK
- 10.1016/j.chb.2013.05.009 is OK
- 10.1177/1745691610393980 is OK
- 10.2139/ssrn.3272468 is OK
- 10.1126/science.aag0592 is OK
- 10.1037/pspa0000056 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[whedon]

Check final proof 👉 openjournals/joss-papers#714

If the paper PDF and Crossref deposit XML look good in openjournals/joss-papers#714, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true

[danielskatz]

@whedon accept deposit=true

[whedon]

Doing it live! Attempting automated processing of paper acceptance...

[whedon]

🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦

[whedon]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

0. Check final PDF and Crossref metadata that was deposited 👉 Creating pull request for 10.21105.joss.01285 joss-papers#715
1. Wait a couple of minutes to verify that the paper DOI resolves https://doi.org/10.21105/joss.01285
2. If everything looks good, then close this review issue.
3. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? notify your editorial technical team...

[whedon]

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](http://joss.theoj.org/papers/10.21105/joss.01285/status.svg)](https://doi.org/10.21105/joss.01285)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.01285">
  <img src="http://joss.theoj.org/papers/10.21105/joss.01285/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: http://joss.theoj.org/papers/10.21105/joss.01285/status.svg
   :target: https://doi.org/10.21105/joss.01285

This is how it will look in your documentation:

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: http://joss.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://numfocus.salsalabs.org/donate-to-joss