[PRE REVIEW]: OpenQuestion: A survey building platform written in Python

[whedon]

Submitting author: @Alcampopiano (Allan Campopiano)
Repository: https://github.com/Alcampopiano/OpenQuestion
Version: 0.0.9
Editor: Pending
Reviewer: Pending
Managing EiC: Arfon Smith

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Author instructions

Thanks for submitting your paper to JOSS @Alcampopiano. Currently, there isn't an JOSS editor assigned to your paper.

@Alcampopiano if you have any suggestions for potential reviewers then please mention them here in this thread (without tagging them with an @). In addition, this list of people have already agreed to review for JOSS and may be suitable for this submission (please start at the bottom of the list).

Editor instructions

The JOSS submission bot @whedon is here to help you find and assign reviewers and start the main review. To find out what @whedon can do for you type:

@whedon commands

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks.

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- None

MISSING DOIs

- None

INVALID DOIs

- None

[whedon]

Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.51 s (248.2 files/s, 16450.6 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
YAML                            50              9              2           2692
Python                          62            894            663           2119
Markdown                        10            210              0            767
CSS                              2            161            108            622
HTML                             1              7              0             78
TeX                              1              2              0             16
-------------------------------------------------------------------------------
SUM:                           126           1283            773           6294
-------------------------------------------------------------------------------


Statistical information for the repository '30844085de8ecb32ba77feca' was
gathered on 2021/02/13.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
alcampopiano                    23           319             93            5.21
campopianoa@hcdsb.or           134          5511           1986           94.79

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
alcampopiano                224           70.2          1.0               11.61
campopianoa@hcdsb.or       3527           64.0          5.7                6.24

[whedon]

PDF failed to compile for issue #3036 with the following error:

Invalid ORDIC digit (x)
Invalid ORDIC digit (x)
Invalid ORDIC digit (x)
Invalid ORDIC digit (x)
Invalid ORDIC digit (x)
Invalid ORDIC digit (x)
Invalid ORDIC digit (x)
Invalid ORDIC digit (x)
Invalid ORDIC digit (x)
Invalid ORDIC digit (x)
Invalid ORDIC digit (x)
Invalid ORDIC digit (x)
Invalid ORDIC digit (x)
Invalid ORDIC digit (x)
Invalid ORDIC digit (x)
/app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/lib/whedon.rb:155:in block in check_orcids': Problem with ORCID (xxxx-xxxx-xxxx-xxxx) for Stu Cork (RuntimeError) from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/lib/whedon.rb:153:in each'
from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/lib/whedon.rb:153:in check_orcids' from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/lib/whedon.rb:90:in initialize'
from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/lib/whedon/processor.rb:38:in new' from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/lib/whedon/processor.rb:38:in set_paper'
from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/bin/whedon:58:in prepare' from /app/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor/command.rb:27:in run'
from /app/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in invoke_command' from /app/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor.rb:387:in dispatch'
from /app/vendor/bundle/ruby/2.6.0/gems/thor-0.20.3/lib/thor/base.rb:466:in start' from /app/vendor/bundle/ruby/2.6.0/bundler/gems/whedon-92346a0773a4/bin/whedon:131:in <top (required)>'
from /app/vendor/bundle/ruby/2.6.0/bin/whedon:23:in load' from /app/vendor/bundle/ruby/2.6.0/bin/whedon:23:in

'

[Alcampopiano]

@whedon generate pdf

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[Alcampopiano]

Hello and thank you for your consideration.

Here is a list of potential reviewers:

humanfactors
mbod
Andytwoods

[Alcampopiano]

@whedon generate pdf

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[Alcampopiano]

@whedon generate pdf

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[Kevin-Mattheus-Moerman]

@Alcampopiano thanks for this submission. I am currently checking if this work is in scope for JOSS. As our submission requirements state, we require software to have a clear research application. Furthermore “Minor utility” or "single-function" packages are not acceptable. Can you help shed light on these points in relation to this work. Perhaps you can expand the text in the paper in terms of the applications to scientific research. You mention how researchers might use this software but it is unclear to me (as somebody not in this domain) if one would feel the need to cite this work when used. This I feel is a basic check if software is a utility tool (which one would likely not cite since it doesn't impact the science at hand) or if it is more than that and can strongly impact the research process and findings. In the latter case one would really feel the need to cite the work because it is important to some scientific outcomes. Can you work on the paper to paint a clearer picture in this respect? Can you cite papers using this work in their scientific process?
This brings me to another point. Currently you only cite two Github repositories. Can you expand the paper with a more detailed review of the scientific context of this work. Can you cite current alternatives? Are there current publications which cite this/implemented this work? Or are there papers which use similar alternative software?

You may also comment and expand on my questions here. Thanks.

[Alcampopiano]

Hello @Kevin-Mattheus-Moerman

I sincerely appreciate you having a look and I will do my best to address your very thoughtful comments.

You mention how researchers might use this software but it is unclear to me (as somebody not in this domain) if one would feel the need to cite this work when used

I completely understand this concern; however, it is not uncommon for folks in the social sciences to cite the software used for data acquisition (just as they would for software used for statistical analysis). For example, both REDcap and LimeSurvey (two platforms used for survey development) provide citations to their users to use in academic publications (here and here). These citations have been added to the paper. I believe this indicates that at least some researchers cite this type of software. To check this more thoroughly, I looked at Google Scholar, and sure enough there are many instances where researchers cite survey development software. For example, here are some search results for various platforms:

REDCap results (used in over 12k articles according to their website)

LimeSurvey results

Qualtrics results

Formstack results

Perhaps you can expand the text in the paper in terms of the applications to scientific research

Absolutely! I have substantially expanded the text to focus on applications to scientific research. Specifically,
I enabled the "reporting" feature in OpenQuestion. The reporting feature gives a Jupyter-like environment for performing interactive data visualization/analysis, describing results with Markdown, and exporting results to interactive HTML reports which can then be shared. All of this is done in context of the data collected from surveys but additional data sets can also be added to any analysis/report, potentially preventing the need to switch to another environment for the analysis and reporting phase of research. (The reporting aspect of the package was previously "turned off" to reduce the scope of the project but with a small tweak to the UI, the whole platforms "hangs" together nicely now). I have now added a section in the documentation on reporting which covers that functionality in detail (please have a look at this page to get a sense of what it can do).

In addition, I have given a few examples of how the JSON-based surveys can benefit researchers who require high volumes of similar surveys to be created (e.g., for longitudinal studies, or otherwise situations where separate but similar surveys are preferred despite being a part of the same study). This type of scaling of survey development is possible since they can be programmatically created (rather than having to use the GUI to build up many similar surveys).

Currently you only cite two Github repositories. Can you expand the paper with a more detailed review of the scientific context of this work. Can you cite current alternatives?

Yes indeed! I have added substantial detail in the paper concerning other software platforms and why OpenQuestion may be preferable for some researchers (e.g., low barrier to entry, interactive reporting/analysis, portable survey format, Python-based, FOSS). Admittedly I have not presented an exhaustive compare and contrast; however, I hope that you find the section sufficiently plump insofar as it paints a reasonable picture of the survey landscape and where OpenQuestion fits in.

Can you cite papers using this work in their scientific process?

Not yet I'm afraid (:frowning_face:) but it is not that old.

---

If there is anything else I can do to provide clarification, please let me know. I completely understand that JOSS has to adhere to a particular scope.

Take care,
Allan

[Alcampopiano]

@whedon generate pdf

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[Kevin-Mattheus-Moerman]

Thanks @Alcampopiano for the clarifying comments.

[Kevin-Mattheus-Moerman]

@kakiac @lorenanicole could one of you edit this submission?

[Kevin-Mattheus-Moerman]

@whedon invite @lorenanicole as editor

[whedon]

@lorenanicole has been invited to edit this submission.

[lorenanicole]

@Kevin-Mattheus-Moerman to confirm were the concerns raised here #3036 (comment) and answered here #3036 (comment) adequately addressed?

[danielskatz]

I think we need to discuss the scope before we proceed

[danielskatz]

@whedon query scope

[whedon]

Submission flagged for editorial review.

[danielskatz]

@Alcampopiano - I'm sorry to say that after discussion amongst the JOSS editors, we have decided that this submission is not research software as defined by JOSS. This does not mean that it is not software that is useful in research, but just that JOSS does not consider it in scope for review as research software. Please see https://joss.readthedocs.io/en/latest/submitting.html#other-venues-for-reviewing-and-publishing-software-packages for other suggestions for how you might receive credit for your work.

[danielskatz]

@whedon reject

[whedon]

Paper rejected.