[PRE REVIEW]: PMLB v1.0: an open source dataset collection for benchmarking machine learning methods

[whedon]

Submitting author: @trang1618 (Trang Le)
Repository: https://github.com/EpistasisLab/pmlb
Version: v1.0.2
Editor: Pending
Reviewer: Pending
Managing EiC: Daniel S. Katz

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Author instructions

Thanks for submitting your paper to JOSS @trang1618. Currently, there isn't an JOSS editor assigned to your paper.

@trang1618 if you have any suggestions for potential reviewers then please mention them here in this thread (without tagging them with an @). In addition, this list of people have already agreed to review for JOSS and may be suitable for this submission (please start at the bottom of the list).

Editor instructions

The JOSS submission bot @whedon is here to help you find and assign reviewers and start the main review. To find out what @whedon can do for you type:

@whedon commands

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks.

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Software report (experimental):

github.com/AlDanial/cloc v 1.84  T=1.57 s (562.1 files/s, 31884.7 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
YAML                           289             35            290          42614
SVG                              1              0             15           1916
Markdown                       286            924              0           1074
RobotFramework                 285              0              0            853
Python                          11            144            328            575
Rmd                              6            232            445            123
JSON                             1              0              0             94
TeX                              1              6              0             78
Jupyter Notebook                 1              0            296             33
CSS                              1              1              0              9
HTML                             1              0              0              3
-------------------------------------------------------------------------------
SUM:                           883           1342           1374          47372
-------------------------------------------------------------------------------


Statistical information for the repository '0820d3e69f123902446bac2a' was
gathered on 2020/10/16.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
John Ramey                       2            90              1            0.72
Randy Olson                      6           374              7            3.01
Trang Le                        22          1133           4669           45.76
Weixuan                         23          3603            234           30.27
Weixuan Fu                       3           337            174            4.03
William La Cava                  5             4              7            0.09
cclauss                          1             1              3            0.03
lacava                          27          1248            737           15.66
praneelc                         1            55              1            0.44

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Randy Olson                 147           39.3          6.3               10.88
Trang Le                    503           44.4          0.6                9.94
Weixuan                     151            4.2          2.3                9.93
Weixuan Fu                   31            9.2         31.6                9.68
lacava                      160           12.8          2.4                8.12
praneelc                     55          100.0          0.2                3.64

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1145/2641190.2641198 is OK
- 10.1016/j.neunet.2012.02.016 is OK
- 10.1145/1143844.1143865 is OK
- 10.1186/s13040-017-0154-4 is OK
- 10.1145/2872427.2883029 is OK
- 10.25080/majora-92bf1922-00a is OK
- 10.1371/journal.pcbi.1004947 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[danielskatz]

👋 @trang1618 - I'm the Associate Editor-in-Chief on duty for JOSS this week, and I'm trying to understand if this submission is in scope for JOSS, and I suspect it is not. Can you explain why why you think it is? It seems to mostly be data, not software, though there is a small amount of software to access and view the data.

[danielskatz]

@whedon query scope

[whedon]

Submission flagged for editorial review.

[trangdata]

Hi @danielskatz, thank you for taking a look.

Here’s a bit of background. The initial PMLB publication from 2017 introduced the data:

PMLB: a large benchmark suite for machine learning evaluation and comparison
Randal S. Olson, William La Cava, Patryk Orzechowski, Ryan J. Urbanowicz, Jason H. Moore
BioData Mining (2017-12) https://doi.org/gfrbw5
DOI: 10.1186/s13040-017-0154-4 · PMID: 29238404 · PMCID: PMC5725843

In this manuscript submission to JOSS, we focus on the recent additions to the open source software around PMLB. This includes the python package, R library, continuous integration infrastructure to facilitate the contribution workflow, and website to summarize each dataset.

We envision future PMLB users will largely interact with the data through the software clients and website, and this manuscript describes these pieces of software.

Reading through the Scope & submission requirements, I see our manuscript fitting under “support[ing] the execution of research experiments” and “extract[ing] knowledge from large data sets”.

Given this additional context, does that change your calculation on whether our manuscript is appropriate for JOSS? Thanks again for the consideration.

[danielskatz]

Thanks @trang1618 - The editors will now look at this, including your statements above and make a decision. We should be able to let you know around the middle of next week.

[kyleniemeyer]

@trang1618 Unfortunately, the JOSS editors have decided that this submission does not meet the substantial scholarly effort criterion for review by JOSS, due to the close connection with the prior publication. Please see https://joss.readthedocs.io/en/latest/submitting.html#other-venues-for-reviewing-and-publishing-software-packages for other suggestions for how you might receive credit for your work.

[kyleniemeyer]

@whedon reject

[whedon]

Paper rejected.

[trangdata]

@kyleniemeyer Thanks for taking a look!