JOSS Review #1

[y0z]

This is the first-round review for JOSS.

According to the review checklist, I carefully read the manuscript and verified the DeepHyper package.
Only minor issues were identified regarding the "general checks," "functionality," and "documentation" aspects, and I made PRs about them (#303, #304).
Regarding the "software paper" aspect, I consider the following critical issues that require a significant manuscript revision.

Clarification of Differences from the Previous Version

I find that the previous version of the DeepHyper paper was published in 2018 and provides more information (2 columns and 10 pages, including numerical results) than the current manuscript. First, the manuscript must mention and cite the previous version to avoid self-plagiarism. The differences (i.e., additional contributions) between the new manuscript and the prior version should be clarified since JOSS does not permit repeated publication of the same work. One idea for improving the manuscript is to enrich the contents briefly described in lines 41-57, new features published in 2022-2024, i.e., apparent updates from 2018.

Comparison & Relation to Other Work

The manuscript contains no comparisons or discussions against related work and other commonly used packages. The authors should discuss the pros and cons of DeepHyper compared to related work (e.g., the ones mentioned in openjournals/joss-reviews#7901 (comment)), and, where possible, provide some numerical results.

Quality of Writing

• All figures must be referenced in the text. Please refer to Figs. 1, 2, and 3 appropriately. Please also refer to the figure in the mathematics section and provide a figure number and caption to it.

Minor Issues

• It is recommended that the section "Mathematics" be renamed more informatively, as it describes DeepHyper's optimization algorithm.
• l36. assurrogate -> as surrogate

[Deathn0t]

Hello @y0z ,

Thank you for taking the time to review our work. I'll implement the minor issues (quality of writing, typos, and #303, #304).

The DeepHyper - 2018 paper you mentioned is indeed a legacy paper from this same project. This 2018 paper was not a software paper submission but an experimental scalability analysis of an early-stage version of the current deephyper.hpo.CBO (submission made at the time to an HPC-related conference). Since the publication of this paper, only the generic high-level logic of using Bayesian optimization remains. The algorithms and software architecture have changed (the software was not open source at the time of this publication, you can still find our oldest tag https://github.com/deephyper/deephyper/tree/0.1.1 to check the difference with the current version) . We can add the citation in the JOSS paper, but we wanted to avoid confusing the user with the legacy results found in this paper (when the recent version is significantly better). To resolve this, we suggest adding the reference to the 2018 paper to our JOSS submission with a synthethic explanation of the legacy from this initial work.

The papers referenced in our submission all include comparison to other works. However, we do not have such plot in the content of the JOSS submission. While looking at other JOSS submissions example of accepted paper it seemed that the content of the paper was synthetic, so we did the same. If you think we should add a comparison plot in our submission, let us know we can work on it. A comparison could include "Software Feature Matrix", "Performance Benchmark for Black-Box Benchmarks".

Best regards,
Romain

[y0z]

Thank you for your response.

To resolve this, we suggest adding the reference to the 2018 paper to our JOSS submission with a synthetic explanation of the legacy from this initial work.

OK. It will resolve the issue.

The papers referenced in our submission all include comparison to other works... A comparison could include "Software Feature Matrix", "Performance Benchmark for Black-Box Benchmarks".

Numerical results can be optional, but the manuscript should cover the following points in the review criteria.

A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?

State of the field: Do the authors describe how this software compares to other commonly-used packages?

Regards,

Yoshihiko

[Deathn0t]

Hello @y0z

I am attaching an updated draft that follows your suggestions. Let me know if this goes in the direction you expect.

paper.pdf

[Deathn0t]

Hello @y0z ,

I just updated the tables for better presentation and removed a duplicated section I had not seen.

paper.pdf

[y0z]

Thank you for the update. Although LLMs can help improve the quality of your manuscript, please double-check it before resubmitting to detect apparent flaws.

I confirmed that the related work section was added to cover comparisons with other work.
Then, I leave some additional comments.

1. Devoting much space to computational experiments is not in line with JOSS's direction. So, the section on black-box optimization benchmarks can be omitted as it does not explain the benchmarking problems and discuss the results obtained, i.e., not helpful for readers.

2. Please provide each table/figure's number, title, and caption and refer to each table/figure in the text, such as "... is shown in Table/Figure X." For example, logo-deephyper-nobg.png does not have its caption nor be referred to in the text. A feature comparison table also does not have the number, title, and caption and is not referred to in the text.

You can find how to add a caption to a table from a paper published in JOSS.
Example:

• Paper: https://joss.theoj.org/papers/10.21105/joss.04684
• Markdown: https://github.com/easystats/datawizard/blob/main/paper/JOSS_files/paper.md?plain=1#L71-L84

3. Please cite the paper for each OSS in the related work section (i.e., adding them to References).

• HyperMapper: https://ieeexplore.ieee.org/document/9124618
• Optuna: https://dl.acm.org/doi/10.1145/3292500.3330701
• Hyperopt: https://iopscience.iop.org/article/10.1088/1749-4699/8/1/014008
• BoTorch: https://proceedings.neurips.cc/paper/2020/hash/f5b1b89d98b7286673128a5fb112cb9a-Abstract.html
• OpenBox: http://jmlr.org/papers/v25/23-0537.html
• HpBandSter: https://proceedings.mlr.press/v80/falkner18a.html
• SMAC: https://jmlr.org/papers/v23/21-0888.html

[Deathn0t]

Hello @y0z ,

1. I will add details about the quantitative benchmarks as the results are available. It is maybe better to improve the presentation rather than remove it, maybe I could keep fewer figures (aggregating them would hide the specificities of each benchmark).

2. I will implement these points. For the figure labels, the logo omits the title on purpose, as it is used as a logo. For the internal references, I tried previously to use this JOSS feature, but the compilation did not work. I will try a different method I just found in this example. Thanks for the reference to table captions.

Thanks for listing the DOIs. I was planning to add links to each software's documentation but I will replace by official references.

[Deathn0t]

Hello @y0z , after checking the citation for BOHB/Hpbandster, I suggest to not include it. This project seems unmaintained/dropped in place of SMAC3 (already cited/compared), see the README file at https://github.com/automl/HpBandSter.

[Deathn0t]

Hello @y0z , we just submitted a new version of the manuscript that should address your comments. Let me know if this is good for you.

[y0z]

@Deathn0t Thank you for the updates.

I confirmed the revised manuscript, and it is almost OK.
I leave two comments to be addressed. Once these are addressed, I think the revision will be complete.

1. The following is described in the multi-objective optimization section, but Optuna's default sampler for multi-objective optimization is NSGA-II, not MOTPE.

While Optuna defaults to MoTPE,

https://optuna.readthedocs.io/en/stable/reference/generated/optuna.study.create_study.html#optuna.study.create_study

2. a typo in summary: accuray -> accuracy

[Deathn0t]

@y0z I updated the sentence/paper to:

Multi-objective Optimization
DeepHyper uses scalarization-based approaches inspired by ParEGO (also used in SMAC), with randomized weights and a variety of scalarization functions. Optuna defaults to NSGA-II, a genetic algorithm that evolves solutions along estimated Pareto fronts. NSGA-II typically converges more slowly but catches up when the evaluation budget is sufficient.

Sorry for that! I don't know why I thought MoTPE was default... good catch thank you.

[Deathn0t]

After confirming that the items are resolved in openjournals/joss-reviews#7975 (comment)
I am closing the issue.