[REVIEW]: TBFMM: A C++ generic and parallel fast multipole method library

[whedon]

Submitting author: @berenger-eu (Bérenger Bramas)
Repository: https://gitlab.inria.fr/bramas/tbfmm
Version: v1.0
Editor: @poulson
Reviewer: @pitsianis, @Himscipy, @sarats
Archive: 10.5281/zenodo.4302384

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/63718bbb2e660a1715587e8f037fc546"><img src="https://joss.theoj.org/papers/63718bbb2e660a1715587e8f037fc546/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/63718bbb2e660a1715587e8f037fc546/status.svg)](https://joss.theoj.org/papers/63718bbb2e660a1715587e8f037fc546)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@pitsianis & @Himscipy & @sarats, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @poulson know.

✨ Please try and complete your review in the next six weeks ✨

Review checklist for @pitsianis

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@berenger-eu) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @Himscipy

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@berenger-eu) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

Review checklist for @sarats

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@berenger-eu) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @pitsianis, @Himscipy, @sarats it looks like you're currently assigned to review this paper 🎉.

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

⭐ Important ⭐

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Reference check summary:

OK DOIs

- 10.7717/peerj-cs.183 is OK
- 10.1002/cpe.3723 is OK
- 10.1137/130915662 is OK
- 10.1063/1.472369 is OK
- 10.1063/1.468354 is OK
- 10.1063/1.2194548 is OK

MISSING DOIs

- https://doi.org/10.1109/aps.1994.407723 may be missing for title: The fast multipole method for periodic structures
- https://doi.org/10.1109/tpds.2017.2697857 may be missing for title: Bridging the Gap Between OpenMP and Task-Based Runtime Systems for the Fast Multipole Method
- https://doi.org/10.1155/2017/5482468 may be missing for title: Inastemp: A Novel Intrinsics-as-Template Library for Portable SIMD-Vectorization
- https://doi.org/10.4208/cicp.020215.150515sw may be missing for title: PVFMM: A parallel kernel independent FMM for particle and volume potentials
- https://doi.org/10.1016/j.enganabound.2012.07.004 may be missing for title: Fast multipole method applied to Symmetric Galerkin boundary element method for 3D elasticity and fracture problems

INVALID DOIs

- https://doi.org/10.1016/0021-9991(87)90140-9 is INVALID because of 'https://doi.org/' prefix
- https://doi.org/10.1016/j.cam.2003.12.011 is INVALID because of 'https://doi.org/' prefix

[whedon]

👉 Check article proof 📄 👈

[sarats]

I have been unable to edit/complete checklist above. Whedon has been forgetful and didn't assign me.

[danielskatz]

@whedon re-invite @sarats as reviewer

[whedon]

The reviewer already has a pending invite.

@sarats please accept the invite by clicking this link: https://github.com/openjournals/joss-reviews/invitations

[danielskatz]

Actually, you have to accept the invitation. See the instructions in the first comment in this issue.

[sarats]

Done. Thought I did this only once when I first reviewed for JOSS. Didn't realize it's required for every review.

[Himscipy]

@berenger-eu : Based on 👉 Check article proof 📄 👈

Here are some of the comments regarding the Software Paper, for you to take them into account.

• Overall Comment; check for repetitions such as "Figure Figure 3".
• TBFMM capabilities support OpenMP version 4, but the results of parallel efficiency is only shown with SPETABARU ( In house task based parallelism run time system) can you provide more details on the performance for OpenMP performance. I suggest for making the TBFMM work cater as a benchmarking tool to the HPC community the OpenMP results will be crucial, to support your statement of need.
• Do you have any performance results considering exa-scale computing clusters architectures ??

Section wise comments..

• Summary: It is well written and explain the overall purpose of the software.
• Background:
  • Check sentence structure and grammar. An example, following sentence " while others have a high degree of parallelism with significant workload available from the early beginning of each iteration.... ".
  • Remove Figure typed twice at line "A schematic view of the group-tree...."
  • Caption for Figure 2 "caption for example figure" doesn't explain anything about figure -2
• Statement of Need:
  • Complex Sentence Structure: "ScalFM has around 170K lines of code.....methods."
  • Check Grammatical errors.
• Genericity:
  • Sentence Structure & Grammar: Ex; " The algorithm has to be templatized.....and it execute method....the tree."

Thank you.

[Himscipy]

@berenger-eu; Following are my comment regarding the documentation for the library.

Overall Comment: The Readme of the code is said to be the main documentation for the code. The Readme is nicely done with details including the installation/compilation instruction ,usage and customization instructions. But navigating a large README for the purpose of usage is challenging. Therefore, I have two options for the author (op-1) Create a Table of Content in the Readme file (low-effort) (op-2) Use the instruction provided at the github-wiki, making pages to create a easy to navigate doc.(medium-effort; hight impact), I would recommend option-2. Another suggestion, would be to define separate wiki-pages for titles such "Cell", "Particle", "Trees", "Algo" etc. for organization purposes.

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?

  • The statement of need is missing in the general description " TBFMM is a Fast Multipole Method (FMM) library parallelized with the task-based method. ......TBFMM is available here...." . The description clearly details the important features of the library. I would suggest to add a title "Description/Introduction" for the paragraph.

• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.

  • The installation instructions are set at "Compiling instructions" in the document, I will highly recommend the author to change the title to "installation instruction". I see there are markdown sub heading (such as OpenMP etc.)which are not shown properly as sub-heading in the document, I understand these are dependencies hence I would suggest the author to tie them up under the "dependency list" to further enhance the ease of documentation usage.. I find all the information present, but just needs some re-organization to check this box. Additionally you can change the "File Organization" heading to a "Code Organization" Heading. There is no Automated Package management solution, I leave it up-to authors discretion to decide if this will further enhance the usage of the code to a wider community.

• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).

  • The How-To section in the documentation describes a basic example of usage in. addition individual section also, contain code examples, name of example code to felicitate the usage of the software. Good work..!

• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?

  • The documentation involves the core functionality of the software documented, in section "TBFMM design", which details the feature and core functionality of the code. As earlier mentioned, I would encourage the author to re-structure/organize the headings and sub-heading with TOC to make ease navigation in the document.

• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?

  • The test coverage report is provided with the software to ensure the functionality of the software, with unit testing feature.

• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

  • The gitlab hosted code list single author and have issue tracking mechanism and contacts.

[berenger-eu]

Dear Himanshu Sharma (@Himscipy)
Thanks for the review.

Here is my answer.

• Overall Comment; check for repetitions such as "Figure Figure 3".

Thanks, this has been changed.

• TBFMM capabilities support OpenMP version 4, but the results of parallel efficiency is only shown with SPETABARU ( In house task based parallelism run time system) can you provide more details on the performance for OpenMP performance. I suggest for making the TBFMM work cater as a benchmarking tool to the HPC community the OpenMP results will be crucial, to support your statement of need.

The new version of the paper includes results with OpenMP. As it is described in the related section, there is a speedup with OpenMP 4 but the scalability is limited (because the mutexinout/commutative access is only supported from version 5).
Therefore, up to 8 threads OpenMP can be used, otherwise Spetaberu is currently the way to go (until the compilers support OpenMP 5).

• Do you have any performance results considering exa-scale computing clusters architectures ??

TBFMM will be executed on ARM CPU soon. But we definitively need to add MPI in order to run on multiple nodes, and this will be our next feature.

• Background:
  Check sentence structure and grammar. An example, following sentence " while others have a high degree of parallelism with significant workload available from the early beginning of each iteration.... ".
  Remove Figure typed twice at line "A schematic view of the group-tree...."
  Caption for Figure 2 "caption for example figure" doesn't explain anything about figure -2

I updated the sentence and tried to check the grammar.

• Statement of Need:
  Complex Sentence Structure: "ScalFM has around 170K lines of code.....methods."
  Check Grammatical errors.

I updated the sentence and tried to check the grammar.

• Genericity:
  Sentence Structure & Grammar: Ex; " The algorithm has to be templatized.....and it execute method....the tree."

I updated the sentence and tried to check the grammar.

• Overall Comment: The Readme of the code is said to be the main documentation for the code. The Readme is nicely done with details including the installation/compilation instruction ,usage and customization instructions. But navigating a large README for the purpose of usage is challenging. Therefore, I have two options for the author (op-1) Create a Table of Content in the Readme file (low-effort) (op-2) Use the instruction provided at the github-wiki, making pages to create a easy to navigate doc.(medium-effort; hight impact), I would recommend option-2. Another suggestion, would be to define separate wiki-pages for titles such "Cell", "Particle", "Trees", "Algo" etc. for organization purposes.

I fully agree that both were needed. I added a TOC in the README, and I add a copy in the wiki: https://gitlab.inria.fr/bramas/tbfmm/-/wikis/home
I feel like the "gitlab" wiki seems less nice than the github version.

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• The statement of need is missing in the general description " TBFMM is a Fast Multipole Method (FMM) library parallelized with the task-based method. ......TBFMM is available here...." . The description clearly details the important features of the library. I would suggest to add a title "Description/Introduction" for the paragraph.

I added sentences from the paper.

• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• The installation instructions are set at "Compiling instructions" in the document, I will highly recommend the author to change the title to "installation instruction". I see there are markdown sub heading (such as OpenMP etc.)which are not shown properly as sub-heading in the document, I understand these are dependencies hence I would suggest the author to tie them up under the "dependency list" to further enhance the ease of documentation usage.. I find all the information present, but just needs some re-organization to check this box. Additionally you can change the "File Organization" heading to a "Code Organization" Heading. There is no Automated Package management solution, I leave it up-to authors discretion to decide if this will further enhance the usage of the code to a wider community.

I renamed the sections.

• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• The How-To section in the documentation describes a basic example of usage in. addition individual section also, contain code examples, name of example code to felicitate the usage of the software. Good work..!

Noted.

• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• The documentation involves the core functionality of the software documented, in section "TBFMM design", which details the feature and core functionality of the code. As earlier mentioned, I would encourage the author to re-structure/organize the headings and sub-heading with TOC to make ease navigation in the document.

I feel that adding the TOC solved the problem, I look forward to your confirmation.

• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• The test coverage report is provided with the software to ensure the functionality of the software, with unit testing feature.

Noted.

• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support
  • The gitlab hosted code list single author and have issue tracking mechanism and contacts.

Noted.

Remark It seems that an invisible bad char was in the paper.md (and the README.md) such that the section names were not right (for instance, the ## Features section was written in plain text in the paper). Therefore, I hope that rebuilding the paper will fix this.

@whedon generate pdf

[berenger-eu]

@whedon generate pdf

[whedon]

👉 Check article proof 📄 👈

[Himscipy]

@berenger-eu Thank you for addressing the documentation comments and for the response.

Here are my comments about the installation process/ functionality testing;

• Please add First Step-1 in the installation procedure : Clone the current repo or rewrite them as individual steps I quote from documentation.
  The build process consists in the following steps: adding git submodules (optional), creating a build directory, running cmake, configuring cmake, running make and that's all. The submodules are Inastemp (for vectorization) and SPETABARU (a task-based runtime system). Both are optional, and to activate them, it is needed to clone their repository before running cmake.

I have couldn't raise issue with the build log file, I cannot create issue on the repo, since it was asking some INRIA login access. I have added the build log below for your reference. I see that the issue could be my compiler but please let me know and if you can provide some additional comment or procedure to build it.

• My Machine Specifics first:

ProductName: Mac OS X
ProductVersion: 10.15.5
cmake version 3.18.1
CMake suite maintained and supported by Kitware (kitware.com/cmake).
Apple clang version 11.0.3 (clang-1103.0.32.62)

Build Error Below

[  8%] Building CXX object CMakeFiles/testRotationKernel.dir/examples/testRotationKernel.cpp.o
In file included from /Users/shar306/Work/Reviews/JOSS_Review/tbfmm/examples/testRotationKernel.cpp:11:
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/kernels/rotationkernel/FRotationKernel.hpp:122:32: error: no matching member function for call to 'reset'
            M2MTranslationCoef.reset(new RealType[treeHeight-1][P+1]);
            ~~~~~~~~~~~~~~~~~~~^~~~~
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/kernels/rotationkernel/FRotationKernel.hpp:898:9: note: in instantiation of member function 'FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::precomputeTranslationCoef' requested here
        precomputeTranslationCoef();
        ^
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:1825:31: note: in instantiation of member function 'FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::FRotationKernel' requested here
            ::new((void*)__p) _Up(_VSTD::forward<_Args>(__args)...);
                              ^
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:1717:18: note: in instantiation of function template specialization 'std::__1::allocator<FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > >::construct<FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, const TbfSpacialConfiguration<double, 3> &>' requested here
            {__a.construct(__p, _VSTD::forward<_Args>(__args)...);}
                 ^
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:1560:14: note: in instantiation of function template specialization 'std::__1::allocator_traits<std::__1::allocator<FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > > >::__construct<FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, const TbfSpacialConfiguration<double, 3> &>' requested here
            {__construct(__has_construct<allocator_type, _Tp*, _Args...>(),
             ^
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/vector:1687:25: note: in instantiation of function template specialization 'std::__1::allocator_traits<std::__1::allocator<FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > > >::construct<FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, const TbfSpacialConfiguration<double, 3> &>' requested here
        __alloc_traits::construct(this->__alloc(),
                        ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/tbfsmspetabarualgorithm.hpp:250:17: note: in instantiation of function template specialization 'std::__1::vector<FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, std::__1::allocator<FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > > >::emplace_back<const TbfSpacialConfiguration<double, 3> &>' requested here
        kernels.emplace_back(configuration);
                ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/examples/testRotationKernel.cpp:128:51: note: in instantiation of member function 'TbfSmSpetabaruAlgorithm<double, FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::TbfSmSpetabaruAlgorithm' requested here
    std::unique_ptr<AlgorithmClass> algorithm(new AlgorithmClass(configuration));    
                                                  ^
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:3909:9: note: candidate template ignored: requirement 'is_convertible<double (*)[13], double (*)[][13]>::value' was not satisfied [with _Yp = double [13]]
        reset(_Yp* __p);
        ^
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:3917:9: note: candidate function template not viable: requires 2 arguments, but 1 was provided
        reset(_Yp* __p, _Dp __d);
        ^
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:3925:9: note: candidate function template not viable: requires 3 arguments, but 1 was provided
        reset(_Yp* __p, _Dp __d, _Alloc __a);
        ^
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:3901:10: note: candidate function not viable: requires 0 arguments, but 1 was provided
    void reset() _NOEXCEPT;
         ^
In file included from /Users/shar306/Work/Reviews/JOSS_Review/tbfmm/examples/testRotationKernel.cpp:11:
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/kernels/rotationkernel/FRotationKernel.hpp:123:32: error: no matching member function for call to 'reset'
            L2LTranslationCoef.reset(new RealType[treeHeight-1][P+1]);
            ~~~~~~~~~~~~~~~~~~~^~~~~
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:3909:9: note: candidate template ignored: requirement 'is_convertible<double (*)[13], double (*)[][13]>::value' was not satisfied [with _Yp = double [13]]
        reset(_Yp* __p);
        ^
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:3917:9: note: candidate function template not viable: requires 2 arguments, but 1 was provided
        reset(_Yp* __p, _Dp __d);
        ^
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:3925:9: note: candidate function template not viable: requires 3 arguments, but 1 was provided
        reset(_Yp* __p, _Dp __d, _Alloc __a);
        ^
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:3901:10: note: candidate function not viable: requires 0 arguments, but 1 was provided
    void reset() _NOEXCEPT;
         ^
In file included from /Users/shar306/Work/Reviews/JOSS_Review/tbfmm/examples/testRotationKernel.cpp:11:
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/kernels/rotationkernel/FRotationKernel.hpp:136:21: error: type 'std::shared_ptr<RealType [][13]>' does not provide a subscript operator
                    M2MTranslationCoef[idxLevel][idx] = minus_1_pow_idx * bPowIdx / factorials[idx];
                    ^~~~~~~~~~~~~~~~~~ ~~~~~~~~
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/kernels/rotationkernel/FRotationKernel.hpp:138:21: error: type 'std::shared_ptr<RealType [][13]>' does not provide a subscript operator
                    L2LTranslationCoef[idxLevel][idx] = bPowIdx / factorials[idx];
                    ^~~~~~~~~~~~~~~~~~ ~~~~~~~~
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/kernels/rotationkernel/FRotationKernel.hpp:149:32: error: no matching member function for call to 'reset'
            M2LTranslationCoef.reset(new RealType[treeHeight][343][P+1]);
            ~~~~~~~~~~~~~~~~~~~^~~~~
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:3909:9: note: candidate template ignored: requirement 'is_convertible<double (*)[343][13], double (*)[][343][13]>::value' was not satisfied [with _Yp = double [343][13]]
        reset(_Yp* __p);
        ^
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:3917:9: note: candidate function template not viable: requires 2 arguments, but 1 was provided
        reset(_Yp* __p, _Dp __d);
        ^
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:3925:9: note: candidate function template not viable: requires 3 arguments, but 1 was provided
        reset(_Yp* __p, _Dp __d, _Alloc __a);
        ^
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:3901:10: note: candidate function not viable: requires 0 arguments, but 1 was provided
    void reset() _NOEXCEPT;
         ^
In file included from /Users/shar306/Work/Reviews/JOSS_Review/tbfmm/examples/testRotationKernel.cpp:11:
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/kernels/rotationkernel/FRotationKernel.hpp:174:37: error: type 'std::shared_ptr<RealType [][343][13]>' does not provide a subscript operator
                                    M2LTranslationCoef[idxLevel][position][idx] = factorials[idx] / bPowIdx1;
                                    ^~~~~~~~~~~~~~~~~~ ~~~~~~~~
In file included from /Users/shar306/Work/Reviews/JOSS_Review/tbfmm/examples/testRotationKernel.cpp:1:
In file included from /Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/spacial/tbfmortonspaceindex.hpp:6:
In file included from /Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/utils/tbfutils.hpp:8:
In file included from /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/array:111:
In file included from /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/algorithm:643:
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:1825:31: error: no matching constructor for initialization of 'FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >'
            ::new((void*)__p) _Up(_VSTD::forward<_Args>(__args)...);
                              ^   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:1717:18: note: in instantiation of function template specialization 'std::__1::allocator<FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > >::construct<FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > &>' requested here
            {__a.construct(__p, _VSTD::forward<_Args>(__args)...);}
                 ^
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:1560:14: note: in instantiation of function template specialization 'std::__1::allocator_traits<std::__1::allocator<FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > > >::__construct<FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > &>' requested here
            {__construct(__has_construct<allocator_type, _Tp*, _Args...>(),
             ^
/Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/vector:1687:25: note: in instantiation of function template specialization 'std::__1::allocator_traits<std::__1::allocator<FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > > >::construct<FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > &>' requested here
        __alloc_traits::construct(this->__alloc(),
                        ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/tbfsmspetabarualgorithm.hpp:241:21: note: in instantiation of function template specialization 'std::__1::vector<FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, std::__1::allocator<FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > > >::emplace_back<FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > &>' requested here
            kernels.emplace_back(kernels[0]);
                    ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/tbfsmspetabarualgorithm.hpp:269:9: note: in instantiation of member function 'TbfSmSpetabaruAlgorithm<double, FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::increaseNumberOfKernels' requested here
        increaseNumberOfKernels(runtime.getNbThreads());
        ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/examples/testRotationKernel.cpp:135:20: note: in instantiation of function template specialization 'TbfSmSpetabaruAlgorithm<double, FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::execute<TbfTree<double, double, 4, double, 4, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > >' requested here
        algorithm->execute(tree);
                   ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/kernels/rotationkernel/FRotationKernel.hpp:888:5: note: candidate constructor not viable: no known conversion from 'FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >' to 'const FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::SpacialConfiguration' (aka 'const TbfSpacialConfiguration<double, SpaceIndexType::Dim>') for 1st argument
    FRotationKernel(const SpacialConfiguration& inConfiguration) :
    ^
In file included from /Users/shar306/Work/Reviews/JOSS_Review/tbfmm/examples/testRotationKernel.cpp:8:
In file included from /Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/tbfalgorithmselecter.hpp:6:
In file included from /Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/sequential/tbfalgorithm.hpp:6:
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/../sequential/tbfgroupkernelinterface.hpp:35:37: warning: variable length arrays are a C99 feature [-Wvla-extension]
        long int positionsOfChildren[spaceSystem.getNbChildrenPerCell()];
                                    ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/tbfsmspetabarualgorithm.hpp:85:35: note: in instantiation of function template specialization 'TbfGroupKernelInterface<TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::M2M<FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, TbfCellsContainer<double, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > >' requested here
                    kernelWrapper.M2M(idxLevel, kernels[SpUtils::GetThreadId()-1], lowerGroup, upperGroup);
                                  ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/tbfsmspetabarualgorithm.hpp:275:13: note: in instantiation of function template specialization 'TbfSmSpetabaruAlgorithm<double, FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::M2M<TbfTree<double, double, 4, double, 4, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > >' requested here
            M2M(runtime, inTree);
            ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/examples/testRotationKernel.cpp:135:20: note: in instantiation of function template specialization 'TbfSmSpetabaruAlgorithm<double, FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::execute<TbfTree<double, double, 4, double, 4, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > >' requested here
        algorithm->execute(tree);
                   ^
In file included from /Users/shar306/Work/Reviews/JOSS_Review/tbfmm/examples/testRotationKernel.cpp:11:
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/kernels/rotationkernel/FRotationKernel.hpp:1002:37: error: type 'const std::shared_ptr<RealType [][13]>' does not provide a subscript operator
        const RealType*const coef = M2MTranslationCoef[inLevel];
                                    ^~~~~~~~~~~~~~~~~~ ~~~~~~~
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/../sequential/tbfgroupkernelinterface.hpp:63:26: note: in instantiation of function template specialization 'FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::M2M<TbfCellsContainer<double, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::CellHeader, std::__1::vector<std::__1::reference_wrapper<const std::__1::array<std::__1::complex<double>, 91> >, std::__1::allocator<std::__1::reference_wrapper<const std::__1::array<std::__1::complex<double>, 91> > > >, std::__1::array<std::__1::complex<double>, 91> >' requested here
                inKernel.M2M(inUpperGroup.getCellSymbData(idxParent),
                         ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/tbfsmspetabarualgorithm.hpp:85:35: note: in instantiation of function template specialization 'TbfGroupKernelInterface<TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::M2M<FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, TbfCellsContainer<double, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > >' requested here
                    kernelWrapper.M2M(idxLevel, kernels[SpUtils::GetThreadId()-1], lowerGroup, upperGroup);
                                  ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/tbfsmspetabarualgorithm.hpp:275:13: note: in instantiation of function template specialization 'TbfSmSpetabaruAlgorithm<double, FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::M2M<TbfTree<double, double, 4, double, 4, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > >' requested here
            M2M(runtime, inTree);
            ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/examples/testRotationKernel.cpp:135:20: note: in instantiation of function template specialization 'TbfSmSpetabaruAlgorithm<double, FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::execute<TbfTree<double, double, 4, double, 4, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > >' requested here
        algorithm->execute(tree);
                   ^
In file included from /Users/shar306/Work/Reviews/JOSS_Review/tbfmm/examples/testRotationKernel.cpp:8:
In file included from /Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/tbfalgorithmselecter.hpp:6:
In file included from /Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/sequential/tbfalgorithm.hpp:6:
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/../sequential/tbfgroupkernelinterface.hpp:132:38: warning: variable length arrays are a C99 feature [-Wvla-extension]
        long int positionsOfNeighbors[spaceSystem.getNbInteractionsPerCell()];
                                     ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/tbfsmspetabarualgorithm.hpp:119:39: note: in instantiation of function template specialization 'TbfGroupKernelInterface<TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::M2LBetweenGroups<FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, TbfCellsContainer<double, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, TbfCellsContainer<double, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, std::__1::vector<TbfXtoXInteraction<long>, std::__1::allocator<TbfXtoXInteraction<long> > > >' requested here
                        kernelWrapper.M2LBetweenGroups(idxLevel, kernels[SpUtils::GetThreadId()-1], groupTarget, groupSrc, std::move(indexesVec));
                                      ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/tbfalgorithmutils.hpp:70:5: note: in instantiation of function template specialization 'TbfAlgorithmUtils::TbfMapIndexesAndBlocks<std::__1::vector<TbfXtoXInteraction<long>, std::__1::allocator<TbfXtoXInteraction<long> > >, std::__1::vector<TbfCellsContainer<double, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, std::__1::allocator<TbfCellsContainer<double, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > > >, std::__1::vector<TbfCellsContainer<double, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, std::__1::allocator<TbfCellsContainer<double, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > > >, (lambda at /Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/tbfsmspetabarualgorithm.hpp:114:48)>' requested here
    TbfMapIndexesAndBlocks(std::forward<IndexContainerClass>(inIndexes), inGroups, idxWorkingGroup, inGroups,
    ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/tbfsmspetabarualgorithm.hpp:113:36: note: in instantiation of function template specialization 'TbfAlgorithmUtils::TbfMapIndexesAndBlocks<std::__1::vector<TbfXtoXInteraction<long>, std::__1::allocator<TbfXtoXInteraction<long> > >, std::__1::vector<TbfCellsContainer<double, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, std::__1::allocator<TbfCellsContainer<double, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > > >, (lambda at /Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/tbfsmspetabarualgorithm.hpp:114:48)>' requested here
                TbfAlgorithmUtils::TbfMapIndexesAndBlocks(std::move(indexesForGroup.second), cellGroups, std::distance(cellGroups.begin(),currentCellGroup),
                                   ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/tbfsmspetabarualgorithm.hpp:278:13: note: in instantiation of function template specialization 'TbfSmSpetabaruAlgorithm<double, FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::M2L<TbfTree<double, double, 4, double, 4, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > >' requested here
            M2L(runtime, inTree);
            ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/examples/testRotationKernel.cpp:135:20: note: in instantiation of function template specialization 'TbfSmSpetabaruAlgorithm<double, FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::execute<TbfTree<double, double, 4, double, 4, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > >' requested here
        algorithm->execute(tree);
                   ^
In file included from /Users/shar306/Work/Reviews/JOSS_Review/tbfmm/examples/testRotationKernel.cpp:11:
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/kernels/rotationkernel/FRotationKernel.hpp:1061:41: error: type 'std::shared_ptr<RealType [][343][13]>' does not provide a subscript operator
            const RealType*const coef = M2LTranslationCoef[inLevel][neighPos[idxNeigh]];
                                        ^~~~~~~~~~~~~~~~~~ ~~~~~~~
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/../sequential/tbfgroupkernelinterface.hpp:159:26: note: in instantiation of function template specialization 'FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::M2L<TbfCellsContainer<double, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::CellHeader, std::__1::vector<std::__1::reference_wrapper<const std::__1::array<std::__1::complex<double>, 91> >, std::__1::allocator<std::__1::reference_wrapper<const std::__1::array<std::__1::complex<double>, 91> > > >, std::__1::array<std::__1::complex<double>, 91> >' requested here
                inKernel.M2L(inCellGroup.getCellSymbData(interaction.globalTargetPos),
                         ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/tbfsmspetabarualgorithm.hpp:119:39: note: in instantiation of function template specialization 'TbfGroupKernelInterface<TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::M2LBetweenGroups<FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, TbfCellsContainer<double, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, TbfCellsContainer<double, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, std::__1::vector<TbfXtoXInteraction<long>, std::__1::allocator<TbfXtoXInteraction<long> > > >' requested here
                        kernelWrapper.M2LBetweenGroups(idxLevel, kernels[SpUtils::GetThreadId()-1], groupTarget, groupSrc, std::move(indexesVec));
                                      ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/tbfalgorithmutils.hpp:70:5: note: in instantiation of function template specialization 'TbfAlgorithmUtils::TbfMapIndexesAndBlocks<std::__1::vector<TbfXtoXInteraction<long>, std::__1::allocator<TbfXtoXInteraction<long> > >, std::__1::vector<TbfCellsContainer<double, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, std::__1::allocator<TbfCellsContainer<double, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > > >, std::__1::vector<TbfCellsContainer<double, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, std::__1::allocator<TbfCellsContainer<double, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > > >, (lambda at /Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/tbfsmspetabarualgorithm.hpp:114:48)>' requested here
    TbfMapIndexesAndBlocks(std::forward<IndexContainerClass>(inIndexes), inGroups, idxWorkingGroup, inGroups,
    ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/tbfsmspetabarualgorithm.hpp:113:36: note: in instantiation of function template specialization 'TbfAlgorithmUtils::TbfMapIndexesAndBlocks<std::__1::vector<TbfXtoXInteraction<long>, std::__1::allocator<TbfXtoXInteraction<long> > >, std::__1::vector<TbfCellsContainer<double, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, std::__1::allocator<TbfCellsContainer<double, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > > >, (lambda at /Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/tbfsmspetabarualgorithm.hpp:114:48)>' requested here
                TbfAlgorithmUtils::TbfMapIndexesAndBlocks(std::move(indexesForGroup.second), cellGroups, std::distance(cellGroups.begin(),currentCellGroup),
                                   ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/tbfsmspetabarualgorithm.hpp:278:13: note: in instantiation of function template specialization 'TbfSmSpetabaruAlgorithm<double, FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::M2L<TbfTree<double, double, 4, double, 4, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > >' requested here
            M2L(runtime, inTree);
            ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/examples/testRotationKernel.cpp:135:20: note: in instantiation of function template specialization 'TbfSmSpetabaruAlgorithm<double, FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::execute<TbfTree<double, double, 4, double, 4, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > >' requested here
        algorithm->execute(tree);
                   ^
In file included from /Users/shar306/Work/Reviews/JOSS_Review/tbfmm/examples/testRotationKernel.cpp:8:
In file included from /Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/tbfalgorithmselecter.hpp:6:
In file included from /Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/sequential/tbfalgorithm.hpp:6:
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/../sequential/tbfgroupkernelinterface.hpp:89:38: warning: variable length arrays are a C99 feature [-Wvla-extension]
        long int positionsOfNeighbors[spaceSystem.getNbInteractionsPerCell()];
                                     ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/tbfsmspetabarualgorithm.hpp:126:35: note: in instantiation of function template specialization 'TbfGroupKernelInterface<TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::M2LInGroup<FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, TbfCellsContainer<double, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, std::__1::vector<TbfXtoXInteraction<long>, std::__1::allocator<TbfXtoXInteraction<long> > > >' requested here
                    kernelWrapper.M2LInGroup(idxLevel, kernels[SpUtils::GetThreadId()-1], currentGroup, indexesForGroup_first);
                                  ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/tbfsmspetabarualgorithm.hpp:278:13: note: in instantiation of function template specialization 'TbfSmSpetabaruAlgorithm<double, FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::M2L<TbfTree<double, double, 4, double, 4, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > >' requested here
            M2L(runtime, inTree);
            ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/examples/testRotationKernel.cpp:135:20: note: in instantiation of function template specialization 'TbfSmSpetabaruAlgorithm<double, FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::execute<TbfTree<double, double, 4, double, 4, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > >' requested here
        algorithm->execute(tree);
                   ^
In file included from /Users/shar306/Work/Reviews/JOSS_Review/tbfmm/examples/testRotationKernel.cpp:8:
In file included from /Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/tbfalgorithmselecter.hpp:6:
In file included from /Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/sequential/tbfalgorithm.hpp:6:
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/../sequential/tbfgroupkernelinterface.hpp:114:22: error: no matching member function for call to 'M2L'
            inKernel.M2L(inCellGroup.getCellSymbData(interaction.globalTargetPos),
            ~~~~~~~~~^~~
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/kernels/rotationkernel/FRotationKernel.hpp:1054:10: note: candidate template ignored: substitution failure [with CellSymbolicData = TbfCellsContainer<double, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::CellHeader, CellClassContainer = std::__1::vector<std::__1::reference_wrapper<const std::__1::array<std::__1::complex<double>, 91> >, std::__1::allocator<std::__1::reference_wrapper<const std::__1::array<std::__1::complex<double>, 91> > > >, CellClass = std::__1::array<std::__1::complex<double>, 91>]
    void M2L(const CellSymbolicData& /*inTargetIndex*/,
         ^
In file included from /Users/shar306/Work/Reviews/JOSS_Review/tbfmm/examples/testRotationKernel.cpp:8:
In file included from /Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/tbfalgorithmselecter.hpp:6:
In file included from /Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/sequential/tbfalgorithm.hpp:6:
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/../sequential/tbfgroupkernelinterface.hpp:176:37: warning: variable length arrays are a C99 feature [-Wvla-extension]
        long int positionsOfChildren[spaceSystem.getNbChildrenPerCell()];
                                    ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/tbfsmspetabarualgorithm.hpp:154:35: note: in instantiation of function template specialization 'TbfGroupKernelInterface<TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::L2L<FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, TbfCellsContainer<double, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > >' requested here
                    kernelWrapper.L2L(idxLevel, kernels[SpUtils::GetThreadId()-1], upperGroup, lowerGroup);
                                  ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/tbfsmspetabarualgorithm.hpp:281:13: note: in instantiation of function template specialization 'TbfSmSpetabaruAlgorithm<double, FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::L2L<TbfTree<double, double, 4, double, 4, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > >' requested here
            L2L(runtime, inTree);
            ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/examples/testRotationKernel.cpp:135:20: note: in instantiation of function template specialization 'TbfSmSpetabaruAlgorithm<double, FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::execute<TbfTree<double, double, 4, double, 4, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > >' requested here
        algorithm->execute(tree);
                   ^
In file included from /Users/shar306/Work/Reviews/JOSS_Review/tbfmm/examples/testRotationKernel.cpp:11:
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/kernels/rotationkernel/FRotationKernel.hpp:1116:37: error: type 'std::shared_ptr<RealType [][13]>' does not provide a subscript operator
        const RealType*const coef = L2LTranslationCoef[inLevel];
                                    ^~~~~~~~~~~~~~~~~~ ~~~~~~~
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/../sequential/tbfgroupkernelinterface.hpp:204:26: note: in instantiation of function template specialization 'FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::L2L<TbfCellsContainer<double, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::CellHeader, std::__1::array<std::__1::complex<double>, 91>, std::__1::vector<std::__1::reference_wrapper<std::__1::array<std::__1::complex<double>, 91> >, std::__1::allocator<std::__1::reference_wrapper<std::__1::array<std::__1::complex<double>, 91> > > > >' requested here
                inKernel.L2L(inUpperGroup.getCellSymbData(idxParent),
                         ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/tbfsmspetabarualgorithm.hpp:154:35: note: in instantiation of function template specialization 'TbfGroupKernelInterface<TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::L2L<FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, TbfCellsContainer<double, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > >' requested here
                    kernelWrapper.L2L(idxLevel, kernels[SpUtils::GetThreadId()-1], upperGroup, lowerGroup);
                                  ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/src/algorithms/smspetabaru/tbfsmspetabarualgorithm.hpp:281:13: note: in instantiation of function template specialization 'TbfSmSpetabaruAlgorithm<double, FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::L2L<TbfTree<double, double, 4, double, 4, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > >' requested here
            L2L(runtime, inTree);
            ^
/Users/shar306/Work/Reviews/JOSS_Review/tbfmm/examples/testRotationKernel.cpp:135:20: note: in instantiation of function template specialization 'TbfSmSpetabaruAlgorithm<double, FRotationKernel<double, 12, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> >::execute<TbfTree<double, double, 4, double, 4, std::__1::array<std::__1::complex<double>, 91>, std::__1::array<std::__1::complex<double>, 91>, TbfMortonSpaceIndex<3, TbfSpacialConfiguration<double, 3>, false> > >' requested here
        algorithm->execute(tree);
                   ^
4 warnings and 11 errors generated.
make[2]: *** [CMakeFiles/testRotationKernel.dir/examples/testRotationKernel.cpp.o] Error 1
make[1]: *** [CMakeFiles/testRotationKernel.dir/all] Error 2
make: *** [all] Error 2

[berenger-eu]

Dear @Himscipy,

Thanks for your message.
Here are my answers to your remarks.

Please add First Step-1 in the installation procedure : Clone the current repo or rewrite them as individual steps I quote from documentation.

You are right that this was definitively missing.
I updated the README to include the cloning stage.

I cannot create issue on the repo, since it was asking some INRIA login access.

Unfortunately, the Inria gitlab access policy changed few weeks ago.
However, there is a mirrored repository on github: https://github.com/berenger-eu/tbfmm
Therefore, you can of course open an issue there, and I wrote it in the README
such that any user can do the same.

error: no matching member function for call to 'reset' ...........

Surprisingly there are some errors with Mac + clang (but it works with Linux + clang or even Mac + gcc).
I detected two problems, one from cmake that was not enabling c++17 correctly on Mac and another from a conversion issue (that should not be a problem).
However, that should be fixed now, I used a Mac where the same error was happening and now it works on it, so I hope that will work on yours too.
Tested on Max with GNU c++ 9.3 and clang++ 10.

Regards.

[Himscipy]

Hi @berenger-eu ,

Thank you for the reply and resolving the issue. I am able to compile the package.
I tried running the test for the package and it has raised following issue, posted on the github page of the code.
Please let me know if this test errors are machine specific or something else which can be avoided.

Regards,
Himanshu

[Himscipy]

👋 @berenger-eu, Thank you for fixing the issue I was able to use the library and the functionality. Although I don't find the TOC in the git repo, which I am able to see on the gitlab.inria.fr so please check.

Thank you @danielskatz for providing the opportunity to review the work, I am done with my review.

Regards,
Himanshu Sharma

[berenger-eu]

Dear @pitsianis @sarats, I just would like to know if I missed something and if the ball is on my side or not. If that is the case, please could you point me to what I am supposed to do? Otherwise, I will simply wait for comments. Thanks a lot.

[sarats]

@berenger-eu Himanshu has done a comprehensive review and got to some of the documentation issues I ran into.

Just a couple of general questions/comments about computational approach:

• What are your plans for execution on GPUs? Note that GPU implementation is not necessary and outside the scope of the JOSS paper.
• I see that your testbed has Intel Cascade Lake (https://ark.intel.com/content/www/us/en/ark/products/192443/intel-xeon-gold-6240-processor-24-75m-cache-2-60-ghz.html) processors. Wondering what kind of performance you would get out of Intel compilers if they are accessible?
• You have identified some factors that explain the OpenMP scalability degradation beyond 8 threads. Additionally, there is a distinct divergence in performance b/w OpenMP and SPETABARU even at 2 threads. Could you add an explanation (difference in load balancing strategy or ...)?

[sarats]

A minor comment:

I have noticed related work at Supercomputing conference series over the years and your related work/references didn't appear to capture those.
I'm not affiliated with the authors, just pointing out one from my preliminary search...
https://ieeexplore.ieee.org/abstract/document/6375552

So, adding some discussion on how your approach differs from those would be great.

[berenger-eu]

@whedon generate pdf

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[berenger-eu]

Dear @sarats thanks for your comments.
I answer each point here.

What are your plans for execution on GPUs? Note that GPU implementation is not necessary and outside the scope of the JOSS paper.

This is our next objective. However, the FMM has an irregular data structure (the tree) and we would like to stay at a high-level (C++ and not C). Consequently, most runtime systems are not adapted. Therefore, we plan first to improve our SPETABARU runtime system to support GPUs (this is a wok in progress) and second to validate this feature on TB-FMM.
For instance, creating a kernel to compute the interactions between the particles (P2P) is not difficult on GPUs/CUDA, but providing an interface and the machinery to let the developer using GPU easily with the task-based method is the challenge.

I see that your testbed has Intel Cascade Lake (https://ark.intel.com/content/www/us/en/ark/products/192443/intel-xeon-gold-6240-processor-24-75m-cache-2-60-ghz.html) processors. Wondering what kind of performance you would get out of Intel compilers if they are accessible?

I did a small test and it crashes with Intel compiler. I did a small investigation, and it seems that Intel compiler is doing something wrong. If I leave the tasks but disable them it crashes (by adding "if(0)" at the end of the pragma => so no tasks but synchronous execution with all the Intel OpenMP machinery). If I simply comment the "#pragma" (so no tasks and no OpenMP machinery) it works. So I conclude that the Intel OpenMP has a bug, which certainly comes from the use of C++ in my opinion. Therefore, I will add in the README that Intel compiler is currently not supported.

You have identified some factors that explain the OpenMP scalability degradation beyond 8 threads. Additionally, there is a distinct divergence in performance b/w OpenMP and SPETABARU even at 2 threads. Could you add an explanation (difference in load balancing strategy or ...)?

Both implementations are very similar, they have the same: number of tasks, tasks and priorities of the tasks. The two differences are the type of data accesses and how the runtime systems are implemented (the potential overhead coming from the tasks/dependencies management of the runtime system). But GNU OpenMP implementation is usually quite good in this aspect. Therefore, the only difference comes from the non-commutative access with OpenMP.
I made it clearer in the document: from 2 to 8 threads the OpenMP performance is only impacted by the non-commutative accesses (so not really a problem from OpenMP but just a missing feature).
As a side note, this is not surprising because with the "write" data access the order that is enforced limits the degree of parallelism (it would be possible to improve OpenMP implementation by inserting tasks following a round-robin fashion, but we prefer to let it as it is and wait that OpenMP supports mutex-inout).

I have noticed related work at Supercomputing conference series over the years and your related work/references didn't appear to capture those. I'm not affiliated with the authors, just pointing out one from my preliminary search... https://ieeexplore.ieee.org/abstract/document/6375552 So, adding some discussion on how your approach differs from those would be great.

Yes, I know the works from G. Biros or also R. Yokota, which are great. The main difference is that we use the task-based parallelization (as we did for ScalFMM). The advantages of task-based are to avoid synchronizations, delegate load-balancing to the scheduler, and thus should be - in theory - more generic against the hardware or the underlying computation kernel. On the other hand, the gap to create a large-scale application that could run on thousands of nodes is huge and depends on the capability of the runtime system (see https://ieeexplore.ieee.org/abstract/document/8226789 for more easy work on regular linear algebra method). Whereas, classical parallelization schemes (OpenMP/CUDA with manul transfers/MPI with manual communications) has proven its potential. Therefore, I will add a single sentence to point to these references.

You can see the commits that changed the paper here:
https://gitlab.inria.fr/bramas/tbfmm/-/commit/93f9fd3930d5a4282ccdfed475c81e4447cdd719
https://gitlab.inria.fr/bramas/tbfmm/-/commit/2ea97723dd4477101aed0c1f55f7d9fc5367776f
https://gitlab.inria.fr/bramas/tbfmm/-/commit/e43277e6600ddc9d0938863e1b9e6fe0284fdbb0

Best regards.

[poulson]

Hi Nikos (@pitsianis) 👋

It's been a few weeks since the other two reviewers completed their reviews and it looks like you have been busy with other work. Would you mind circling back to let the author know if there are any desired changes so that we can procede?

With Thanks,
Jack

[berenger-eu]

Dear @pitsianis
Thanks I updated the readme and the wiki.

[poulson]

@whedon generate pdf

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[poulson]

@whedon check references

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/0021-9991(87)90140-9 is OK
- 10.1109/aps.1994.407723 is OK
- 10.1109/tpds.2017.2697857 is OK
- 10.7717/peerj-cs.183 is OK
- 10.1155/2017/5482468 is OK
- 10.1002/cpe.3723 is OK
- 10.1137/130915662 is OK
- 10.4208/cicp.020215.150515sw is OK
- 10.1016/j.enganabound.2012.07.004 is OK
- 10.1063/1.472369 is OK
- 10.1063/1.468354 is OK
- 10.1063/1.2194548 is OK
- 10.1016/j.cam.2003.12.011 is OK
- 10.1137/18M1173599 is OK

MISSING DOIs

- 10.1145/1654059.1654118 may be a valid DOI for title: A massively parallel adaptive fast-multipole method on heterogeneous architectures
- 10.1007/978-3-319-10705-9_60 may be a valid DOI for title: FMMTL: FMM Template Library A Generalized Framework for Kernel Matrices

INVALID DOIs

- None

[poulson]

Hi @berenger-eu (:wave:) :

In addition to fixing the above two missing DOIs flagged by whedon, I recommend the following minor grammatical improvements to the article:

"In addition, the algorithm also requires that the kernel to approximate far interaction exists as providing an approximation kernel for a physical equation can be challenging" =>
"The algorithm also requires that the kernel to approximate far interactions exists; providing an approximation kernel for a physical equation can be challenging"

I recommend making the following a separate sentence: ", however, when the algorithmneeds cells outside of the boundaries, it selects cells at the opposite side of the simulation box."

"This method needs nothing more than a FMM kernel," =>
"This method needs nothing more than an FMM kernel,"

"We remind that OpenMP 4.5 does not support mutexinout, the commutative data access." =>
"We recall that OpenMP 4.5 does not support mutexinout for commutative data access."

"In our case, the resulting degree of parallelism is then too limited to feed all the cores available" =>
"In our case, the resulting degree of parallelism is then too limited to feed all available cores"

I recommend making the following phrase a separate sentence rather than being separated with a comma: ", instead, we will wait that the OpenMP library implementations supportmutexinout"

"test cases: two simulations of one and ten millions of particles randomly distributed in a cube." =>
"test cases: two simulations, of one and ten million particles, randomly distributed in a cube."

[berenger-eu]

@whedon check references

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/0021-9991(87)90140-9 is OK
- 10.1109/aps.1994.407723 is OK
- 10.1109/tpds.2017.2697857 is OK
- 10.7717/peerj-cs.183 is OK
- 10.1155/2017/5482468 is OK
- 10.1002/cpe.3723 is OK
- 10.1137/130915662 is OK
- 10.4208/cicp.020215.150515sw is OK
- 10.1016/j.enganabound.2012.07.004 is OK
- 10.1063/1.472369 is OK
- 10.1063/1.468354 is OK
- 10.1063/1.2194548 is OK
- 10.1016/j.cam.2003.12.011 is OK
- 10.1145/1654059.1654118 is OK
- 10.1137/18M1173599 is OK
- 10.1007/978-3-319-10705-9_60 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[berenger-eu]

@whedon generate pdf

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[berenger-eu]

Dear @poulson

Thanks a lot for the comments!
I updated the paper accordingly.

[poulson]

Thank you, @berenger-eu! Would you mind creating a DOI on Zenodo and sharing it here?

[berenger-eu]

@poulson Yes, here it is https://doi.org/10.5281/zenodo.4302384

[poulson]

@whedon set 10.5281/zenodo.4302384 as archive

[whedon]

OK. 10.5281/zenodo.4302384 is the archive.

[poulson]

@whedon accept

[whedon]

Attempting dry run of processing paper acceptance...

[whedon]

👋 @openjournals/joss-eics, this paper is ready to be accepted and published.

Check final proof 👉 openjournals/joss-papers#1954

If the paper PDF and Crossref deposit XML look good in openjournals/joss-papers#1954, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true

[berenger-eu]

I had a quick look, seems OK.

[kyleniemeyer]

@whedon accept deposit=true

[whedon]

Doing it live! Attempting automated processing of paper acceptance...

[whedon]

🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦

[whedon]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

0. Check final PDF and Crossref metadata that was deposited 👉 Creating pull request for 10.21105.joss.02444 joss-papers#1956
1. Wait a couple of minutes to verify that the paper DOI resolves https://doi.org/10.21105/joss.02444
2. If everything looks good, then close this review issue.
3. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? Notify your editorial technical team...

[kyleniemeyer]

Congrats @berenger-eu on your article's publication in JOSS!

Many thanks to @pitsianis, @Himscipy, and @sarats for reviewing this, and @poulson for editing.

[whedon]

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.02444/status.svg)](https://doi.org/10.21105/joss.02444)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.02444">
  <img src="https://joss.theoj.org/papers/10.21105/joss.02444/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.02444/status.svg
   :target: https://doi.org/10.21105/joss.02444

This is how it will look in your documentation:

We need your help!

Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: https://joss.theoj.org/reviewer-signup.html
• Making a small donation to support our running costs here: https://numfocus.org/donate-to-joss