[REVIEW]: Julia for HPC: In Situ data Analysis with Julia for Climate Simulations at Large Scale

[whedon]

Submitting author: @ltang85 (Li Tang)
Repository: https://github.com/ltang85/In-Situ-data-Analysis-with-Julia-for-Climate-Simulations-at-Large-Scale
Branch with paper.md (empty if default branch):
Version:
Editor: @luraess
Reviewers: @simonbyrne, @williamfgc
Archive: Pending

Status

Status badge code:

HTML: <a href="https://proceedings.juliacon.org/papers/d9888b7336413fb1ab432e8d864fd6cb"><img src="https://proceedings.juliacon.org/papers/d9888b7336413fb1ab432e8d864fd6cb/status.svg"></a>
Markdown: [![status](https://proceedings.juliacon.org/papers/d9888b7336413fb1ab432e8d864fd6cb/status.svg)](https://proceedings.juliacon.org/papers/d9888b7336413fb1ab432e8d864fd6cb)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@simonbyrne & @williamfgc, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @luraess know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @simonbyrne

Conflict of interest

• As the reviewer I confirm that I have read the JuliaCon conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• I confirm that I read and will adhere to the JuliaCon code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Authorship: Has the submitting author (@ltang85) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Paper format

• Authors: Does the paper.tex file include a list of authors with their affiliations?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
• Page limit: Is the page limit for full papers respected by the submitted document?

Content

• Context: is the scientific context motivating the work correctly presented?
• Methodology: is the approach taken in the work justified, presented with enough details and reference to reproduce it?
• Results: are the results presented and compared to approaches with similar goals?

Review checklist for @williamfgc

Conflict of interest

• As the reviewer I confirm that I have read the JuliaCon conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• I confirm that I read and will adhere to the JuliaCon code of conduct.

General checks

• Repository: Is the source code for this software available at the repository url?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Authorship: Has the submitting author (@ltang85) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the function of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Paper format

• Authors: Does the paper.tex file include a list of authors with their affiliations?
• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?
• Page limit: Is the page limit for full papers respected by the submitted document?

Content

• Context: is the scientific context motivating the work correctly presented?
• Methodology: is the approach taken in the work justified, presented with enough details and reference to reproduce it?
• Results: are the results presented and compared to approaches with similar goals?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @simonbyrne, @williamfgc it looks like you're currently assigned to review this paper 🎉.

⚠️ JOSS reduced service mode ⚠️

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

⭐ Important ⭐

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/JuliaCon/proceedings-review) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/JuliaCon/proceedings-review:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Failed to discover a Statement of need section in paper

[whedon]

Wordcount for paper.tex is 6545

[whedon]

Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.02 s (709.2 files/s, 331315.9 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
TeX                              9            425           1744           4676
Ruby                             1              8              4             45
YAML                             1              1              0             43
Markdown                         1             12              0             29
TOML                             1              2              0              8
Julia                            2              3              1              7
-------------------------------------------------------------------------------
SUM:                            15            451           1749           4808
-------------------------------------------------------------------------------


Statistical information for the repository '6508ac24f7eee7b98ab151a5' was
gathered on 2023/10/12.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Li Tang - LANL                   3           114             57          100.00

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Li Tang - LANL               57           50.0          0.0                7.02

[whedon]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1137/141000671 is OK
- 10.1287/opre.38.5.911 is OK
- 10.1175/JCLI3996.1 is OK
- 10.5194/gmd-9-1937-2016 is OK
- 10.1175/JCLI-D-16-0465.1 is OK
- 10.1145/3144769.3144778 is OK
- 10.1109/LDAV.2011.6092322 is OK
- 10.2312/EGPGV/EGPGV11/101-109 is OK
- 10.1145/1383529.1383533 is OK
- 10.1177/1094342020935991 is OK
- 10.11578/E3SM/dc.20180418.36 is OK
- 10.1111/j.1467-8659.2011.01964.x is OK
- 10.1109/CLUSTER.2018.00036 is OK
- 10.1109/TVCG.2014.2346458 is OK
- 10.1175/JCLI-D-16-0465.1 is OK
- 10.1109/TVCG.2015.2467411 is OK

MISSING DOIs

- 10.1029/2018jd028927 may be a valid DOI for title: Parametric sensitivity and uncertainty quantification in the version 1 of E3SM atmosphere model based on short perturbed parameter ensemble simulations
- 10.1109/drbsd754563.2021.00009 may be a valid DOI for title: TributaryPCA: Distributed, Streaming PCA for In Situ Dimension Reduction with Application to Space Weather Simulations
- 10.1036/1097-8542.757451 may be a valid DOI for title: Middle atmosphere dynamics
- 10.1109/smartgridcomm.2016.7778841 may be a valid DOI for title: Smartsim: A device-accurate smart home simulator for energy analytics
- 10.1145/3426462.3426465 may be a valid DOI for title: Chimbuko: A workflow-level scalable performance trace analysis tool
- 10.1175/jcli-d-15-0835.1 may be a valid DOI for title: Detection and attribution of changes in extreme temperatures at regional scale
- 10.1007/978-3-540-30218-6_19 may be a valid DOI for title: Open MPI: Goals, concept, and design of a next generation MPI implementation
- 10.2139/ssrn.3287769 may be a valid DOI for title: The Decline of Computers as a General Purpose Technology: Why Deep Learning and the End of Moore’s Law are Fragmenting Computing
- 10.1145/3490138.3490142 may be a valid DOI for title: In Situ Climate Modeling for Analyzing Extreme Weather Events
- 10.1175/1520-0442(2001)014<1959:codaet>2.0.co;2 may be a valid DOI for title: Characteristics of daily and extreme temperatures over Canada
- 10.5194/ascmo-2-79-2016 may be a valid DOI for title: Estimating changes in temperature extremes from millennial-scale climate simulations using generalized extreme value (GEV) distributions
- 10.1137/080731992 may be a valid DOI for title: Communication-optimal parallel and sequential QR and LU factorizations

INVALID DOIs

- https://doi.org/10.1029/2018MS001603 is INVALID because of 'https://doi.org/' prefix
- https://doi.org/10.1029/2020RG000708 is INVALID because of 'https://doi.org/' prefix

[simonbyrne]

@luraess the links to the conflict of interest policy and code of conduct don't work.

[luraess]

@simonbyrne thanks for flagging, while we work on a fix, you can access those here https://juliacon.github.io/proceedings-guide/reviewer/#review_guidelines_and_process

[whedon]

👋 @simonbyrne, please update us on how your review is going (this is an automated reminder).

[whedon]

👋 @williamfgc, please update us on how your review is going (this is an automated reminder).

[luraess]

@simonbyrne @williamfgc Human asking about the status of the revision? Ideally, we can try to get this done in the coming weeks (2 ideally). Thanks so much for your contribution!

[simonbyrne]

I'm on it, will update later this week.

[williamfgc]

Same here, I did a quick pass, but I need to go through it more carefully.

[luraess]

@simonbyrne @williamfgc also let me know if you still have issue checking tick-boxes.

[simonbyrne]

This was an interesting application of embedding Julia inside another program for in situ analysis. Although I am not able to assess the scientific validity of the analysis (which seems incomplete), from a technical perspective, this is an important problem: as simulations reach higher resolutions, it becomes more expensive and difficult to store all the data for post-processing analysis, thus it is helpful if such analysis can be done while the simulation is running ("in situ").

Overall, I thought the paper was reasonable, but the exposition could be improved in a few places.

There is no linked code, (other than the TributaryPCA.jl package, and this doesn't meet most of the above criteria, such as documentation), so I can't really check off those relevant boxes. While the code may not strictly be required, it is a shame as I think this could be of interest to others who are attempting something similar. I would encourage the authors to make those available if possible.

Specific comments

3.3 C Interface

It is also important to conduct garbage collection
(GC) in this C interface for every time step. This is because arr
and p_arr in the code snippet are allocated and then deallocated
in E3SM for every time step, and the Julia runtime is not aware of
this and hence needs explicit GC

This is not what the C code in code 2 is doing: as described in https://docs.julialang.org/en/v1/manual/embedding/#Memory-Management, the JL_GC_PUSH3 macro “roots” the variables so they are not collected, and JL_GC_POP simply removes that root (so they can be cleaned up by the garbage collector). The Julia garbage collector runs automatically (unless it is disabled, but this is not described in the paper)

3.5 Noninvasive Compilation

Some more details could be useful here: how did you load the Julia code? (e.g. did you make use of a precompiled system image?) How did you switch between (or switch off) the in situ analysis codes? Again, having the actual code available could make this easier.

4.1 Sudden Stratospheric Warming (SSW)

I'm not sure Algorithm 1 is correct: shouldn't it reset the counter to 0 if "global zonal mean zonal wind >= 0"? If it is negative for 20 days, does that count as 2 events?

There isn't much detail on the actual GEV fitting, and I found it difficult to understand. Is it fitting a different GEV distribution at every point? What is the prior distribution used for the GEV parameters? Some more details on this would be helpful (as well as linking to the code).

4.2 TributaryPCA

While I understand the problem it is trying to solve, I found it difficult to understand algorithm, and the Code 4 block is somewhat unclear:

• What is X_par? Is it a matrix or a vector, and what are the dimensions?
• What is the initial value (and size) of alpha and V_par?
• Some lines are written using in-place allocation free functions (e.g. mul!), yet not others. There are several calls to copy which seem completely unnecessary.

(I don't have access to the TributaryPCA paper, so couldn't look it up there either).

I would suggest either having a mathematical exposition of the algorithm, and/or making the code a bit more readable.

Is the data centered (i.e. do you subtract the mean)? The lack of red in Fig 3, PC1 suggests that it isn't, but that isn't clear. If you aren't then the interpretation changes slightly.

5.2 In Situ SSW Detection and Characterization Results

I don't quite understand this: it doesn't actually analyse the SSW case? It just seems to be an illustration of how one might do an analysis.

5.3 PCA Results

The details are a bit lacking. Was the data collected every time step? Or is it based on daily snapshots or averages? How long was the simulation?

I disagree with the interpretation of figure 3: you can't really assign "warmer vs colder" to principal components (the sign of eigenvectors doesn't matter). My interpretation of the figures is that:

• PC1: the lack of red suggests the data hasn't been centered (or is centered around the equatorial temperature?). As a results it is just explaining that "poles are colder than the equator"
• PC2 and PC3: these look to be roughly the first Fourier components by longitude, i.e. are some orthogonal variation of sin(long) and cos(long): my guess is that they're just capturing the daily temperature changes as the sun rotates around the earth? Can you plot the corresponding eigenvalues?

While it's useful that you can capture this, it's not particularly scientifically interesting.

5.4 Performance

The SSW seems to have negligible overhead (which is not surprising since it is evaluated infrequently), so there is not much to say about the analysis.

It could be helpful if you could add the number of MPI ranks to Table 1?

I'd be interested to know more about the performance bottlenecks of the PCA code? Which parts were the most expensive? Did the Julia garbage collector cause problems with scaling?

Minor comments

• Should “in situ” be italicized? I don't know if there is a style guide.
• The formatting of the references is somewhat messed up (e.g. many abbreviations are uncaptialized), and many DOIs are missing (e.g. the TributaryPCA one).
• Since you make use of it, it would be nice if you could cite the MPI.jl paper (https://github.com/JuliaParallel/MPI.jl#citation) [disclaimer: I’m one of the authors, but it is the only software mentioned which is uncited]
• Slurm appears both capitalized and uncapitalized.

[williamfgc]

Review: Julia for HPC In Situ Data Analysis with Julia for Climate Simulations at Large Scale

The work present the implementation of an “in-situ” framework with components written in Julia coupled with the well-known E3SM framework model. Two main post-processing tasks are evaluated: Sudden Stratospheric Warming (SSW) and principal component analysis (PCA). SSW is a smaller component while PCA is more computationally intensive due to the required Cholesky decomposition execution. I must admit I am not a doing expert in E3SM, but I can appreciate the computational effort in plugging Julia into an existing framework for a novel application.

Computational experiments show that in-situ provides reasonable overheads for a larger number of processing elements (PEs) usage per node.

Overall, it’s a straight-forward application of how Julia can be added to an existing framework and add value for post-processing tasks.

Major points:

1. Make the abstract more to the point for this work. Write in 250 words what was done for this paper and is unique. It reads more like an introduction. Please see your references for examples for self-contained abstracts
2. The introduction needs fewer technical details as they can go in the background section. Instead, explain why the “challlenges” in the second paragraph are actual challenges. It needs a more high-level explanation of why Julia is chosen over current alternatives earlier in the introduction. Provide the motivation, hypothesis, etc. Also, describe the outline of the paper: Section 2 describes X, Section 3 introduces Y, etc. That way the reader has a good idea what to expect.
3. Explain the motivation for in-situ analysis: is it data bottlenecks? Is it the only way this could be achieved? What are the potential trade-offs? JIT, back end support, ecosystem maturity etc.
4. Overhead in Figures 4 and 5 results is unclear or not noticeable, perhaps use a different scale or plot style
5. Please mention efforts in SCORPIO, as it’s been an effort to add more I/O backend to E3SM. It would be good to compare with existing in-situ methods.
6. One major question is not described, does the current data consumption pattern is independent of the simulation output? Meaning there is some buffer (near real-time) or this is not required as the post-processing is almost negligible.
7. One thing it’s not clear is what’s the quantified advantage over just storing data and doing post-processing. Is this only possible using “in-situ”?
8. Also, add codes available for reproducibility purposes.
9. The motivation for SSW is unclear. It's mostly "there is no overhead" type of case, but how does it compare with using disk directly and not in-situ?

Minor points:

1. Spell out acronyms on first use (e.g. R&D, EAM) starting in the introduction section.
2. LLVM is no longer low-level virtual machine
3. The phrase: “Julia can couple with C/C+++/Fortran applications that are naturally supported by LLVM.” I think it is not completely accurate. Julia can interoperate with C and Fortran libraries not necessarily compiled or “supported” by LLVM
4. ADIOS2 is mentioned but the cited reference is for an earlier ADIOS work. Please cite the appropriate ADIOS2 reference.
5. Related to 4. I’d highly encourage citing papers related to packages used (e.g. MPI.jl)
6. If possible, add a table to describe the hardware characteristics
7. Improve Table 1 caption PE Layout is also the 1st column, maybe add MPI information

[luraess]

Thanks for handing in your revision @simonbyrne @williamfgc . IMPORTANT Please go through the tick-box list and check all points that are addressed leaving the remaining as "to-dos". All tick-boxes will have to be checked upon publication.

[luraess]

@ltang85 now that the 2 reviewers gave you feedback, please address their comments as soon as possible. Also, please take a look at #134 (comment) as you'd need to fix those references as well.

[luraess]

@ltang85 just checking in about the status of the revisions, given that you've got feedback since one month ago. Please try to address the issues raised by the reviewers in the coming weeks, and do not hesitate to reach out if there is anything we can help you with. Thanks.

[ltang85]

@luraess I am still working on it and should be able to address all the comments in 2 weeks. Thanks.

[luraess]

@luraess I am still working on it and should be able to address all the comments in 2 weeks. Thanks.

Thanks - take your time. Was just checking in about progress.

[luraess]

@ltang85 seems there was no activity update for a while here. Can we get the process to a final stage in the coming week?

[ltang85]

@luraess sorry for the delay, I will try to finish it by next week.

[luraess]

Excellent. We would like to wrap up the previous submissions ideally in the coming 1-2 weeks and then open the system for JuliaCon23.

[ltang85]

@luraess Hi, here is the update: we have addressed most of the reviewers' comments but will need a few more days for the rest of the comments.

[luraess]

Great, thanks @ltang85 for the update and looking forward to finalising your submission!

[ltang85]

@whedon generate pdf

[editorialbot]

My name is now @editorialbot

[ltang85]

@editorialbot generate pdf

[editorialbot]

⚠️ An error happened when generating the pdf.

[ltang85]

@luraess I have addressed all the comments and uploaded the revised version. However, I failed to generate the pdf here. Will fix it tomorrow or Tuesday.

[luraess]

Thanks! Seems some figure is missing (https://github.com/JuliaCon/proceedings-papers/actions/runs/8683268959/job/23809007037#step:3:543).

[ltang85]

@editorialbot generate pdf

[editorialbot]

⚠️ An error happened when generating the pdf.

[ltang85]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[ltang85]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[ltang85]

@luraess I have fixed the pdf issues and it is working now. I have also uploaded a response letter adressing all the reviewers' comments in the paper folder. Thanks.

@simonbyrne @williamfgc Thank you very much for reviewing this paper and provding the comments.

[luraess]

@ltang85 Thanks for submitting the revised version of the manuscript. I am checking it now and will let you know if any further changes are needed before we can accept the manuscript.

@simonbyrne @williamfgc If you find some time, I'd like to get your general feedback on whether the author addressed your suggestions in a way you are happy with. Also, please take some time to thick the boxes in here #134 (comment) .

Thanks!

[luraess]

Bump @simonbyrne @williamfgc - Thanks for swiftly verifying ☝️ such that we can finalise the publication! 🙏

[williamfgc]

@luraess I couldn't check the boxes in #134 (comment)

[luraess]

@luraess I couldn't check the boxes in #134 (comment)

I recall now this issue we had already at the beginning - let's leave that out. Could you however briefly assess whether the revised version of the manuscript draft adresses your comments and suggestions @williamfgc ? Thanks!

[williamfgc]

I just spotted typos: e.g. consqeuent...please proof-read. Other than that, my comments were addressed. Thanks!

[ltang85]

@williamfgc @luraess

Thanks for the comments. We just proof-read it again and have fixed the typo you mentioned and a few other typos.

[luraess]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[luraess]

@ltang85 Thanks for addressing the proof reading. While in the process of accepting your submission, I bumped on the README you link to from your paper regarding the steps to reproduce your research https://github.com/ltang85/In-Situ-data-Analysis-with-Julia-for-Climate-Simulations-at-Large-Scale/tree/main/src .

Please format this README using standard markdown syntax, in order to clearly separate code blocks from plain text. Use heading hierarchy to structure the content as well, and please remove the redundant horizontal lines. Once this last bit is fixed, we could proceed with final publication. Thanks!