IB Particle Mesh Output

[danieljvickers]

Description

Updated IB state writing to write using parallel MPI IO for better write times.

Also added post-processing affects using the ib_state_wrt flag to generate a particle mesh in the output SILO files, which can be used for analysis and smoother plotting in visualization.

Type of change

• New feature
• Refactor

Testing

I added the changes to the 2D_mibm_shock_cylinder example case and plotted the output using the new point mesh for visualization.

Checklist

• I added or updated tests for new behavior
• I updated documentation if user-facing behavior changed

GPU changes (expand if you modified src/simulation/)

• GPU results match CPU results
• Tested on NVIDIA GPU or AMD GPU

[github-actions]

Claude Code Review

Head SHA: 4836c1f

Files changed:

• 8
• examples/2D_mibm_shock_cylinder/case.py
• src/post_process/m_data_output.fpp
• src/post_process/m_start_up.fpp
• src/post_process/p_main.fpp
• src/simulation/m_data_output.fpp
• src/simulation/m_start_up.fpp
• src/simulation/m_time_steppers.fpp
• toolchain/mfc/params/namelist_parser.py

Findings

1. Race condition: serial IB state write from all ranks

File: src/simulation/m_start_up.fpp, src/simulation/m_data_output.fpp

The call site in m_start_up.fpp was changed from:

if (ib .and. proc_rank == 0) call s_write_ib_state_file()

to:

if (ib) call s_write_ib_state_file(t_step)

The proc_rank == 0 guard was removed, so all MPI ranks now call s_write_ib_state_file. When parallel_io is .false., this dispatches to s_write_serial_ib_state, which opens the same path with status='replace' and writes from every rank:

open (newunit=file_unit, file=trim(file_loc), form='unformatted', access='stream', status='replace', iostat=ios)
...
do i = 1, num_ibs
    write (file_unit) ib_buf
end do
close (file_unit)

All N ranks simultaneously truncate and write the same file. The resulting file content is undefined. The initial write at step 0 (call s_write_ib_state_file(0)) has the same problem. The parallel path (s_write_parallel_ib_state) correctly uses MPI_FILE_WRITE_AT with per-rank offsets and is not affected.

Fix: Guard s_write_serial_ib_state with if (proc_rank == 0) internally, or restore the call-site rank guard when parallel_io is false.

---

2. Silo post-process writes duplicate IB bodies from every rank

File: src/post_process/m_data_output.fpp, s_write_ib_bodies_to_formatted_database_file

After the MPI broadcast, all ranks hold the complete set of nBodies IB bodies. All ranks then call:

err = DBPUTPM(dbfile, 'ib_bodies', 9, 3, px, py, pz, nBodies, DB_DOUBLE, DB_F77NULL, ierr)

and s_write_ib_variable (also unguarded by rank):

err = DBPUTPV1(dbfile, ..., data, nBodies, DB_DOUBLE, DB_F77NULL, ierr)

Each rank writes all nBodies bodies to its own local Silo file. The master multimesh (written only by rank 0) lists one entry per rank:

do i = 1, num_procs
    write (meshnames(i), '(A,I0,A,I0,A)') '../p', i - 1, '/', t_step, '.silo:ib_bodies'

VisIt / Silo will interpret this as num_procs independent domains each containing nBodies bodies, yielding num_procs * nBodies duplicate bodies in the visualization.

Fix: Either (a) only rank 0 writes the point mesh to its local file and the master mesh references only that one entry, or (b) distribute bodies across ranks matching the simulation-side s_write_parallel_ib_state round-robin partitioning.

---

3. DB_DOUBLE hardcoded for real(wp) arrays

File: src/post_process/m_data_output.fpp

Both DBPUTPM (point coordinates) and every DBPUTPV1 call in s_write_ib_variable pass DB_DOUBLE as the Silo datatype while the actual arrays are declared real(wp):

err = DBPUTPM(dbfile, 'ib_bodies', 9, 3, px, py, pz, nBodies, DB_DOUBLE, DB_F77NULL, ierr)
...
err = DBPUTPV1(dbfile, ..., data, nBodies, DB_DOUBLE, DB_F77NULL, ierr)

In a --single build wp is 32-bit, but Silo is told to read 64-bit values — corrupting all IB body output. The Silo type constant should track the wp width (analogous to the mpi_p pattern used elsewhere).

[coderabbitai]

Warning

Rate limit exceeded

@danieljvickers has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 16 minutes and 19 seconds before requesting another review.

Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 16 minutes and 19 seconds.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 3388341d-1afe-47e6-b18f-c58767c80f88

📥 Commits

Reviewing files that changed from the base of the PR and between 92585b1 and 4836c1f.

📒 Files selected for processing (10)

• examples/2D_mibm_shock_cylinder/case.py
• src/post_process/m_data_output.fpp
• src/post_process/m_start_up.fpp
• src/post_process/p_main.fpp
• src/simulation/m_data_output.fpp
• src/simulation/m_start_up.fpp
• src/simulation/m_time_steppers.fpp
• tests/7FA04E95/golden-metadata.txt
• tests/7FA04E95/golden.txt
• toolchain/mfc/params/namelist_parser.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[codecov]

Codecov Report

❌ Patch coverage is 71.79487% with 44 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.73%. Comparing base (92585b1) to head (4836c1f).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
src/post_process/m_data_output.fpp	68.67%	21 Missing and 5 partials ⚠️
src/simulation/m_start_up.fpp	6.25%	15 Missing ⚠️
src/simulation/m_data_output.fpp	94.64%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1375      +/-   ##
==========================================
+ Coverage   64.62%   64.73%   +0.11%     
==========================================
  Files          71       71              
  Lines       18407    18527     +120     
  Branches     1516     1523       +7     
==========================================
+ Hits        11895    11993      +98     
- Misses       5555     5573      +18     
- Partials      957      961       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.