Skip to content

Minutes

Jump to bottom
Jay Kalinani edited this page Sep 21, 2026 · 39 revisions
September 21, 2026 Attendees: Allen, Jamie, Jay, Maxwell, Helvi, Lucas, Steven, Roland

Minutes: Helvi

Jay: Many milestones done: GPUs, KerrSchild, Integration with ODESolvers, CI testing, AMR (sphere test functional), Subcycling, Weak Rates coupling, multibox.

Still in progress:

  • nurates/WeakRates testing. Likely a bug in nurates in pair production bremsstrahlung reaction, disabled fixes magTOV issues. WeakRates evo gives good agreement with THC (also WeakRates based), symmetric and reasonable small.
  • optimization
  • Science case: e.g. binary neutron star

THC vs nuX:

  • both using WeakRates
  • still a relative difference of O(1) at center in y component of the flux for magTOV
  • in the interior of the TOV star, it seems that flux, F_{y}, is reset at some points in x-z plane,
    • the magnetic fields in the test are different
    • TODO: redo test with the same magnitude of the magnetic field
    • TODO: redo test with zero magnetic field
  • Some major asymmetry seen in y component of the flux for magTOV in x-z plane (not in the x-y plane),
    • asymmetry is also seen along x, when plotting the difference $F(x,y)+F(x,-y)$
  • resetting may be due to flux limiter in Eq.(32) of 2111.14858
    • Jamie suggests to switch off the flux limiter to test if it causes the asymmetry
    • add output for $\kappa_{ave}$ in Eq. (37) as diagnostic

Jay profiling:

  • bottle-neck is NuX opacity calculation, using nurates

  • TODO: redo profiling with WeakRates

  • Jay has been now using IMEX, so reasonable to expect some differences between THC; fall-back to xxx

Audit from Zach / Jay:

August 24, 2026 Attendees: Jamie, Jay, Maxwell, Allen, Yosef, Helvi, Lucas, Steven

Minutes: Maxwell

Jay: Many milestones done: GPUs, KerrSchild, Integration with ODESolvers, CI testing, AMR (sphere test functional), Subcycling, Weak Rates coupling, multibox. Still in progress: nurates/WeakRates testing. Likely a bug in nurates in pair production bremsstrahlung reaction, disabled fixes magTOV issues. WeakRates evo gives good agreement with THC (also WeakRates based), symmetric and reasonable small.

Some major asymmetry seen in y component of the flux for magTOV, bug in the jacobian discovered and fixed in nuX (not THC - Allen), potential race issue that Roland pointed out previously. nuX has multiple bugfixes that THC doesn't have, hard to say concretely if it is nuX or THC that has the issue. Jay has been now using IMEX, so reasonable to expect some differences between THC.

Allen: Further grid asymmetries as IGM does not like including the origin, shift in coordinates compared to nuX. CarpetX staggered grid further complicates.

Jay profiling: nuX can result in a factor 4-5 slowdown with nurates coupling, most expensive thing is the calculation of the opacities, >60% of evolution cost.

July 13, 2026 Attendees: Jamie, Jay, Maxwell, Steven, Yosef

Minutes: Jay

Jay: Magnetised TOV short run works with M1 using constant temperature slice. Magnetised TOV with constant entropy slice collapses in 0.5 ms of evolution. This is due to unphysical opacities; currently debugging the coupling of the code with the nurates library. In parallel, Jay is looking into coupling the code with ODESolvers and subcycling in order to speed up the tests.

Max: Suggested designed a simpler test to test the nurates library coupling. Jay will look into this.

Steve: Suggested he has tests to simplify 3D tests to 1D or 2D geometries and will look into it.

Jamie: Agrees TOV to be an intermediate test before starting BNS runs.

Next meeting to be held in a month (August 10, 2026) since Jamie and Jay will be away for the ATPESC workshop.

May 18, 2026 Attendees: Allen, Jamie, Jay, Lucas, Maxwell, Roland, Steven, Yosef, Roland

Minutes: Jamie

Jay: Kerr "light-bending" test, version with Minkowski background. No sign of checkerboard pattern in Pzz, suggesting that was caused by the boundary. The THC test seems to use "flat" BCs. The CarpetX test used Neumann.

Lucas: The Carpet "flat" boundary condition can be achieved using the CarpetX Dirichlet BC but with interpolation order = 0.

Jay: Simple 2x2 test, checking values at individual points at iteration 1.

Proposal: have a parameter to turn off the hydro evolution for testing.

Jay: Could use some help for doing unit tests in THC to compare values at specific points in THC vs nuX.

Max: We need to all agree on a thornlist and parfile, at least for one test. That would make testing easier.

Allen: There are parfiles for the THC tests inside the THC repository.

May 4, 2026

Attendees: Allen, Jay, Maxwell, Steve, Helvi, Yosef

Minutes: Allen

Jay: Kerr-Schild BH light bending test. Nonzero components of rPij still appear even if coordinates are swapped, or if BH mass is very small. Checkerboard pattern in rPzz in small BH test.

Yosef: Check boundary conditions vs. THC

Allen: Check chi at bad rPij points.

To Do: Get THC_M1 tests running locally, create unit test comparing closure root finding algorithm of THC vs. nuX, compare sphere test at higher resolution.

April 20, 2026 Attendees: Allen, Helvi, Jamie, Jay, Lucas, Maxwell, Roland

Minutes: Maxwell

Jay: Still looking at the Kerr-schild beam test, brought 2D difference plot in rE[0], nuX vs THC. NuX off-diagonal components of the stress tensor or y component of the flux are sizable when they should remain at zero (which they start at everywhere). 2D rE profiles are very similar initially, but differences build up and grow.

Roland: Worth testing directionality, align the beam along different axis and see the result. "Flat" boundary condition might be better suited here. Also, discrepancy between where the energy is injected.

Misc: Beam originates at a boundary point, so the initial error/behavior is coupled to the boundary condition.

Spacetime quantities (lapse) show some difference between THC/nuX, on order 1e-5.

THC KerrSchild test is a true 2D test with 1 point along an axis, differing from nuX true 3D test.

Apr 6, 2026

Attendees: Allen, Roland, Jamie, Jay, Lucas, Maxwell, Steve

Minutes: Jay

Jay: Kerr-Schild BH light bending test. Showed 2D movies of rN, rE, rF, rP, comparing THC vs nuX along with relative error. Off-diagonal components of some of the quantites, which are expected to be zero, are non-zero. Bug likely somewhere here. The data outputs for nuX made sure to be at y=0 slice.

Tried higher order FD as well as implemented GSL root-finder, but results did not change.

Allen: Suggested setting gr-sources to no, and compare with THC.

To do: get Kerr-Schild BH light bending test working, then do TOV test, then do OS collapse test.

Mar 23, 2026

Attendees: Allen, Roland, Jamie, Jay, Lucas, Maxwell, Yosef, Helvi, Steve, Liwei

Minutes: Jamie

Jay: Kerr-Schild BH light bending test. Previously we saw that the energy with the nuX was higher compared to THC. Went back to low-res, trying to match the results of THC with nuX. Still have not found the bug yet. There may be a bug in THC with the calculation of the Jacobians in the source terms, so will try to add this bug into nuX to see if it then gives the same result as THC.

It does appear that the nuX left-most boundary condition is constant in time.

Jamie: Found appropriate references to test of Oppenheimer-Snyder collapse with neutrino radiation, (see https://journals.aps.org/prd/pdf/10.1103/PhysRevD.105.104028 Appendix F and https://journals.aps.org/prd/pdf/10.1103/PhysRevD.40.1858)

To do: get Kerr-Schild BH light bending test working, then do TOV test, then do OS collapse test.

Mar 9, 2026 Attendees: Allen, Roland, Jamie, Jay, Lucas, Maxwell, Yosef

Minutes: Allen

Jay: Shadow test: Std. resolution shadow test shows strong bumps at edge of beam that don't show up in low resolution. Kerr-Schild test: beam is gaining energy as it travels, sensitive to injecting beam in boundary or interior. Interior injection better recreates energy from THC and analytic geodesic solutions.

Allen: Removed syncs until only syncs of fluxes, state vector remain.

Lucas: Suggestion to try presync-only, may have time to couple nuX_M1 with ODESolvers next semester

Group updates to ToDo list, some tasks delegated.

Feb 23, 2026 Attendees: Lucas, Jay, Steve, Roland, Liwei, Yosef

Minutes: Lucas

Pre update topics

  • Jay discussed some of the results of Allen's visit in UIUC. An updated TODO list has been compiled here.
  • Roland asked about using Weak Rates vs New Rates vs Weak lib. Jay clarified that microphysics libraries are a planned runtime parameter in the code, i.e., the code is microphysics agnostic.
  • Liwei asked if the infrastructure from GramX can be lifted. Roland does not think so.

Updates

  • Jay showed us a successful shadow test result.
  • He also mentioned that he and Allen got a successful result for the absorbing/emitting sphere of radiation test.
  • He is currently doing a Kerr-Schild test, where a beam of light gets curved around a BH. He comments that the bending is fine but the energy injection is wrong. He suspects boundary condition issues.
  • The advection test that was giving us trouble is also passing now. Jay suspects the problem was probably due to wrong root finding settings. The results for these tests have neem sent to our mailing list.
  • A few tests from the original THC paper are missing but things are promising.
December 8, 2025 Attendees: Allen, Helvi, Jamie, Jay, Liwei, Maxwell, Roland

Minutes: Maxwell

Bug updates from Jay:

  • Multiple indexing errors in the fluxes have been fixed, still not completely confident.
  • Not sure about Boundary Conditions (radiative/outflow), although shouldn't have a large effect initially, lead into a discussion with Allen.
  • Unpushed changes moving from L/R primitive recon to direct flux recon, along the lines of the THC M1 code, finite differencing.

Allen changes:

  • Added a select BC function, and a sync for (intermediate?) states, found that it had an effect. Roland found this suspicious, sync should either be required (read values are invalid) or a no op (already synced). Explained by a combination of some GFs being constant when they shouldn't be, and GFs initialized to zero, means they are labeled valid everywhere always. Probably want to remove the initialization to zero everywhere, and add the necessary sync statements.

Misc:

  • CarpetX was not designed for multiple time levels in mind, not clear how foo*_p and foo*_p_p work.
  • Add dependencies to nuX interface.ccl.

Next bug fixes:

  • Remove zeroing everywhere of fluxes and other GFs, like RHS, recovering the function of valid/invalid.
  • Further fix flux indexing issues, finalize finite differencing implementation, cell centered fluxes and flux recon.
  • Add dependencies to the interface, and either remove/rename the _p/_p_p variables or justify their use.
  • Properly apply boundary conditions and sync statements.

No meeting 2 weeks from now, 12/29/25.

November 24, 2025 Attendees: Allen, Jamie, Jay, Lucas, Liwei, Maxwell, Roland, Yosef

Minutes: Maxwell

Updates from Jay:

  • Still working on first nuX test, 1D advection. Iterations are being completed, but a strong oscillation is created on the upwind side of the shock, in the Neutrino energy. This feature is not seen on the downwind side of the shock, little to nothing is propagated downwind, with near zero fluxes downwind. This is observed for both minerbo and thin closure options, using FakeRates, without any sources.
  • Allen pointed out the nontrivial component of the stress energy tensor is T_zz/P_zz, additionally the opacities should be all zero, and minerbo closure should reduce to optically thin. A quick investigation revealed that the opacities look to be zero and functioning properly. They also brought up a point of resolution differences between the paper and the nuX test, which is unlikely to be the origin of the issue. Debugging suggestions:
  • Jay proposed a rigorous comparison of the fluxes at early iterations against THC_M1.
  • Jay agrees that more unit tests could be useful here in determining the location of the issue, Johnny has been developing some. Reiterated testing is occurring on nuX "debug" branch and AsterX "frozen" branch.
  • Roland points out that the observed upwind oscillations look like CFL condition violations, and as the test only has 2 cells in x/y and 50-100 in z, the spacing along each axis is very different.
  • Jamie suggests decreasing the amplitude of the discontinuity, along the lines of decreasing the instability, Maxwell suggests increasing the amplitude of the discontinuity to avoid any small amplitude issues, along with reflecting the test for parity/directional preferences.
October 27, 2025 Attendees: Allen, Erik, Jamie, Jay, Lucas, Maxwell, Roland, Steve, Yosef

Minutes: Maxwell

TimeIntegratorStage (T.I.S.) issues:

  • Jay has a temporary hack fix setting to previous iteration value if the stage executes CalcUpdate with T.I.S == 0, able to complete a few iterations of evolution, radiative energy terms appear to be reset to atmosphere shortly after it 0, it 0 values also look incorrect (failing c2p).
  • Roland points out there is a racing condition with tiling and parallelizing if the CalcUpdate routine decrements T.I.S. . Recommends separate (global?) routine which only will decrement T.I.S. . OMP_Single could be used, but could still encounter multiple decrements per thread with tiling. Consensus is to use "OPTIONS:LEVEL", effectively a global routine, to do the decrementing.
  • There currently is not a Global mode in CarpetX and level mode is effectively the same thing, with/without subcycling.
  • In Carpet, level mode calls you once for each level per timestep, while global mode is once per overall?
October 13, 2025 Attendees: Allen, Erik, Helvi, Jamie, Jay, Lucas, Maxwell, Roland, Steve, Yosef

Minutes: Maxwell

Updates from Jay related to calc_closure issue:

  • An issue remains in calc_closure, Jay is pointing us to the one dimensional root solver for Xi, used in the closure function. The Brent solver from the Algo thorn contains an assert that the function must change signs between the bracketed interval, which is failing. Allen brought up that this is only entering from the minerbo closure which isn't necessary for the first test, which is optically thin. Jay says that the closure should fallback to optically thin if minerbo fails, which may not occurring. Jay quickly tested with optically thin closure and the assertion error remains.

  • If the assertion is commented out, there are infinities in the change in the source terms. Jamie points us to a line that gets a step of dt from dt/TimeIntegratorStage, where the stage goes backwards from 2,1 to 0, so there is reason to expect infinite updates in source terms. Allen says that it should not run when the step goes to zero, with a check in the nuX schedule to only timestep when the TimeIntegrator stage is nonzero.

  • Max asked if the function is monotonic, or if there is a possibility that the solution is simply a minimum or a root to machine precision. The consensus seems to be that this is a quartic function, and reasonably monotonic between the initial brackets. Max also suggested that the check for convergence should happen before the assertion for the bracket to change signs, which it seems to do. Steve pointed out that it is slightly inconsistent to have an assert checking the function changes sign on the brackets, along with a different check for convergence.

  • Yosef brings up that given f(a) and f(b) are extremely small, (10^-45), machine error entering the expression for convergence could create problems like multiple false roots, or failing convergence tests when it has converged.

Documetation/ Resources requests:

  • Steve requested an updated thornlist for easy replication, and Jamie asked for links to the papers. Jay points us to the nuX wiki page/ github repo which contains an up to date thornlist and ArXiV links.

  • nuX code could use some documentation linking specific lines of code to specific equations in the paper.

Microphysics:

  • Hardcoded parameter to control the number of ___ from the distribution function, which led to some results dependent on this parameter.
  • nuRates assumed Fermi-Dirac for optically thin and Maxwell-Boltzmann for optically thick, and interpolates between the two (M1). MC you know the distribution function to sampling noise. Difference in a static distribution function vs a separate distribution function that needs to be evolved.
  • Jay says that it would be preferrable if we could have both to see the differences between the two, Allen asks which method do we prefer apriori.
Sept 29, 2025 Attendees: Allen, Erik, Helvi, Jamie, Jay, Lucas, Maxwell, Michail, Roland, Steve, Yosef

Minutes: Maxwell

meetings biweekly

Aims:

  • update on CarpetX
  • assign tasks in parallel so that everyone can work on something

Updates from Jay:

  • Hackathon was very successful, we accomplished all of the bullets on the To-Do list. Jay is now debugging an issue with the Fluxes, via running the first test (advection, velocity jump test). The issue is within the calc_fluxes function call.

Requests from others:

  • Update nuX wiki with links to the papers of interest, additionally the flow chart. Jamie also requested some more instructions for installation and compilation to ensure everyone is on the same page, Helvi suggested the CanudaX repository as a starting point.

Debugging Suggestions:

  • Jamie: Potentially can test more than just the first test (adv. vel. jump) to see if the bugs are consistent across the tests.
  • Lucas: Property unit tests: Unit tests for functions like the Flux or other critical modules.

Microphysics library:

  • Still encountering long compiling times with bns_nurates. Allen is investigating differences between Kokkos and AMReX, specifically looking for the exact spot in CarpetX where the GPU compiler is called. Trying to pin down the large discrepancy in compile times.
  • Roland says looking for "nvcc" brings all things relevant to the nvidia compiler. Also says that individual thorns can be compiled by the Cactus compiler. Suggests trying to compile just this thorn by itself to have a better idea of what just the single thorn is doing.
  • Allen says Liwei has been investigating another microphysics thorn (newlib/weaklib), however he is not exactly ready to jump ship just yet.
  • Discussion that it is indeed worth coupling to a microphysics library and not building it ourselves, due to the complexity and requirement to align with experiment.
  • We began with bns_nurates as it does not require any interpolation, as opposed to tabulated results which may present performance and memory issues.
  • Debugging suggestion from Erik: Worth doing a 1D shocktube test for evaluating the microphysics library performance and accuracy.
  • Opportunity for others to help try and compile, requests for a up to date thornlist so others can replicate, Allen says he will include his most up to date thornlist, along with some tracking for the few issues that come with it. (Issues tab on github suggested).

More Suggestions (Microphysics library related):

  • Erik's suggestion is getting at really reducing the system to a 1 dimensional problem, pure hydro + neutrinos, to get at testing the microphysics library.
  • Allen says he does not think that nurates or related microphysics libraries have been tested in pure one dimension.
  • Perhaps further testing is not necessary at this point, and the microphysics library issues is functioning, it is just compiler optimizations that is driving the time to compile up.
Feb 9 , 2025 Attendees: Allen, Helvi, Jamie, Jay, Lucas, Maxwell, Michail, Freddy, Roland, Yosef

Minutes: Maxwell

Updates from Jay: The 1D advection test shows a reflected wave that persists as resolution increases. Diffusion test still shows some oscillations to the right of the packet, oscillations are bounded. An initial spurious feature/wave is created similarly to the adv. test, which moves left. Seen along multiple axis and resolutions, and periodic/non-periodic grids.

Changing the update procedure from fluxes -> sync -> rhs to x fluxes -> x rhs -> y fluxes -> y rhs had a nontrivial effect, results improved.

Roland: CarpetX scheduled functions called one tile after another, can't call F.D. from within a scheduled function, race condition with other tiles. Testing with one tile/ one grid -> one mpi rank may help decouple scheduling/racing bugs from physics bugs.

A problem with thc_M1_calc_fluxes scheduling. Currently, there are 2 passes, first compute fluxes, second compute rhs. On tile boundaries the function will try to access a neighboring tile, which hasn't necessarily been updated yet, leading to another racing condition.

Michail: Debugging idea: comment out/disable perpendicular fluxes for the 1D tests, remove any possibility of issues from the components which (should) be zero, testing the effect of the perpendicular fluxes.

Sept 29, 2025 Attendees: Jamie, Jay, Max, Helvi, Lucas, Yosef, Steve

Minutes: Jamie

Jay: Kerr lightbending test now working. The issue was that the THC run was done with a fake "2D" Kerr spacetime. The y (perpendicular) component of the fluxes, which should be zero, is now down to order 1e-16 i.e. machine precision. Jay's latest test has a parameter to remove the y component of the metric and the y derivatives, to replicate what was used in the original THC paper.

Helvi: make sure the nuX write up specifies exactly what metric is used in the Kerr lightbending test.

Jay: Also tried to evolve a magnetized TOV star with nuX with real nuX_Rates. For constant initial temperature: neutrino radiation shows non-axisymmetric artifacts. For constant entropy initial data show more of a spherically symmetric neutrino radiation, but Jay is repeating test with a larger simulation domain size.

Previously: nuX follows THC using internal implicit timestepping which is not coupled to the overall CarpetX timestepping and subcycling. That will make it very slow for full BNS mergers. Jay has started to try to use the IMEX support in the odesolvers branch, following the Palenzuela group, using LLMs to help copy it to CarpetX(?). Adding new routines to the existing CarpetX ODESolvers thorn to support implicit timestepping. The M1 in general needs implicit timstepping.

Allen is producing the initial data from the NRPyPlusTOVID thorn (in BNSTools) which produces TOV initial data with a finite temp EOS.

remaining things TO DO: Add subcycling, compare TOV test vs THC, check performance, test on Binary NS merger.

Max: We could check whether the spherical ball test is actually maintaining spherical symmetry appropriately (check with Allen who performed the test).

Jamie: what is the performance vs no neutrinos, or vs the M0 version? (but M0 is definitely not the priority here).

Max: re-ran all the 1D tests vs THC, carefully lining up grids & output times, with order 1.e-6 - 1.e-7 relative error for most of the tests, with machine precision error for the square test. Issues with poisoning when trying to run the 2D test.

Lucas: would be interested to see design doc for the implicit time integration methods / the design for that.

Clone this wiki locally