Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Save action times on multi-stream CPU celer-sim #1065

Merged
merged 17 commits into from
Dec 21, 2023

Conversation

sethrj
Copy link
Member

@sethrj sethrj commented Dec 14, 2023

When we switched the regression problems to "merge_events": false on CPU, we lost the action-granularity timing results. This restores them. To be consistent with current output, the timing is averaged over the streams.

@sethrj sethrj added enhancement New feature or request app Changes targeting application front ends labels Dec 14, 2023
@sethrj sethrj marked this pull request as ready for review December 14, 2023 12:08
app/celer-sim/Runner.cc Outdated Show resolved Hide resolved
@sethrj
Copy link
Member Author

sethrj commented Dec 18, 2023

I can't reproduce the celer-sim crash locally :(

@sethrj
Copy link
Member Author

sethrj commented Dec 18, 2023

@amandalund Do you have any idea why this could be segfaulting now? I'm at a loss.

@amandalund
Copy link
Contributor

I'll see if I can reproduce it.

@sethrj
Copy link
Member Author

sethrj commented Dec 18, 2023

Also failed to reproduce on wildstyle/openmp/cuda for debug, release, and reldeb :(

@amandalund
Copy link
Contributor

No luck either so far...

@amandalund
Copy link
Contributor

amandalund commented Dec 18, 2023

Ok, I was able to reproduce it with debug assertions on. In the new transporter accessor in the Runner, looks like the transporter isn't allocated:

status: Creating states
status: Celeritas core state initialization complete
status: Warming up
status: Transporting 5 primaries
status: Creating states
status: Creating states
status: Celeritas core state initialization complete
status: Celeritas core state initialization complete
status: Transporting 5 primaries
status: Transporting 5 primaries
/home/alund/celeritas_project/celeritas/app/celer-sim/Transporter.cc:119: error: Exceeded step count of 256: aborting transport loop
/home/alund/celeritas_project/celeritas/app/celer-sim/Transporter.cc:119: error: Exceeded step count of 256: aborting transport loop
/home/alund/celeritas_project/celeritas/app/celer-sim/Transporter.cc:119: error: Exceeded step count of 256: aborting transport loop
/home/alund/celeritas_project/celeritas/app/celer-sim/celer-sim.cc:268: critical: While running input at <stdin>: /home/alund/celeritas_project/celeritas/app/celer-sim/Runner.cc:543:
celeritas: postcondition failed: result
status: Saving output
Failure written to simple-cms-cpu.out.failed.json
fatal: run failed with error 1

@amandalund
Copy link
Contributor

Are there more threads than events in this test?

Copy link
Contributor

@amandalund amandalund left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @sethrj!

@sethrj sethrj enabled auto-merge (squash) December 21, 2023 03:44
@sethrj sethrj merged commit 87e4421 into celeritas-project:develop Dec 21, 2023
19 of 20 checks passed
@sethrj sethrj deleted the action-times branch December 21, 2023 15:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
app Changes targeting application front ends enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants