Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Turn off Statistics (Min/Max) in ADIOS2 backend #831

Merged
merged 1 commit into from
Dec 4, 2020

Conversation

franzpoeschel
Copy link
Contributor

As documented here, ADIOS2 will produce statistics for variables by default. This is a time-intensive procedure and we don't use the information anyway, so we override this and turn it off.

If a user really wants these, they can be turned on again via JSON. The statistics cannot be accessed via openPMD, but some people may be loading data manually via ADIOS2.

Some little time measurements: I ran a single-threaded simulation with PIConGPU, producing per data dump between 14 and 17GB of data (dumping every 50 time steps for 400 steps in total).

  • With statistics: 6 minutes, 31 seconds
  • Without: 6 minutes, 4 seconds

That is a speedup of nearly half a minute.

Also, replace a dead link in the documentation while I'm at it.

Copy link
Member

@ax3l ax3l left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great catch.
I remember we turned them off by default in ADIOS1 as well, and making it a configuration option is splendid! ✨

@ax3l ax3l requested a review from guj December 4, 2020 02:13
@ax3l ax3l merged commit 1d32118 into openPMD:dev Dec 4, 2020
@ax3l
Copy link
Member

ax3l commented Dec 4, 2020

cc @guj just FYI, this might be visible in the next benchmarks as an improvement.

@franzpoeschel franzpoeschel deleted the fix-adios2-turn-off-statistic branch January 28, 2021 13:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants