Further improve GSD performance.

[joaander]

Description

Add write buffers to further improve GSD performance.

Motivation and context

See glotzerlab/gsd#237.

Most users will not need to change the write buffer size. I make the API available so that users wishing to push the limits can increase the buffer size.

How has this been tested?

• Existing unit tests pass after adding needed calls to flush().
• Will report on performance tests on Frontier.

TODO:

• Test and update tutorial notebooks as needed.

Change log

Added:

* ``hoomd.write.GSD.maximum_write_buffer_size`` - Set the maximum size of the GSD write buffer.
* ``hoomd.write.GSD.flush`` - flush the write buffer of an open GSD file.
* On importing ``hoomd``, install a ``SIGTERM`` handler that calls ``sys.exit(1)``.

Checklist:

• I have reviewed the Contributor Guidelines.
• I agree with the terms of the HOOMD-blue Contributor Agreement.
• My name is on the list of contributors (sphinx-doc/credits.rst) in the pull request source branch.

[joaander]

Frontier performance (Orion) with 1 rank writing frames with N positions:

• Default buffer size (64 MiB) (356 frames/second)
  • N=64000: 296 MiB/s
  • N=1000000: 188 MiB/s
• 256 MiB write buffer:
  • N=64000: 344 MiB/s (480 frames/second)
  • N=1000000: 227 MiB/s
• 1 byte buffer for comparison to Improve write.GSD and get_snapshot performance. #1538 (comment)
  • N=64000: 10.6 MiB/s
  • N=1000000: 98.2 MiB/s

Log-only files show a massive improvement:

• Default buffer size (64 MiB): 31134 frames/second
• 256 MiB write buffer: 32028 frames/second
• 1 byte buffer: 82 frames/second

The limitation here might be the index buffer limit. Perhaps we should re-implement the index buffer limit as a fraction of the write buffer so they scale together. Not that there is anything wrong with a 380x speedup, but there might be room for a little more.

[b-butler]

LGTM