Feat/gsd with queue #1543

b-butler · 2023-05-02T18:16:56Z

Description

Add a hoomd.write.Deque class which stores a std::deque of logger and trajectory data up to a specified maximum size N. The deque pushes data to the front and pops old data off the back of the deque. The analyze method only collects and removes data as necessary. All writes are triggered by the dump method exposed in Python. The purpose of this class is to allow selective high frequency data storing, by allowing users to determine when to write data.

One foreseen pattern of use would be to create a custom action which composes a Deque object calling dump when the specified conditions have been met.

Motivation and context

To provide a means for high-performance, high-frequency writes in HOOMD.

How has this been tested?

Tests based on the GSD tests were added.

Change log

Added: `hoomd.write.Deque` class for enabling selective high-frequency frame writing.

Checklist:

I have reviewed the Contributor Guidelines.
I agree with the terms of the HOOMD-blue Contributor Agreement.
My name is on the list of contributors (sphinx-doc/credits.rst) in the pull request source branch.

This will more easily allow for derived classes of GSDDumpWriter since they can choose when to write or not.

- Move initFileIO to protected - Add method for querying file initialization

This class provided a triggerable multi frame write.

Also move initFileIO to the constructor. This prevents the need for double caching of the buffer size prior to opening the file.

Ensure that we do not rely on Python's garbage collector to ensure that the file is written after the call to write() ends.

Also check for errors in new GSD API calls.

While this is unexpected, it is required to keep performance high as until the first write for a file occurs, all data is recorded and sorted in a GSDDumpWriter::GSDFrame object. By writing the first frame, we prevent this. However, we then don't know if the first frame should be included in the actual analysis.

All tests including logging tests now pass.

AlainKadar

I just had one suggestion

AlainKadar · 2023-05-09T15:00:02Z

hoomd/md/pytest/test_burst_writer.py

+
+
+@pytest.fixture(scope='function')
+def hoomd_snapshot(lattice_snapshot_factory):


Do we want to call this frame instead of snapshot? Or better to keep consistent with the name of the module? Are there plans to rename this module to frame in the future?

No, these are hoomd.Snapshot objects. I renamed gsd.Snapshot to gsd.Frame to avoid confusion with hoomd.Snapshot.

b-butler · 2023-05-09T19:27:15Z

@joaander The test errored due to timeout which is currently set at 3000 second or 50 minutes but MPI tests finish in 30 seconds on hodges. Also, MPI tests for the neighbor list are erroring for so reason on hodges. Will look into that.

joaander · 2023-05-09T20:02:52Z

@joaander The test errored due to timeout which is currently set at 3000 second or 50 minutes but MPI tests finish in 30 seconds on hodges. Also, MPI tests for the neighbor list are erroring for so reason on hodges. Will look into that.

I can't look at it right now. Maybe later this week. If you can find a configuration that reproduces the deadlock, I usually debug them by 1) Running the tests with mpirun and wait for the deadlock. 2) Attach a debugger to the processes and see where each rank is waiting.

b-butler · 2023-05-12T21:19:42Z

pre-commit.ci autofix

for more information, see https://pre-commit.ci

b-butler · 2023-05-15T19:42:32Z

The deadlocking is solved. I forgot to not call gsd functions on all ranks.

joaander

Thanks! The implementation is very clean. I have a few suggestions to improve the documentation.

joaander · 2023-05-17T13:25:25Z

hoomd/GSDDumpWriter.cc

+                          frame.dihedral_data,
+                          frame.improper_data,
+                          frame.constraint_data,
+                          frame.pair_data);
            }
        }

    // emit on all ranks, the slot needs to handle the mpi logic.
    m_write_signal.emit(m_handle);


If this code was used, it would be a problem for the burst writer. However, this has now been replaced by Logger. I opened #1554 to remove these dead code paths. No need to make any changes in this pull request.

By this, I refer to m_write_signal.emit.

joaander · 2023-05-17T13:27:31Z

hoomd/md/pytest/test_burst_writer.py

+
+
+@pytest.fixture(scope='function')
+def hoomd_snapshot(lattice_snapshot_factory):


No, these are hoomd.Snapshot objects. I renamed gsd.Snapshot to gsd.Frame to avoid confusion with hoomd.Snapshot.

hoomd/write/gsd_burst.py

Co-authored-by: Joshua A. Anderson <joaander@umich.edu>

This behavior is not intuitive but required for good performance.

b-butler · 2023-05-19T20:06:24Z

@joaander I just thought of a potential mitigation of the initial write from the burst writer. We could either

provide a method to write similar to GSD.write
or show how to get this to run at the very beginning of a simulation with an And trigger.

What do you think?

joaander · 2023-05-22T11:02:09Z

Yes, we could encourage users to use GSD.write to write the initial file and then open it in append mode. We could also enforce this behavior by making it an error for BurstWriter to create files or append to files with 0 frames.

Adds option to run immediately upon attaching for those still wishing to use this.

This improve the ability for writers to immediately upon attaching write log data depending on one or more computes or tuners.

joaander

I pushed some documentation revisions directly to the branch. Please review and make any further edits as needed. It looks like you need to merge in conflicts from trunk-major.

I have one comment on parameter naming to think about. Other than that, this is good to merge.

hoomd/write/gsd_burst.py

joaander

Thanks!

b-butler and others added 22 commits April 27, 2023 13:29

refactor: Separate gsd frame writing from analyze.

d6181bb

This will more easily allow for derived classes of GSDDumpWriter since they can choose when to write or not.

refactor: Make GSDDumpWriter more amenable to subclassing

5b5d626

- Move initFileIO to protected - Add method for querying file initialization

feat: Add C++ GSDDequeWriter

64a3b35

This class provided a triggerable multi frame write.

feat: Add Deque writer to Python.

9741ae3

test: Test existing Deque writer functionality.

f4bc01b

refactor: Move writeLogQuantities to C++

3b5fe05

feat: Add logging ability to Deque.

66d0746

refactor: Remove now redundant IO check in Deque

47e0da0

Embed latest gsd implementation.

186d24d

Add GSD.flush

bef3e02

Add necessary flush() calls to gsd unit tests.

4367b13

Add maximum_write_buffer_size to GSD.

3ac9698

Also move initFileIO to the constructor. This prevents the need for double caching of the buffer size prior to opening the file.

Flush on calls to GSD.write.

a8ed1e1

Ensure that we do not rely on Python's garbage collector to ensure that the file is written after the call to write() ends.

Run pre-commit.

75cb42d

Scale index buffer size with write buffer size.

f44a6c0

Also check for errors in new GSD API calls.

Install SIGTERM handler.

a3734ec

Sync gsd.c with gsd package.

9c4cca6

Update migration guide.

192f609

refactor: Make m_handle available to subclasses (GSDDumpWriter).

92e5dfc

fix: Deque.__init__ correctly call base class constructor.

1996e53

test: Update Deque tests.

590ef8e

All tests including logging tests now pass.

b-butler force-pushed the feat/gsd-with-queue branch 2 times, most recently from 46c68d8 to f141d6e Compare May 8, 2023 20:24

b-butler added 2 commits May 8, 2023 17:13

refactor: Rename Deque writer to Burst

54ae6a0

doc: Add Burst to documentation.

021d2c8

b-butler force-pushed the feat/gsd-with-queue branch from f141d6e to 021d2c8 Compare May 8, 2023 21:21

Merge branch 'trunk-major' into feat/gsd-with-queue

42a2bd2

b-butler marked this pull request as ready for review May 8, 2023 21:36

b-butler requested a review from a team as a code owner May 8, 2023 21:36

AlainKadar approved these changes May 9, 2023

View reviewed changes

b-butler added 2 commits May 12, 2023 17:02

fix: Correctly set m_write_initial_frame with MPI

7cd1bcd

test: Fix the burst writer tests.

e1b02e3

pre-commit-ci bot and others added 3 commits May 12, 2023 21:20

[pre-commit.ci] auto fixes from pre-commit.com hooks

59eb771

for more information, see https://pre-commit.ci

fix: Non-MPI GSDDequeWriter m_write_initial_frame setting

377efd5

test: Reduce runtime of Burst writer test

01cae4c

joaander requested changes May 17, 2023

View reviewed changes

b-butler and others added 2 commits May 19, 2023 15:59

doc: Improve Burst writer documentation

87685c6

Co-authored-by: Joshua A. Anderson <joaander@umich.edu>

doc: Add warning about initial Burst write.

9c94979

This behavior is not intuitive but required for good performance.

b-butler added 4 commits May 31, 2023 15:34

refactor: Error by default when using empty file for Burst.

f783db6

Adds option to run immediately upon attaching for those still wishing to use this.

test: Update Burst tests

8a84515

test: Test new write_on_start Burst attribute

2d4fe88

refactor: Move writers to end of Operations._schedule

2e430c7

This improve the ability for writers to immediately upon attaching write log data depending on one or more computes or tuners.

b-butler force-pushed the feat/gsd-with-queue branch from 6eb4630 to 2e430c7 Compare May 31, 2023 19:44

b-butler requested a review from joaander May 31, 2023 20:07

Revise docs.

07c8cbd

joaander reviewed Jun 1, 2023

View reviewed changes

hoomd/write/gsd_burst.py Show resolved Hide resolved

b-butler added 2 commits June 1, 2023 13:55

Merge branch 'trunk-major' into feat/gsd-with-queue

1bc8580

fix: mistake in previous merge

70363b9

b-butler added the validate Execute long running validation tests on pull requests label Jun 2, 2023

b-butler requested a review from joaander June 2, 2023 20:46

joaander approved these changes Jun 5, 2023

View reviewed changes

joaander merged commit fa388ef into trunk-major Jun 5, 2023
72 of 73 checks passed

joaander deleted the feat/gsd-with-queue branch June 5, 2023 16:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat/gsd with queue #1543

Feat/gsd with queue #1543

b-butler commented May 2, 2023

AlainKadar left a comment

AlainKadar May 9, 2023

joaander May 17, 2023

b-butler commented May 9, 2023

joaander commented May 9, 2023

b-butler commented May 12, 2023

b-butler commented May 15, 2023

joaander left a comment

joaander May 17, 2023

joaander May 17, 2023

joaander May 17, 2023

b-butler commented May 19, 2023

joaander commented May 22, 2023

joaander left a comment

joaander left a comment



		@pytest.fixture(scope='function')
		def hoomd_snapshot(lattice_snapshot_factory):

Feat/gsd with queue #1543

Feat/gsd with queue #1543

Conversation

b-butler commented May 2, 2023

Description

Motivation and context

How has this been tested?

Change log

Checklist:

AlainKadar left a comment

Choose a reason for hiding this comment

AlainKadar May 9, 2023

Choose a reason for hiding this comment

joaander May 17, 2023

Choose a reason for hiding this comment

b-butler commented May 9, 2023

joaander commented May 9, 2023

b-butler commented May 12, 2023

b-butler commented May 15, 2023

joaander left a comment

Choose a reason for hiding this comment

joaander May 17, 2023

Choose a reason for hiding this comment

joaander May 17, 2023

Choose a reason for hiding this comment

joaander May 17, 2023

Choose a reason for hiding this comment

b-butler commented May 19, 2023

joaander commented May 22, 2023

joaander left a comment

Choose a reason for hiding this comment

joaander left a comment

Choose a reason for hiding this comment