Implements an in-memory buffer for SyncedCollections

[vyasr]

Description

In case readers want to skip to them, benchmarks are at the bottom of the motivation section.
Here's the notebook I used to run the benchmarks (saved as a text file because Github doesn't know how to handle anything else, but just rename it to ipynb and it'll load fine).

This PR implements an alternative to the current buffering strategy used by SyncedCollections (and by the JSONDict in signac master). Rather than serializing and deserializing the data into a JSON-encoded buffer on every operation, a buffering strategy that saves on I/O costs but still has to do a lot of work, it allows all SyncedCollections pointing a given file to directly share their internal data so that any operation on one is transparently reflected by the others. In addition to skipping serialization, this method also avoids the need for recursive updates of nested collections, which can be expensive. In a sense it's no longer true buffering because we're not simulating a write by serializing the data in the same manner that we would if we were truly writing to an intermediate buffer, we're stopping at the data being stored in memory.

Motivation and Context

The current buffering implementation is quite expensive. Although it largely avoids I/O operations, maintains a fully JSON-encoded buffer of all the data associated with existing SyncedCollection objects, which has a few notable benefits relative to the approach in this PR:

1. Data in the buffer can be directly written to disk
2. JSON-encoded data can be hashed to perform relatively cheap equality checks.
3. The total buffer size is trivially the sum of all JSON-encoded bytes in the buffer, so a buffer capacity can be set explicitly to control buffer flushes.

However, this approach also has significant performance implications:

1. Every write operation requires a JSON encoding step
2. Every read requires decoding the encoded JSON
3. Every read requires updating the collection's internal data store to match the one in the buffer, which requires an expensive recursive traversal of the (possibly nested) data structure.

The approach in this PR circumvents these three performance bottlenecks by making all collections synced with a particular file directly share their underlying data attribute (an object of the "base type" that contains the data, e.g. a dict for a SyncedAttrDict). Since all mutable collections in Python are references, any change to one instance is transparently conveyed to others.

There are some tradeoffs with this approach when considering the advantages of the old approach listed above, but they can be mitigated to some extent:

1. While data in the buffer cannot be directly written to disk, this just means that we're deferring the cost of JSON-encoding to when the buffer flush occurs, it does not change the total amount of work to be done.
2. The equality checks are primarily of use as a way to determine whether or not the buffered data has changed from the version on disk, which indicates that it needs to be written. The new approach can't directly check this without performing a dict equality check, which I have not implemented but I would expect to be slow. However, we can use a simpler approach (which is currently implemented) that just checks whether any mutating operation has been called (e.g. setting an element); if the only operations that occur are read operations, then there's no need to write back to disk. The only scenario where this method will be slower than the old approach is in code that sets elements of a collection to the exact original values, which I don't think we need to optimize for. If that use case truly becomes a bottleneck (e.g. in status updates in flow), it's simple enough to just add a conditional check in client code that verifies whether the value will actually be changed, e.g. if value != synced_dict[key]: synced_dict[key] = value.
3. Although we can no longer exactly calculate the buffer size, we can employ various techniques to approximate it. The crudest one that I can think of would just be to assume that the collection's size won't increase by more than some factor -- say an order of magnitude -- during one instance of buffering. We already use stat calls to get file metadata when a collection first inserts into the buffer, so we could just also insert the original file size (which is also provided by os.stat) and then use that as a benchmark for how much can be inserted before a buffer flush is necessary. I'm open to other suggestions for this as well.

The performance payoff is shown in the table below. To get a sense for how much I/O contributes to the performance characteristics of buffered mode, I tested on three separate file systems. I tested both read (d[key]) and write (d[key] = value) operations. To capture the balance between in-memory operations and disk reads/writes, I perform 1000 total operations in each benchmark, distributed over different numbers of files. For example, some benchmarks read 1000 values from the same JSON file, while others read 1 value from 1000 different files. Note that there's some significant variability in the results because I only ran 10 loops in the benchmark, but I think the numbers are accurate enough for the purposes of this PR.

System	Operation	Number of dicts	Elements per dict	Speed multiplier
Mac SSD	setitem	1	1000	24.72
Mac SSD	setitem	10	100	21.79
Mac SSD	setitem	100	10	12.19
Mac SSD	setitem	1000	1	3.97
Mac SSD	getitem	1	1000	182.96
Mac SSD	getitem	10	100	187.33
Mac SSD	getitem	100	10	166.74
Mac SSD	getitem	1000	1	98.47
Linux NFS	setitem	1	1000	16.39
Linux NFS	setitem	10	100	7.58
Linux NFS	setitem	100	10	2.51
Linux NFS	setitem	1000	1	1.33
Linux NFS	getitem	1	1000	133.48
Linux NFS	getitem	10	100	138.15
Linux NFS	getitem	100	10	115.78
Linux NFS	getitem	1000	1	72.95
Linux scratch	setitem	1	1000	22.35
Linux scratch	setitem	10	100	20.48
Linux scratch	setitem	100	10	11.93
Linux scratch	setitem	1000	1	3.25
Linux scratch	getitem	1	1000	143.59
Linux scratch	getitem	10	100	141.50
Linux scratch	getitem	100	10	116.29
Linux scratch	getitem	1000	1	71.02

Types of Changes

• Documentation update
• Bug fix
• New feature
• Breaking change¹

¹The change breaks (or has the potential to break) existing functionality.

Checklist:

• I am familiar with the Contributing Guidelines.
• I agree with the terms of the Contributor Agreement.
• My name is on the list of contributors.
• My code follows the code style guideline of this project.
• The changes introduced by this pull request are covered by existing or newly introduced tests.

If necessary:

• I have updated the API documentation as part of the package doc-strings.
• I have created a separate pull request to update the framework documentation on signac-docs and linked it here.
• I have updated the changelog and added all related issue and pull request numbers for future reference (if applicable). See example below.

[codecov]

Codecov Report

Merging #462 (bb67d6f) into feature/synced_collections (0b2bbcf) will increase coverage by 0.38%.
The diff coverage is 92.03%.

@@                      Coverage Diff                       @@
##           feature/synced_collections     #462      +/-   ##
==============================================================
+ Coverage                       77.85%   78.24%   +0.38%     
==============================================================
  Files                              56       57       +1     
  Lines                            6575     6688     +113     
  Branches                         1232     1251      +19     
==============================================================
+ Hits                             5119     5233     +114     
+ Misses                           1155     1152       -3     
- Partials                          301      303       +2     

Impacted Files	Coverage Δ
...ore/synced_collections/file_buffered_collection.py	94.54% <ø> (+9.09%)	⬆️
...e/synced_collections/memory_buffered_collection.py	90.90% <90.90%> (ø)
signac/core/synced_collections/collection_json.py	98.86% <100.00%> (+0.21%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0b2bbcf...bb67d6f. Read the comment docs.

[bdice]

@vyasr The concept seems sound. I haven’t reviewed the code carefully but your description of the feature and the large speedups indicate this is probably a very good idea. The benchmarks for the slow network filesystems are the most important in my opinion. It makes the penalty of excess system calls most obvious.

[vyasr]

@bdice just to make sure that we're on the same page, this PR does not change the amount of I/O performed at all (except in the one edge case mentioned in point 2 above). The slow nfs benchmarks simply indicate that on nfs I/O costs dramatically outweigh all others to the point where removing JSON-serialization and internal updates only leads to marginal performance improvements. So the other benchmarks are much more representative of how expensive all of the other tasks are on a fast filesystem. On the other hand, the nfs benchmarks indicate the limited impact those changes will have when the filesystem is slow. I'm not sure if that's what you mean by "most important".

[bdice]

@vyasr Ok! Thanks for clarifying, that was helpful.

[bdice]

A few comments. This looks good overall. I find it a little funny that the buffer capacity doesn't have the same meaning as in the other (serialized) buffer type, but there isn't a good solution to that. We'll just need to rely on good documentation (and perhaps avoiding use of the same names to indicate "object count" and "byte count").