Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/synced collection/replace jsondict #472

Merged
Merged
Show file tree
Hide file tree
Changes from 40 commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
294b2da
Replace old JSONDict with new BufferedJSONDict.
vyasr Jan 13, 2021
f9acc38
Verify that replacing BufferedJSONDict with MemoryBufferedJSONDict.
vyasr Jan 13, 2021
b0deb4e
Remove largely redundant _reset_sp method.
vyasr Jan 13, 2021
700624e
Remove single-use internal functions in Job to reduce surface area fo…
vyasr Jan 14, 2021
013291c
Move logic from _init into init.
vyasr Jan 14, 2021
3b74e48
Working implementation of statepoint using new SyncedCollection.
vyasr Jan 19, 2021
f93f79b
Remove _check_manifest.
vyasr Jan 19, 2021
fc0ba1d
Expose loading explicitly to remove the need for internal laziness in…
vyasr Jan 19, 2021
12bbdf9
Simplify the code as much as possible by inlining move method and cat…
vyasr Jan 19, 2021
b961308
Improve documentation of context manager for statepoint loading.
vyasr Jan 19, 2021
fd8045d
Replace MemoryBufferedJSONDict in Project for now.
vyasr Jan 19, 2021
82c6aa9
Add documentation of why jobs must be stored as a list in the statepo…
vyasr Jan 19, 2021
474fceb
Address PR comments.
vyasr Jan 20, 2021
ff82868
Merge branch 'feature/synced_collections' into feature/synced_collect…
bdice Jan 20, 2021
8109d39
Merge branch 'feature/synced_collections' into feature/synced_collect…
bdice Jan 20, 2021
52db85e
Add back import.
bdice Jan 20, 2021
985443e
Ensure _StatepointDict is always initialized in constructor.
vyasr Jan 21, 2021
a2f1627
Change _StatepointDict to validate id on load.
vyasr Jan 20, 2021
58e5293
Refactor error handling into _StatepointDict class.
bdice Jan 20, 2021
af907c4
Update docstrings.
bdice Jan 20, 2021
d6e71db
Update comment.
bdice Jan 20, 2021
fba0fab
Fix some docstrings.
vyasr Jan 21, 2021
aafd813
Remove redundant JobsCorruptedError check.
vyasr Jan 21, 2021
44fca05
Rewrite reset_statepoint to not depend on creating another job.
vyasr Jan 22, 2021
e6fac09
Reduce direct accesses of internal attributes and do some simplificat…
vyasr Jan 22, 2021
c48c4e8
Reraise errors in JSONCollection.
vyasr Jan 23, 2021
c3a88cc
Change reset to require a non-None argument and to call _update inter…
vyasr Jan 23, 2021
97db9a0
Add reset_data method to provide clear access points of the _Statepoi…
vyasr Jan 23, 2021
c2a9fa4
Create new internal method for handling resetting.
vyasr Jan 23, 2021
61ef343
Move statepoint resetting logic into the statepoint object itself.
vyasr Jan 23, 2021
4078701
Stop accessing internal statepoint filename attribute directly and re…
vyasr Jan 23, 2021
b2c0696
Make statepoint thread safe.
vyasr Jan 23, 2021
98c19d5
Some minor cleanup.
vyasr Jan 23, 2021
f60a271
Remove now unnecessary protection of the filename key.
vyasr Jan 23, 2021
24378ce
Explicitly document behavior of returning None from _load_from_resource.
vyasr Jan 23, 2021
e70c2f0
Apply suggestions from code review
vyasr Jan 23, 2021
153ee48
Merge branch 'feature/synced_collection/replace_jsondict' of https://…
vyasr Jan 23, 2021
5689aca
Rename SCJSONEncoder to SyncedCollectionJSONEncoder.
vyasr Jan 23, 2021
4692e7f
Only access old id once.
vyasr Jan 23, 2021
3c40dc8
Move lazy attribute initialization into one location.
vyasr Jan 23, 2021
0facff5
Address PR requests that don't cause any issues.
vyasr Jan 24, 2021
22b3ae8
Remove the temporary state point file backup.
vyasr Jan 24, 2021
ab11d7d
Make as many old buffer tests as possible.
vyasr Jan 24, 2021
dfa7c4c
Reset buffer size after test.
vyasr Jan 24, 2021
d71a49c
Last changes from PR review.
vyasr Jan 25, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 2 additions & 2 deletions signac/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,10 @@
from .contrib import get_job, get_project, index, index_files, init_project
from .core.h5store import H5Store, H5StoreManager
from .core.jsondict import JSONDict
vyasr marked this conversation as resolved.
Show resolved Hide resolved
from .core.jsondict import buffer_reads_writes as buffered
from .core.jsondict import flush_all as flush
from .core.jsondict import get_buffer_load, get_buffer_size
from .core.jsondict import in_buffered_mode as is_buffered
from .core.synced_collections.buffered_collection import buffer_all as buffered
from .core.synced_collections.buffered_collection import is_buffered
from .db import get_database
from .diff import diff_jobs
from .version import __version__
Expand Down
4 changes: 2 additions & 2 deletions signac/contrib/errors.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@ class DestinationExistsError(Error, RuntimeError):

Parameters
----------
destination :
The destination object causing the error.
destination : str
The destination causing the error.

"""

Expand Down
8 changes: 7 additions & 1 deletion signac/contrib/hashing.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@
import hashlib
import json

from ..core.synced_collections.utils import SyncedCollectionJSONEncoder
from ..errors import KeyTypeError

# We must use the standard library json for exact consistency in formatting


Expand All @@ -27,7 +30,10 @@ def calc_id(spec):
Encoded hash in hexadecimal format.

"""
blob = json.dumps(spec, sort_keys=True)
try:
blob = json.dumps(spec, cls=SyncedCollectionJSONEncoder, sort_keys=True)
except TypeError:
raise KeyTypeError
vyasr marked this conversation as resolved.
Show resolved Hide resolved
m = hashlib.md5()
m.update(blob.encode())
return m.hexdigest()
2 changes: 1 addition & 1 deletion signac/contrib/import_export.py
Original file line number Diff line number Diff line change
Expand Up @@ -766,7 +766,7 @@ def _copy_to_job_workspace(src, job, copytree):
raise DestinationExistsError(job)
raise
else:
job._init()
job.init()
return dst


Expand Down