/dat

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

change log ids in data keys and current table #207

Closed
wants to merge 3 commits into
base: master
from

Conversation

Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@ekg @maxogden
@ekg
Contributor

ekg commented Oct 17, 2014

The addition of change log ids to the data keys (after the versions), will allow us to quickly extract the state of the data at particular point in the change log. This can be accomplished via a linear scan of the keys in the data table, requiring that the change id of a particular object is <= the target point in the log.

If we did not include this data alongside the data, we would be forced to complete a reconstruction of the dataset via the change log. This would complicate the process of rolling back particular subsets of the data to predetermined points in the history. Additionally, it wouldn't be possible to quickly determine the relative age of two objects, which has a number of possible applications in reproducibility and logging.

No functionality is yet tested which is based on the change ids, but the next step should be to implement a commit/checkout or checkpoint/rollback model on top of it.

ekg added some commits Oct 16, 2014

@ekg
add references to change index to data keys 
By adding a reference to the change index id to data table keys, we can quickly
revert the current table of the repository to a particular checkpoint.

These changes only enable the storage of the change index keys.

This is not a stable commit. A majority of tests now pass, but there are still
significant issues.
0694c9b
@ekg
Merge branch 'master' of https://github.com/maxogden/dat
030df1e
@ekg
store change ids alongside data to enable fast rollback 
The addition of change log ids to the data keys (after the versions), will
allow us to quickly extract the state of the data at particular point in the
change log. This can be accomplished via a linear scan of the keys in the data
table, requiring that the change id of a particular object is <= the target
point in the log.

If we did not include this data alongside the data, we would be forced to
complete a reconstruction of the dataset via the change log. This would
complicate the process of rolling back particular subsets of the data to
predetermined points in the history. Additionally, it wouldn't be possible to
quickly determine the relative age of two objects, which has a number of
possible applications in reproducibility and logging.

The level-dat backend will support these change ids as of 4.5.0.

No functionality is yet tested which is based on the change ids, but the next
step should be to implement a commit/checkout or checkpoint/rollback model on
top of it.

With this update we now pass 616/616 tests.
58d5358

@ekg ekg referenced this pull request Oct 17, 2014

Open

change log ids in data keys and current table #1

@ekg

This comment has been minimized.

Show comment
Hide comment
@ekg

ekg Oct 17, 2014

Contributor

This depends on mafintosh/level-dat#1
Contributor

ekg commented Oct 17, 2014

This depends on mafintosh/level-dat#1
@ekg

This comment has been minimized.

Show comment
Hide comment
@ekg

ekg Oct 17, 2014

Contributor

It looks like the build error results from the requirement that we update the level-dat version.
Contributor

ekg commented Oct 17, 2014

It looks like the build error results from the requirement that we update the level-dat version.
@maxogden

This comment has been minimized.

Show comment
Hide comment
@maxogden

maxogden Nov 15, 2014

Member

Just commenting here for posterity, we have discussed this pull request and are still investigating if it is the right approach. Keeping it open for now
Member

maxogden commented Nov 15, 2014

Just commenting here for posterity, we have discussed this pull request and are still investigating if it is the right approach. Keeping it open for now
@maxogden

This comment has been minimized.

Show comment
Hide comment
@maxogden

maxogden Dec 4, 2014

Member

gonna close for now, but we will definitely make sure this gets in after @mafintosh refactors the storage stuff
Member

maxogden commented Dec 4, 2014

gonna close for now, but we will definitely make sure this gets in after @mafintosh refactors the storage stuff

@maxogden maxogden closed this Dec 4, 2014

@maxogden maxogden referenced this pull request Dec 23, 2014

Closed

snapshots/checkout support #256

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment