Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

635 revision avail in rev tree #671

Merged
merged 9 commits into from

2 participants

@neojski
Owner

We now hold every revision status (available/missing) in revision tree.

Main changes:

  • parseDoc parses status
  • doMerge is status aware
  • compaction got a little redesigned
  • get does not have to ping every document to know if it is there or got compacted (revs_info)
src/pouch.merge.js
@@ -48,6 +48,12 @@ function mergeTree(in_tree1, in_tree2) {
var tree1 = item.tree1;
var tree2 = item.tree2;
+ if (tree1[1].status === 'available' || tree2[1].status === 'available') {
+ tree1[1].status = 'available';
+ } else {
+ tree1[1].status = 'missing';
+ }
+
@daleharvey Owner

Would something like:

if (tree1[1].status || tree2[1].status) { 
  tree1[1].status = (tree1[1].status === 'available' || tree2[1].status === 'available') ? 'available' : 'missing';
}

work here? it is nice to make things those options things opaque, so they only act on information they are given to them, I dont think we should be changing the tests for this change, if we merge {} {} we get {}

@neojski Owner
neojski added a note

Yes, this works. I just wanted to make it explicit as we assume that this information is there and we don't want to optimize it right now. Maybe for doMerge this can be less explicit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
src/adapters/pouch.websql.js
((15 lines not shown))
+ if (!result.rows.length) {
+ return call(callback);
+ }
+ var metadata = JSON.parse(result.rows.item(0).metadata);
+ metadata.rev_tree = rev_tree;
+
+ var sql = 'DELETE FROM ' + BY_SEQ_STORE + ' WHERE doc_id_rev IN (' +
+ revs.map(function(rev){return quote(docId + '::' + rev);}).join(',') + ')';
+
+ tx.executeSql(sql, [], function(tx, result) {
+ var sql = 'UPDATE ' + DOC_STORE + ' SET json = ? WHERE id = ?';
+
+ tx.executeSql(sql, [JSON.stringify(metadata), docId], function(tx, result) {
+ callback();
+ }, function(tx, res){
+ console.log(res);
@daleharvey Owner

just pointing out so we remember to remove

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@daleharvey
Owner

This is looking great, good job, few comments in there

neojski added some commits
@neojski neojski (#635) - compact deleted document (test)
Added test so we make sure that also deleted document get compacted
dc3c5bc
@neojski neojski (#635) - compact deleted document as well
Now deleted document are also compacted.
4187ae3
@neojski neojski (#635) - low level method document availability support
parseDoc and merge have to support new availability option
ef45f7e
@neojski neojski (#635) - api.get support availability in rev tree d71e3b5
@neojski neojski (#635) - make comapction use status from rev_tree
Start with idb adapter
0d1dc21
@neojski neojski (#635) - refactor websql and leveldb adapters 99b7bd6
@neojski neojski (#635) - move more logic to generic adapter (auto_compaction)
In order auto_compaction to work I moved modification of the whole
rev_tree into compactDocument in generic adapter. It then sends
revisions which we want to delete and modified rev_tree to adapter
specific method which does the compaction process.
8016337
@neojski neojski (#635) - merge_rev_tree tests now support availability
That's quite to much said as they don't check if the availability
is merged correctly.
753a23d
@neojski neojski (#635) - make merging status less explicit
If we merge statuses {} and {} we now get {} and not {status: missing}
769f782
@neojski
Owner

Rebased this.

@daleharvey
Owner

\o/

@daleharvey daleharvey merged commit 769f782 into pouchdb:master
@neojski neojski deleted the neojski:635-revision-avail-in-rev_tree branch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Apr 5, 2013
  1. @neojski

    (#635) - compact deleted document (test)

    neojski authored
    Added test so we make sure that also deleted document get compacted
  2. @neojski

    (#635) - compact deleted document as well

    neojski authored
    Now deleted document are also compacted.
  3. @neojski

    (#635) - low level method document availability support

    neojski authored
    parseDoc and merge have to support new availability option
  4. @neojski
  5. @neojski

    (#635) - make comapction use status from rev_tree

    neojski authored
    Start with idb adapter
  6. @neojski
  7. @neojski

    (#635) - move more logic to generic adapter (auto_compaction)

    neojski authored
    In order auto_compaction to work I moved modification of the whole
    rev_tree into compactDocument in generic adapter. It then sends
    revisions which we want to delete and modified rev_tree to adapter
    specific method which does the compaction process.
  8. @neojski

    (#635) - merge_rev_tree tests now support availability

    neojski authored
    That's quite to much said as they don't check if the availability
    is merged correctly.
  9. @neojski

    (#635) - make merging status less explicit

    neojski authored
    If we merge statuses {} and {} we now get {} and not {status: missing}
This page is out of date. Refresh to see the latest.
View
46 src/adapters/pouch.idb.js
@@ -841,7 +841,6 @@ var IdbPouch = function(opts, callback) {
call(callback, null);
};
- // compaction internal functions
api._getRevisionTree = function(docId, callback) {
var txn = idb.transaction([DOC_STORE], 'readonly');
var req = txn.objectStore(DOC_STORE).get(docId);
@@ -855,24 +854,39 @@ var IdbPouch = function(opts, callback) {
};
};
- api._removeDocRevisions = function(docId, revs, callback) {
- var txn = idb.transaction([BY_SEQ_STORE], IDBTransaction.READ_WRITE);
- var index = txn.objectStore(BY_SEQ_STORE).index('_doc_id_rev');
- revs.forEach(function(rev) {
- var key = docId + "::" + rev;
- index.getKey(key).onsuccess = function(e) {
- var seq = e.target.result;
- if (!seq) {
- return;
- }
- var req = txn.objectStore(BY_SEQ_STORE)['delete'](seq);
- };
- });
+ // This function removes revisions of document docId
+ // which are listed in revs and sets this document
+ // revision to to rev_tree
+ api._doCompaction = function(docId, rev_tree, revs, callback) {
+ var txn = idb.transaction([DOC_STORE, BY_SEQ_STORE], IDBTransaction.READ_WRITE);
+
+ var index = txn.objectStore(DOC_STORE);
+ index.get(docId).onsuccess = function(event) {
+ var metadata = event.target.result;
+ metadata.rev_tree = rev_tree;
+
+ var count = revs.length;
+ revs.forEach(function(rev) {
+ var index = txn.objectStore(BY_SEQ_STORE).index('_doc_id_rev');
+ var key = docId + "::" + rev;
+ index.getKey(key).onsuccess = function(e) {
+ var seq = e.target.result;
+ if (!seq) {
+ return;
+ }
+ var req = txn.objectStore(BY_SEQ_STORE)['delete'](seq);
+
+ count--;
+ if (!count) {
+ txn.objectStore(DOC_STORE).put(metadata);
+ }
+ };
+ });
+ };
txn.oncomplete = function() {
- callback();
+ call(callback);
};
};
- // end of compaction internal functions
return api;
};
View
33 src/adapters/pouch.leveldb.js
@@ -724,7 +724,6 @@ var LevelPouch = function(opts, callback) {
}
};
- // compaction internal functions
api._getRevisionTree = function(docId, callback){
stores[DOC_STORE].get(docId, function(err, metadata) {
if (err) {
@@ -735,14 +734,12 @@ var LevelPouch = function(opts, callback) {
});
};
- api._removeDocRevisions = function(docId, revs, callback) {
- if (!revs.length) {
- callback();
- }
+ api._doCompaction = function(docId, rev_tree, revs, callback) {
stores[DOC_STORE].get(docId, function(err, metadata) {
var seqs = metadata.rev_map; // map from rev to seq
- var count = revs.count;
+ metadata.rev_tree = rev_tree;
+ var count = revs.length;
function done() {
count--;
if (!count) {
@@ -750,19 +747,25 @@ var LevelPouch = function(opts, callback) {
}
}
- revs.forEach(function(rev) {
- var seq = seqs[rev];
- if (!seq) {
- done();
- return;
- }
- stores[BY_SEQ_STORE].del(seq, function(err) {
- done();
+ if (!count) {
+ callback();
+ }
+
+ stores[DOC_STORE].put(metadata.id, metadata, function() {
+ revs.forEach(function(rev) {
+ var seq = seqs[rev];
+ if (!seq) {
+ done();
+ return;
+ }
+
+ stores[BY_SEQ_STORE].del(seq, function(err) {
+ done();
+ });
});
});
});
};
- // end of compaction internal functions
return api;
};
View
28 src/adapters/pouch.websql.js
@@ -665,7 +665,7 @@ var webSqlPouch = function(opts, callback) {
});
}
};
- // comapction internal functions
+
api._getRevisionTree = function(docId, callback) {
db.transaction(function (tx) {
var sql = 'SELECT json AS metadata FROM ' + DOC_STORE + ' WHERE id = ?';
@@ -679,16 +679,30 @@ var webSqlPouch = function(opts, callback) {
});
});
};
- api._removeDocRevisions = function(docId, revs, callback) {
+
+ api._doCompaction = function(docId, rev_tree, revs, callback) {
db.transaction(function (tx) {
- var sql = 'DELETE FROM ' + BY_SEQ_STORE + ' WHERE doc_id_rev IN (' +
- revs.map(function(rev){return quote(docId + '::' + rev);}).join(',') + ')';
- tx.executeSql(sql, [], function(tx, result) {
- callback();
+ var sql = 'SELECT json AS metadata FROM ' + DOC_STORE + ' WHERE id = ?';
+ tx.executeSql(sql, [docId], function(tx, result) {
+ if (!result.rows.length) {
+ return call(callback);
+ }
+ var metadata = JSON.parse(result.rows.item(0).metadata);
+ metadata.rev_tree = rev_tree;
+
+ var sql = 'DELETE FROM ' + BY_SEQ_STORE + ' WHERE doc_id_rev IN (' +
+ revs.map(function(rev){return quote(docId + '::' + rev);}).join(',') + ')';
+
+ tx.executeSql(sql, [], function(tx, result) {
+ var sql = 'UPDATE ' + DOC_STORE + ' SET json = ? WHERE id = ?';
+
+ tx.executeSql(sql, [JSON.stringify(metadata), docId], function(tx, result) {
+ callback();
+ });
+ });
});
});
};
- // end of compaction internal functions
return api;
};
View
80 src/pouch.adapter.js
@@ -204,34 +204,47 @@ var PouchAdapter = function(opts, callback) {
return call(callback);
}
var height = computeHeight(rev_tree);
- var nonLeaves = [];
+ var candidates = [];
+ var revs = [];
Object.keys(height).forEach(function(rev) {
if (height[rev] > max_height) {
- nonLeaves.push(rev);
+ candidates.push(rev);
}
});
- customApi._removeDocRevisions(docId, nonLeaves, callback);
+
+ Pouch.merge.traverseRevTree(rev_tree, function(isLeaf, pos, revHash, ctx, opts) {
+ var rev = pos + '-' + revHash;
+ if (opts.status === 'available' && candidates.indexOf(rev) !== -1) {
+ opts.status = 'missing';
+ revs.push(rev);
+ }
+ });
+ customApi._doCompaction(docId, rev_tree, revs, callback);
});
};
// compact the whole database using single document
// compaction
api.compact = function(callback) {
- api.allDocs(function(err, res) {
- var count = res.rows.length;
+ api.changes({complete: function(err, res) {
+ if (err) {
+ call(callback); // TODO: silently fail
+ return;
+ }
+ var count = res.results.length;
if (!count) {
call(callback);
return;
}
- res.rows.forEach(function(row) {
- compactDocument(row.key, 0, function() {
+ res.results.forEach(function(row) {
+ compactDocument(row.id, 0, function() {
count--;
if (!count) {
call(callback);
}
});
});
- });
+ }});
};
/* Begin api wrappers. Specific functionality to storage belongs in the _[method] */
@@ -311,9 +324,6 @@ var PouchAdapter = function(opts, callback) {
}
var doc = result;
- function finish() {
- call(callback, null, doc);
- }
if (opts.conflicts) {
var conflicts = Pouch.merge.collectConflicts(metadata);
@@ -324,53 +334,35 @@ var PouchAdapter = function(opts, callback) {
if (opts.revs || opts.revs_info) {
var paths = rootToLeaf(metadata.rev_tree);
- paths.map(function(path, i) {
- paths[i].ids = path.ids.map(function(x) { return x.id; });
- });
var path = arrayFirst(paths, function(arr) {
- return arr.ids.indexOf(doc._rev.split('-')[1]) !== -1;
+ return arr.ids.map(function(x) { return x.id; })
+ .indexOf(doc._rev.split('-')[1]) !== -1;
});
- path.ids.splice(path.ids.indexOf(doc._rev.split('-')[1]) + 1);
+
+ path.ids.splice(path.ids.map(function(x) {return x.id;})
+ .indexOf(doc._rev.split('-')[1]) + 1);
path.ids.reverse();
if (opts.revs) {
doc._revisions = {
start: (path.pos + path.ids.length) - 1,
- ids: path.ids
+ ids: path.ids.map(function(rev) {
+ return rev.id;
+ })
};
}
if (opts.revs_info) {
- // TODO: it could be slow to test status like this
- var count = path.ids.length;
- var pos = path.pos + path.ids.length - 1;
- doc._revs_info = [];
-
- path.ids.forEach(function(hash) {
- var rev = pos + '-' + hash;
- var info = {
- rev: rev,
- status: "available"
- };
+ var pos = path.pos + path.ids.length;
+ doc._revs_info = path.ids.map(function(rev) {
pos--;
- doc._revs_info.push(info);
-
- api.get(id.docId, {rev: rev}, function(err, ok) {
- if (err) {
- info.status = "missing";
- }
- count--;
- if (!count) {
- finish();
- }
- });
+ return {
+ rev: pos + '-' + rev.id,
+ status: rev.opts.status
+ };
});
- } else {
- finish();
}
- } else {
- finish();
}
-
+ call(callback, null, doc);
});
};
View
5 src/pouch.merge.js
@@ -48,6 +48,11 @@ function mergeTree(in_tree1, in_tree2) {
var tree1 = item.tree1;
var tree2 = item.tree2;
+ if (tree1[1].status || tree2[1].status) {
+ tree1[1].status = (tree1[1].status === 'available' ||
+ tree2[1].status === 'available') ? 'available' : 'missing';
+ }
+
for (var i = 0; i < tree2[2].length; i++) {
if (!tree1[2][0]) {
conflicts = 'new_leaf';
View
6 src/pouch.utils.js
@@ -79,7 +79,7 @@ var parseDoc = function(doc, newEdits) {
var nRevNum;
var newRevId;
var revInfo;
- var opts = {};
+ var opts = {status: 'available'};
if (doc._deleted) {
opts.deleted = true;
}
@@ -96,7 +96,7 @@ var parseDoc = function(doc, newEdits) {
}
doc._rev_tree = [{
pos: parseInt(revInfo[1], 10),
- ids: [revInfo[2], {}, [[newRevId, opts, []]]]
+ ids: [revInfo[2], {status: 'missing'}, [[newRevId, opts, []]]]
}];
nRevNum = parseInt(revInfo[1], 10) + 1;
} else {
@@ -114,7 +114,7 @@ var parseDoc = function(doc, newEdits) {
if (acc === null) {
return [x, opts, []];
} else {
- return [x, {}, [acc]];
+ return [x, {status: 'missing'}, [acc]];
}
}, null)
}];
View
17 tests/test.compaction.js
@@ -196,6 +196,23 @@ adapters.map(function(adapter) {
});
});
+ asyncTest('Compact deleted document', function() {
+ initTestDB(this.name, function(err, db) {
+ db.put({_id: "foo"}, function(err, res) {
+ var firstRev = res.rev;
+ db.remove({_id: "foo", _rev: firstRev}, function(err, res) {
+ db.compact(function() {
+ db.get("foo", {rev: firstRev}, function(err, res) {
+ ok(err, "got error");
+ strictEqual(err.reason, "missing", "correct reason");
+ start();
+ });
+ });
+ });
+ });
+ });
+ });
+
if (autoCompactionAdapters.indexOf(adapter) > -1) {
asyncTest('Auto-compaction test', function() {
initTestDB(this.name, {auto_compaction: true}, function(err, db) {
View
2  tests/test.get.js
@@ -232,7 +232,7 @@ adapters.map(function(adapter) {
var simpleTree = [
[
{_id: "foo", _rev: "1-a", value: "foo a"},
- {_id: "foo", _rev: "2-b", value: "foo d"},
+ {_id: "foo", _rev: "2-b", value: "foo b"},
{_id: "foo", _rev: "3-c", value: "foo c"}
],
[
Something went wrong with that request. Please try again.