Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compaction API #350

Merged
merged 5 commits into from Oct 28, 2019
Merged
Changes from 1 commit
Commits
File filter...
Filter file types
Jump to…
Jump to file
Failed to load files.

Always

Just for now

Add test for compactCategory

  • Loading branch information
darkdh committed Oct 18, 2019
commit df40ca15cff0d0ba253680457f5f8eef2d424081
@@ -447,7 +447,6 @@ RequestUtil.prototype.compactRecords = function (category) {
let latestRecords = {}

This comment has been minimized.

Copy link
@bridiver

bridiver Oct 21, 2019

Contributor

this is going to keep the most recent record in each batch, which is fine, but I think we should add a comment to make that clear

let s3ObjectsToDelete = []
const recordObjects = this.s3ObjectsToRecords(s3Objects.contents)
console.error(recordObjects)
recordObjects.forEach((recordObject) => {
const record = recordObject.record
This conversation was marked as resolved by AlexeyBarabash

This comment has been minimized.

Copy link
@AlexeyBarabash

AlexeyBarabash Oct 21, 2019

Contributor

I would ask to make additional logging of the records being purged at compactCategory: action, objectId and timestamp to test and diagnose it easier.

if (latestRecords[record.objectId]) {
@@ -350,8 +350,6 @@ module.exports.deleteObjects = function (s3, bucket, objects) {
}, function (err, data) {
if (err) {
console.error(err, err.stack)
} else {
console.log(data)
}
})
}
@@ -387,7 +387,7 @@ test('client RequestUtil', (t) => {
t.equals(response.length, 1, `${t.name} deletes records`)
const s3Record = response[0].record
t.assert(s3Record.device && s3Record.device.name, `${t.name} preserves device records`)
testCanLimitResponse(t)
testCanDoCompaction(t)
})
.catch((error) => { t.fail(error) })
})
@@ -397,6 +397,77 @@ test('client RequestUtil', (t) => {
})
}

const testCanDoCompaction = (t) => {
t.test('#compact bookmarks', (t) => {
t.plan(3)
const recordObjectId = testHelper.newUuid()
const record2ObjectId = testHelper.newUuid()
const record = {
action: 'CREATE',
deviceId: new Uint8Array([0]),
objectId: recordObjectId,
bookmark: {
site: {
location: `https://brave.com?q=${'x'.repeat(4)}`,
title: 'BRAVE',
lastAccessedTime: 1480000000 * 1000,
creationTime: 1480000000 * 1000
},
isFolder: false,
hideInToolbar: false,
order: '1.0.0.1'
}
}
const record2 = {
action: 'CREATE',
deviceId: new Uint8Array([0]),
objectId: record2ObjectId,
bookmark: {
site: {
location: `https://brave.com?q=${'x'.repeat(4096)}`,
title: 'BRAVEE',
lastAccessedTime: 1480000000 * 1000,
creationTime: 1480000000 * 1000
},
isFolder: false,
hideInToolbar: false,
order: '1.0.0.2'
}
}
let record_update = record
record_update.action = 'UPDATE'
let record2_update = record2
record2_update.action = 'UPDATE'

requestUtil.put(proto.categories.BOOKMARKS, record)
requestUtil.put(proto.categories.BOOKMARKS, record2)
for (let i = 0; i < 100; ++i) {
record_update.bookmark.site.title = `${record.bookmark.site.title} ${i}`
record2_update.bookmark.site.title = `${record2.bookmark.site.title} ${i}`
requestUtil.put(proto.categories.BOOKMARKS, record_update)
requestUtil.put(proto.categories.BOOKMARKS, record2_update)
}
requestUtil.compactRecords(proto.categories.BOOKMARKS)
setTimeout(() => {
requestUtil.list(proto.categories.BOOKMARKS)
.then(s3Objects => requestUtil.s3ObjectsToRecords(s3Objects.contents))
.then((response) => {
t.equals(response.length, 2, `${t.name} check records number`)
const s3Record = response[0].record
const s3Record2 = response[1].record
t.deepEquals(s3Record.objectId, record.objectId, `${t.name}: objectId`)
t.deepEquals(s3Record.bookmark.site.title, record_update.bookmark.site.title, `${t.name}: bookmark.title`)
requestUtil.deleteCategory(proto.categories.BOOKMARKS)
.then((_response) => {
testCanLimitResponse(t)
})
.catch((error) => { t.fail(error) })
})
.catch((error) => { t.fail(error) })
}, 60000)
})
}

const testCanLimitResponse = (t) => {
t.test('limitResponse undefined', (t) => {
t.plan(3)
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.