Skip to content

/ sync

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compaction API #350

Merged
merged 5 commits into from Oct 28, 2019
Merged

Compaction API #350

merged 5 commits into from Oct 28, 2019

Conversation

@darkdh
Copy link
Member

darkdh commented Oct 18, 2019 •
edited

fix #341
fix #289
brave/brave-browser#6552 will use this API to initiate the compaction
darkdh added 3 commits Oct 16, 2019
@darkdh
Compact records based on timestamp and keep the latest one
Verified
This commit was signed with a verified signature.
darkdh Anthony Tseng
GPG key ID: E14094A20D3AAEEC Learn about signing commits
5fe3f99
@darkdh
Add test for compactCategory
Verified
This commit was signed with a verified signature.
darkdh Anthony Tseng
GPG key ID: E14094A20D3AAEEC Learn about signing commits
df40ca1
@darkdh
Export COMPACT_SYNC_CATEGORY
Verified
This commit was signed with a verified signature.
darkdh Anthony Tseng
GPG key ID: E14094A20D3AAEEC Learn about signing commits
Loading status checks…
b95ffb2
@darkdh darkdh self-assigned this Oct 18, 2019
@AlexeyBarabash
AlexeyBarabash reviewed Oct 21, 2019
View changes
client/requestUtil.js Outdated Show resolved Hide resolved
@darkdh darkdh requested a review from SergeyZhukovsky Oct 21, 2019
@bridiver
bridiver reviewed Oct 21, 2019
View changes
client/requestUtil.js Outdated
this.withRetry(() => {
return s3Helper.listObjects(this.s3, options, false)
}).then((s3Objects) => {
let latestRecords = {}

This comment has been minimized.

Sign in to view
Copy link
@bridiver

bridiver Oct 21, 2019

Contributor

this is going to keep the most recent record in each batch, which is fine, but I think we should add a comment to make that clear
@darkdh
Address review feedbacks
Verified
This commit was signed with a verified signature.
darkdh Anthony Tseng
GPG key ID: E14094A20D3AAEEC Learn about signing commits
Loading status checks…
6e945b2
@AlexeyBarabash
AlexeyBarabash reviewed Oct 21, 2019
View changes
Copy link
Contributor

AlexeyBarabash left a comment

Looks good.
@darkdh darkdh mentioned this pull request Oct 21, 2019
Merged
8 of 32 tasks complete
@AlexeyBarabash
AlexeyBarabash reviewed Oct 22, 2019
View changes
client/requestUtil.js Outdated Show resolved Hide resolved
@AlexeyBarabash
AlexeyBarabash reviewed Oct 22, 2019
View changes
client/requestUtil.js Outdated Show resolved Hide resolved
@SergeyZhukovsky
SergeyZhukovsky reviewed Oct 22, 2019
View changes
client/requestUtil.js Outdated Show resolved Hide resolved
@SergeyZhukovsky
SergeyZhukovsky reviewed Oct 22, 2019
View changes
Copy link
Member

SergeyZhukovsky left a comment

++
@darkdh darkdh dismissed stale reviews from SergeyZhukovsky and AlexeyBarabash via 65e0165 Oct 25, 2019
@darkdh darkdh force-pushed the compaction branch 3 times, most recently from bd636b1 to 3669788 Oct 25, 2019
@SergeyZhukovsky
SergeyZhukovsky reviewed Oct 25, 2019
View changes
Copy link
Member

SergeyZhukovsky left a comment

++
@darkdh
Do compaction in batch with 15 seconds timeout, also support cross batch
Verified
This commit was signed with a verified signature.
darkdh Anthony Tseng
GPG key ID: E14094A20D3AAEEC Learn about signing commits
Loading status checks…
78e8d89 
compaction for up to 100 records

Add LRU cache(50 records) for missing part for decryption so it can be decrypted in
future round
@darkdh darkdh force-pushed the compaction branch from 3669788 to 78e8d89 Oct 25, 2019
@bridiver
bridiver approved these changes Oct 28, 2019
View changes
@SergeyZhukovsky
SergeyZhukovsky approved these changes Oct 28, 2019
View changes
Copy link
Member

SergeyZhukovsky left a comment

++
@darkdh darkdh merged commit 6f8795a into staging Oct 28, 2019
2 checks passed
2 checks passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
@darkdh darkdh deleted the compaction branch Oct 28, 2019
darkdh added a commit that referenced this pull request Oct 28, 2019
@darkdh
Merge pull request #350 from brave/compaction
Verified
This commit was signed with a verified signature.
darkdh Anthony Tseng
GPG key ID: E14094A20D3AAEEC Learn about signing commits
Loading status checks…
001c90e 
Compaction API
This was referenced Oct 28, 2019
Merged
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AlexeyBarabash AlexeyBarabash

@bridiver bridiver

@SergeyZhukovsky SergeyZhukovsky

@diracdeltas diracdeltas

Assignees

@darkdh darkdh

Labels
None yet
Projects
None yet
Milestone
No milestone
Linked issues

Successfully merging this pull request may close these issues.

Implement sync record compaction
Sync record from SQS is not decoded sometimes
4 participants
@darkdh @bridiver @SergeyZhukovsky @AlexeyBarabash
You can’t perform that action at this time.