Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SnapbackSM - Phase 1 - Recurring Sync #892

Merged
merged 76 commits into from
Oct 13, 2020
Merged
Show file tree
Hide file tree
Changes from 53 commits
Commits
Show all changes
76 commits
Select commit Hold shift + click to select a range
60d2007
Add config to convict
hareeshnagaraj Sep 4, 2020
3316def
Writing out endpoint config
hareeshnagaraj Sep 4, 2020
0028bc5
Propagate config to yml
hareeshnagaraj Sep 4, 2020
1769511
Merge branch 'master' into hn_node_identity
hareeshnagaraj Sep 8, 2020
5666d57
Retrieval of spID from eth-contracts
hareeshnagaraj Sep 8, 2020
9f01ae1
Happy case initialization + skeleton class
hareeshnagaraj Sep 8, 2020
d3a409c
More progress, init fully in class now
hareeshnagaraj Sep 8, 2020
1f96d39
separate function
hareeshnagaraj Sep 9, 2020
fdc0a8a
Merge branch 'master' into hn_node_identity
hareeshnagaraj Sep 9, 2020
b5d55db
Lint
hareeshnagaraj Sep 9, 2020
333a362
Lint 2
hareeshnagaraj Sep 9, 2020
d055d4b
NIT
hareeshnagaraj Sep 9, 2020
d3db93b
More lints
hareeshnagaraj Sep 9, 2020
bc7b534
warning msg
hareeshnagaraj Sep 9, 2020
a990675
NIT
hareeshnagaraj Sep 9, 2020
dd581b0
Merge branch 'master' into hn_node_identity
hareeshnagaraj Sep 10, 2020
4be405f
Added test, address PR comms
hareeshnagaraj Sep 10, 2020
0e89ca6
Run tests update
hareeshnagaraj Sep 10, 2020
66fc1b8
Default empty config
hareeshnagaraj Sep 10, 2020
bedc489
LINT
hareeshnagaraj Sep 10, 2020
5e5503e
RM unused functions
hareeshnagaraj Sep 10, 2020
6019f3b
RM only
hareeshnagaraj Sep 11, 2020
c6ac974
Merge master, adding svc command changes
hareeshnagaraj Sep 25, 2020
5c5fa93
Config update
hareeshnagaraj Sep 25, 2020
d653861
Scripts update
hareeshnagaraj Sep 25, 2020
13c5499
Merge branch 'master' into hn_node_identity
hareeshnagaraj Oct 5, 2020
5631a73
Update to disable snapback for usermetadata
hareeshnagaraj Oct 5, 2020
e85de67
Constructor fix
hareeshnagaraj Oct 5, 2020
6127ff1
Test fix
hareeshnagaraj Oct 5, 2020
619944c
New config
hareeshnagaraj Oct 6, 2020
4020fb7
first pass of queries
hareeshnagaraj Oct 6, 2020
67682ad
Users for node query
hareeshnagaraj Oct 6, 2020
f6d1714
Do not init in serviceRegistry
hareeshnagaraj Oct 6, 2020
b666a0e
Merge branch 'master' into hn_snapbacksm_phase1_recurring_sync
hareeshnagaraj Oct 7, 2020
57b95ff
Naive sync evaluation completed
hareeshnagaraj Oct 7, 2020
ddc75a3
About to enqueue sync op
hareeshnagaraj Oct 7, 2020
39688c5
Disabling traditional sync for testing
hareeshnagaraj Oct 7, 2020
cdf82d2
issueSecondarySync abstraction
hareeshnagaraj Oct 7, 2020
22316c0
Functioning batch query
hareeshnagaraj Oct 7, 2020
5a1b455
Merge branch 'master' into hn_snapbacksm_phase1_recurring_sync
hareeshnagaraj Oct 7, 2020
7eb32e3
TODO nodeQueryList
hareeshnagaraj Oct 7, 2020
fe95be1
Merge branch 'master' into hn_snapbacksm_phase1_recurring_sync
hareeshnagaraj Oct 7, 2020
c124b30
Batched clock requests now sent
hareeshnagaraj Oct 8, 2020
bf4a420
Fix for scenario with multiple increments per block
hareeshnagaraj Oct 8, 2020
a67dccb
Sync monitoring...demonstrate the behavior of repeated failure locall…
hareeshnagaraj Oct 8, 2020
33ccc06
Swithc to POST request
hareeshnagaraj Oct 8, 2020
d4a937e
Parallel sync limit
hareeshnagaraj Oct 8, 2020
d6830a7
Sync clock check fix, minor logging change
hareeshnagaraj Oct 8, 2020
924eb1c
Updated dp from self comments
hareeshnagaraj Oct 8, 2020
7762c98
Confirm functionality + linting
hareeshnagaraj Oct 8, 2020
05f305e
DP linting smh
hareeshnagaraj Oct 8, 2020
aadecac
Merge branch 'master' into hn_snapbacksm_phase1_recurring_sync
hareeshnagaraj Oct 8, 2020
652b18b
[TEST] SnapbackSM - Phase 1 - Recurring Sync (#909)
hareeshnagaraj Oct 9, 2020
93f3cc2
Minor updates
hareeshnagaraj Oct 9, 2020
549ae8e
Users list filtering
hareeshnagaraj Oct 9, 2020
9aa429b
TODO: Randomize starting slice
hareeshnagaraj Oct 9, 2020
d360709
rm async pre-processing, batch primary clock queries, disc prov query…
hareeshnagaraj Oct 11, 2020
bffd728
Porting test branch changes
hareeshnagaraj Oct 11, 2020
14a3faf
RM test func
hareeshnagaraj Oct 11, 2020
5564c43
LINTLINT
hareeshnagaraj Oct 11, 2020
77d6f24
Update config name and func
hareeshnagaraj Oct 11, 2020
d1da057
Custom convict format for bool, issue opened against node-convict
hareeshnagaraj Oct 12, 2020
c05819f
Reuse internal func
hareeshnagaraj Oct 12, 2020
5514296
Throw if no selected discovery provider and re-init
hareeshnagaraj Oct 12, 2020
bce1fd2
Compose config update
hareeshnagaraj Oct 12, 2020
9c31500
Config format fix + query update
hareeshnagaraj Oct 12, 2020
c82f6b6
Merge branch 'master' into hn_snapbacksm_phase1_recurring_sync
hareeshnagaraj Oct 12, 2020
d9355f7
linting, addressing PR comments
hareeshnagaraj Oct 13, 2020
562ea9e
nodeSync behavior change to log success if up to date
hareeshnagaraj Oct 13, 2020
c829952
schedule fix attempt
hareeshnagaraj Oct 13, 2020
eee9832
Continue if init succeeded instead of waiting for next slice
hareeshnagaraj Oct 13, 2020
9cf0619
Log out slice
hareeshnagaraj Oct 13, 2020
aed1fb0
Consume cached primaryClockValue
hareeshnagaraj Oct 13, 2020
4f91905
Reset to 24
hareeshnagaraj Oct 13, 2020
d7e85a9
Merge branch 'master' into hn_snapbacksm_phase1_recurring_sync
hareeshnagaraj Oct 13, 2020
794e3e0
Logging cron
hareeshnagaraj Oct 13, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions creator-node/compose/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ services:
- delegateOwnerWallet=${delegateOwnerWallet}
- delegatePrivateKey=${delegatePrivateKey}
- creatorNodeEndpoint=${creatorNodeEndpoint}
- triggerSyncOnWrite=${triggerSyncOnWrite}
hareeshnagaraj marked this conversation as resolved.
Show resolved Hide resolved
env_file:
- ./env/base.env
ports:
Expand Down
6 changes: 6 additions & 0 deletions creator-node/src/config.js
Original file line number Diff line number Diff line change
Expand Up @@ -378,6 +378,12 @@ const config = convict({
format: 'nat',
env: 'rehydrateMaxConcurrency',
default: 10
},
triggerSyncOnWrite: {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not change this config to be more clearly dev like 'snapbackDevModeEnabled' and then use that to either schedule cron or a repeated job (like is currently enabled)?

doc: 'TEST ONLY. DO NOT CONFIGURE MANUALLY. Disables automatic secondary sync issuing in order to test SnapbackSM.',
format: Boolean,
env: 'triggerSyncOnWrite',
default: true
}

// unsupported options at the moment
Expand Down
2 changes: 1 addition & 1 deletion creator-node/src/middlewares.js
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ async function ensurePrimaryMiddleware (req, res, next) {
* @dev - Is not a middleware so it can be run before responding to client.
*/
async function triggerSecondarySyncs (req) {
if (config.get('isUserMetadataNode')) return
if (config.get('isUserMetadataNode') || !config.get('triggerSyncOnWrite')) return
try {
if (!req.session.nodeIsPrimary || !req.session.creatorNodeEndpoints || !Array.isArray(req.session.creatorNodeEndpoints)) return
const [primary, ...secondaries] = req.session.creatorNodeEndpoints
Expand Down
8 changes: 5 additions & 3 deletions creator-node/src/routes/nodeSync.js
Original file line number Diff line number Diff line change
Expand Up @@ -308,19 +308,21 @@ async function _nodesync (req, walletPublicKeys, creatorNodeEndpoint, dbOnlySync
transaction
})
const fetchedLatestBlockNumber = fetchedCNodeUser.latestBlockNumber
const fetchedLatestClockVal = fetchedCNodeUser.clock
hareeshnagaraj marked this conversation as resolved.
Show resolved Hide resolved

// Delete any previously stored data for cnodeUser in reverse table dependency order (cannot be parallelized).
if (cnodeUser) {
// Ensure imported data has higher blocknumber than already stored.
// TODO - replace this check with a clock check (!!!)
const latestBlockNumber = cnodeUser.latestBlockNumber
const latestClockValue = cnodeUser.clock

if (!dbOnlySync) {
hareeshnagaraj marked this conversation as resolved.
Show resolved Hide resolved
if ((fetchedLatestBlockNumber === -1 && latestBlockNumber !== -1) ||
hareeshnagaraj marked this conversation as resolved.
Show resolved Hide resolved
(fetchedLatestBlockNumber !== -1 && fetchedLatestBlockNumber <= latestBlockNumber)
(latestClockValue >= fetchedLatestClockVal)
) {
throw new Error(`Imported data is outdated, will not sync. Imported latestBlockNumber \
${fetchedLatestBlockNumber} Self latestBlockNumber ${latestBlockNumber}`)
${fetchedLatestBlockNumber} Self latestBlockNumber ${latestBlockNumber}. \
fetched latestClockVal: ${fetchedLatestClockVal}, self latestClockVal: ${latestClockValue}`)
hareeshnagaraj marked this conversation as resolved.
Show resolved Hide resolved
}
}

Expand Down
20 changes: 20 additions & 0 deletions creator-node/src/routes/users.js
Original file line number Diff line number Diff line change
Expand Up @@ -170,4 +170,24 @@ module.exports = function (app) {

return successResponse({ clockValue })
}))
/**
* Returns latest clock value stored in CNodeUsers entry given wallet, or -1 if no entry found
*/
app.post('/users/batch_clock_status', handleResponse(async (req, res) => {
// TODO: Limit max size
hareeshnagaraj marked this conversation as resolved.
Show resolved Hide resolved
const { walletPublicKeys } = req.body
const cnodeUsers = await models.CNodeUser.findAll({
where: {
walletPublicKey: {
[models.Sequelize.Op.in]: walletPublicKeys
}
hareeshnagaraj marked this conversation as resolved.
Show resolved Hide resolved
}
})
return successResponse({
users: cnodeUsers.map(cnodeUser => ({
walletPublicKey: cnodeUser.walletPublicKey,
clock: cnodeUser.clock
hareeshnagaraj marked this conversation as resolved.
Show resolved Hide resolved
}))
})
}))
}
Loading