Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CON-281] - Add CNodeToSpIdMapManager init in stateMachineManager init #3553

Merged
merged 2 commits into from
Jul 27, 2022

Conversation

vicky-g
Copy link
Contributor

@vicky-g vicky-g commented Jul 26, 2022

Description

On deploy, the cadence of which bull adds jobs does not change. Therefore, even though the logic is correct, the job wouldnt run immediately on deploy. This was thus causing the class to not have the proper data to perform reconfig and propagating false negatives that there are mismatched spIDs on staging.

This PR is to always init the class on stateMachineManager init

Tests

Deployed to stage content node 8 with no change in fetch cnode to spID mapping and no more mismatch spIDs occurred following the deploy.

Monitoring - How will this change be monitored? Are there sufficient logs / alerts?

look for the prefix _findReplicaSetUpdatesForUser(): and mismatched spIDlog

@vicky-g vicky-g requested a review from theoilie July 26, 2022 23:09
@vicky-g vicky-g marked this pull request as ready for review July 26, 2022 23:09
@vicky-g vicky-g changed the title Add cnodetospidmapmanager init Add CNodeToSpIdMapManager init in stateMachineManager Jul 26, 2022
@vicky-g vicky-g changed the title Add CNodeToSpIdMapManager init in stateMachineManager Add CNodeToSpIdMapManager init in stateMachineManager init Jul 26, 2022
@vicky-g vicky-g changed the title Add CNodeToSpIdMapManager init in stateMachineManager init [CON-281] - Add CNodeToSpIdMapManager init in stateMachineManager init Jul 26, 2022
@dmanjunath
Copy link
Contributor

@vicky-g so your theory is there's a race condition here where it tries to find reconfigs before the mapping exists? this fix makes sense in that case

Copy link
Contributor

@dmanjunath dmanjunath left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks good to me but will let @theoilie give the final approval

@vicky-g
Copy link
Contributor Author

vicky-g commented Jul 26, 2022

@vicky-g so your theory is there's a race condition here where it tries to find reconfigs before the mapping exists? this fix makes sense in that case

not a race condition, but when the content node is deployed, the bull queue doesnt initialize by adding a job and instead maintains the current cadence. so the existing class data just gets blasted until the next update cnode to sp id mapping happens (on staging, within <= 4hours, since the job runs every 4 hours)

Copy link
Contributor

@theoilie theoilie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes sense! now I can close #3551 since it fixes the same test

@vicky-g vicky-g merged commit 6f4ee64 into master Jul 27, 2022
@vicky-g vicky-g deleted the vg-fix-mismatch-spID-3 branch July 27, 2022 03:15
audius-infra pushed a commit that referenced this pull request Jul 27, 2022
## Changelog

- 2022-07-27 [bca55d5] Dont set the nginx cache in the application layer (#3561) [Dheeraj Manjunath]
- 2022-07-27 [5d147c5] Bump to version 0.3.63 (#3559) [Cheran]
- 2022-07-27 [d0cb450] Unpin Python to fix discovery build (#3558) [Cheran]
- 2022-07-27 [b91aa19] Factor _score in to script_score script (#3556) [Steve Perkins]
- 2022-07-27 [36dcb1b] [CON-201] Remove QueueInterfacer + increase RS update concurrency (#3554) [Theo Ilie]
- 2022-07-27 [f9ac2dd] [CON-283] Add upload success metrics (#3546) [Johannes Naylor]
- 2022-07-26 [6f4ee64] [CON-281] - Add CNodeToSpIdMapManager init in stateMachineManager init (#3553) [vicky :)]
- 2022-07-26 [57560fb] [CON-294] Make manual sync queue less backed up (#3549) [Theo Ilie]
- 2022-07-26 [87c4f2c] Log cnode to sp id map when determining replica set updates (#3550) [vicky :)]
- 2022-07-26 [bd76c8e] Fix diffing (#3548) [Sebastian Klingler]
- 2022-07-23 [5b31feb] Add container-side pidfile for celery (#3544) [Raymond Jacobson]
- 2022-07-23 [bf82366] State machine - split queues and fix many bugs (#3539) [Theo Ilie]
- 2022-07-22 [8a1b901] Fix prometheus metrics name error (#3543) [Johannes Naylor]
- 2022-07-22 [0bec2b4] Add prometheus metrics for `secondarySyncFromPrimary` (#3532) [Johannes Naylor]
- 2022-07-22 [8dfd837] Fix bucketing fn + variable name change (#3534) [vicky :)]
- 2022-07-22 [0b895ab] Add force_export query param to /export route for MergePrimaryAndSecondary (#3541) [Sid Sethi]
- 2022-07-22 [0e5d0dc] Skip bad notifs (#3540) [Saliou Diallo]
- 2022-07-22 [5a007c3] CN - Fix export bug (#3537) [Sid Sethi]
- 2022-07-22 [1050286] [CON-265] Add percentage based metrics to network monitoring (#3525) [Johannes Naylor]
- 2022-07-22 [1ea1dbb] [PAY-433] Fixed Ray's broken prod account (#3523) [Reed]
- 2022-07-22 [9048086] Only track metrics for explicit routes (#3527) [vicky :)]
- 2022-07-22 [bc0256a] Add error msg to a write quorum error flow (#3535) [vicky :)]
audius-infra pushed a commit that referenced this pull request Jul 27, 2022
## Changelog

- 2022-07-27 [ad5188c] CircleCI GCP image bake env var was undefined (#3562) [Dheeraj Manjunath]
- 2022-07-27 [bca55d5] Dont set the nginx cache in the application layer (#3561) [Dheeraj Manjunath]
- 2022-07-27 [5d147c5] Bump to version 0.3.63 (#3559) [Cheran]
- 2022-07-27 [d0cb450] Unpin Python to fix discovery build (#3558) [Cheran]
- 2022-07-27 [b91aa19] Factor _score in to script_score script (#3556) [Steve Perkins]
- 2022-07-27 [36dcb1b] [CON-201] Remove QueueInterfacer + increase RS update concurrency (#3554) [Theo Ilie]
- 2022-07-27 [f9ac2dd] [CON-283] Add upload success metrics (#3546) [Johannes Naylor]
- 2022-07-26 [6f4ee64] [CON-281] - Add CNodeToSpIdMapManager init in stateMachineManager init (#3553) [vicky :)]
- 2022-07-26 [57560fb] [CON-294] Make manual sync queue less backed up (#3549) [Theo Ilie]
- 2022-07-26 [87c4f2c] Log cnode to sp id map when determining replica set updates (#3550) [vicky :)]
- 2022-07-26 [bd76c8e] Fix diffing (#3548) [Sebastian Klingler]
- 2022-07-23 [5b31feb] Add container-side pidfile for celery (#3544) [Raymond Jacobson]
- 2022-07-23 [bf82366] State machine - split queues and fix many bugs (#3539) [Theo Ilie]
- 2022-07-22 [8a1b901] Fix prometheus metrics name error (#3543) [Johannes Naylor]
- 2022-07-22 [0bec2b4] Add prometheus metrics for `secondarySyncFromPrimary` (#3532) [Johannes Naylor]
- 2022-07-22 [8dfd837] Fix bucketing fn + variable name change (#3534) [vicky :)]
- 2022-07-22 [0b895ab] Add force_export query param to /export route for MergePrimaryAndSecondary (#3541) [Sid Sethi]
- 2022-07-22 [0e5d0dc] Skip bad notifs (#3540) [Saliou Diallo]
- 2022-07-22 [5a007c3] CN - Fix export bug (#3537) [Sid Sethi]
- 2022-07-22 [1050286] [CON-265] Add percentage based metrics to network monitoring (#3525) [Johannes Naylor]
- 2022-07-22 [1ea1dbb] [PAY-433] Fixed Ray's broken prod account (#3523) [Reed]
- 2022-07-22 [9048086] Only track metrics for explicit routes (#3527) [vicky :)]
- 2022-07-22 [bc0256a] Add error msg to a write quorum error flow (#3535) [vicky :)]
sliptype pushed a commit that referenced this pull request Sep 10, 2023
…3553)

Co-authored-by: Marcus Pasell <3690498+rickyrombo@users.noreply.github.com>
sliptype pushed a commit that referenced this pull request Sep 10, 2023
[a16abc2] Revert force mobile full staging build (#3564) nicoback2
[27f0a1b] [PAY-1386] DMs: Mark chat as read on client when new message from self comes through websocket (#3562) Marcus Pasell
[ab41a52] Force mobile build again (#3561) nicoback2
[ac383dc] DMs: Fix broken messages (#3559) Marcus Pasell
[29b49ee] Force mobile staging build - Android (#3560) nicoback2
[57ff5c5] Force mobile staging build (#3558) nicoback2
[5456271] Update plist to 1.1.66 (#3557) Marcus Pasell
[48c70b1] [PAY-1398] Fix ChatUserListScreen scrolling with keyboard (#3552) Reed
[ba19c1d] [PAY-1423] Fix empty chat message box with currently playing track (#3553) Reed
[970224f] Mobile ChatMessageListItem uses message id + selector (#3551) Reed
[49ff351] [PAY-1426] DMs: Simplify unfurl styles for mobile, fix border radius shadow (#3555) Marcus Pasell
[63faad0] [PAY-1403] Fix chat track unfurl reactions (#3550) Reed
[a8e92b7] [PAY-1425][PAY-1424] DMs: Fix image unfurls, fix purple halo around unfurls (#3554) Marcus Pasell
[940aebe] [PAY-1367][PAY-1400][PAY-1401][PAY-1402][PAY-1419] Fix chat tile unfurl bugs (#3541) Saliou Diallo
[61b6735] [PAY-1378] Cap image size on mobile DMs (#3548) Reed
[a495c35] [PAY-1409] DMs: Deep link into chat from notification (#3549) Marcus Pasell
[a4e6426] [PAY-1393] DMs: Hide message underneath popover (#3547) Marcus Pasell
[a6711f1] [C-2691] Carry values through the form fields (#3539) Andrew Mendelsohn
[d653f5c] [C-2747] Fix track title layout with offline + ai attribution (#3546) Andrew Mendelsohn
[8e54ff7] [PAY-1408][PAY-1351][PAY-1383][PAY-1417][PAY-1240][PAY-1345] DMs: Web UI polishes and bugfixes (#3543) Marcus Pasell
[cc1bc46] [C-2746] Fix favorites album tab styling (#3544) Kyle Shanks
[1dbb753] [PAY-1406][PAY-1374][PAY-1353][PAY-1411] Fix DMs UI Issues (#3542) Michael Piazza
[b76eef2] [PAY-1348] clean up ChatUserListScreen and use followers as default list (#3538) Randy Schott
[3ed58ac] DMs: Wait for tip confirmation then refetch permissions (#3537) Marcus Pasell
[bb4f317] [PAY-1366] Fix web chat tile artwork and mobile link preview bugs (#3528) Saliou Diallo
[ea47ecf] DMs: Fix deletions not causing perms rechecks (#3540) Marcus Pasell
[0b8c993] [PAY-1356] Mobile chats scroll to bottom when re-entering chat screen (#3536) Reed
[52f7b94] [PAY-1363] DMs: Add CTA to Create Chat user rows (#3530) Marcus Pasell
[4c22b12] [C-2461, C-2476] Prevent user from dragging hidden tracks to public playlists (#3535) Kyle Shanks
[f2ad1ce] [C-1873, C-2551, C-2552, C-2553] Add hidden track logic for playlists on web (#3529) Kyle Shanks
[083b0ff] [PAY-1315] Fix mobile chat search screen takes 2 presses (#3534) Reed
[da64d93] [PAY-1347] DMs: Don't show unread indicator for first messages (#3532) Marcus Pasell
[7ae88cb] [PAY-1394] DMs: Show global unread indicator if there's unread messages in a loaded chat (#3533) Marcus Pasell
[40abf58] [PAY-1350] Update send button styling on web (#3523) Michael Piazza
[cd1e238] [PAY-1149] Mobile chat reactions add android offset (#3531) Reed
[3595ce8] [PAY-1362] Mobile press chat message closes reaction popup (#3527) Reed
[06d10e2] [PAY-1361] Fix reactions button hit area (#3524) Michael Piazza
[e97ed82] [PAY-1309] Use proper tracking of hasMore for chats list (#3517) Randy Schott
[fe72398] [PAY-1365] Disable fullscreen gestures on mobile chat screen (#3526) Reed
[03cbcac] [PAY-1392] Mobile mark chat as read onscroll, on navigate away (#3522) Reed
[e6af9d7] [PAY-919][PAY-1312] Chat track and playlist tiles on mobile (#3455) Saliou Diallo
[e2896fe] [C-2719] Clean up and document non v1 usage (#3520) Raymond Jacobson
[2cf5bb7] Add reason to account slice, handle fetch account failure with sentry (#3525) Raymond Jacobson
[4451447] [PAY-1342][PAY-1360] DMs: Show users as chat unlocked by default, optimistic chat navigation (#3514) Marcus Pasell
[8262221] [PAY-1391] ensure all collections are fetched before attempting to download (#3521) Randy Schott
[e29d35f] [PAY-1385] Mobile chat send icon updates (#3516) Reed
[8cda7b2] [PAY-1390] fix inverted condition for computing track count (#3519) Randy Schott
[3a3b206] [PAY-1333] Mobile dms end of history only shows with >10 messages (#3515) Reed
[f1f5dc0] [PAY-1370] Fix reaction touch targets (#3513) Reed
[9b81487] Upgrade sdk to 2.0.3-beta.36 (#3512) Dylan Jeffers
[b48777a] Remove content node cid query route (#3408) Michelle Brier
[69e5656] [C-2717] Fix playlist create tile bug (#3510) Kyle Shanks
[828015f] Remove yield* in js files (#3511) Sebastian Klingler
[7d26233] Add mobile hidden track playlist logic (#3505) Kyle Shanks
[cf9c5d5] [PAY-1137][PAY-1132] Infinite loading of albums on mobile favorites screen (#3398) Randy Schott
@AudiusProject AudiusProject deleted a comment from linear bot Sep 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants