Content Node Chain of Trust [Phase 2] - automatic node self-registration on URSM (And new Content Node Init pattern)

[SidSethi]

Description

What is the purpose of this PR? What is the current behavior? New behavior? Relevant links (e.g. Trello) and/or information pertaining to PR?

Linear card
Notion doc

The PR for Phase 1 of this task is #1260

This is Phase 2 of chain of trust authentication flow for content nodes to register with new UserReplicaSetManager contract (URSM) in order to be eligible to be added to user's replica sets.
This PR "activates" the code from Phase 1, by adding a CN service that automatically registers the node against URSM on startup, including requesting signatures from registered nodes using the code from Phase 1 above.

Included:

• new ServiceRegistry init flow, separating L1 identity recovery, SnapbackSM init, and URSMRegistration into discrete and independent steps
• new CN Libs instance config - now has internalWeb3 logged in with DelegatePrivateKey to support chain writes (required for URSMRegistration)
• new URSMRegistrationManager service, responsible for registering the node. This service is entirely backwards compatible, working before and after contract is deployed
• Moves L1 identity recovery (spID) out of SnapbackSM service for better logical separation of concerns. Now handled by ServiceRegistry
• adding rate limit to ursm_request_for_signature route
• SnapbackSM integration test fix
• update maddog integration to test to wait for all CN to register on URSM before testing - fixes race condition in maddog CI
• new "devMode" for CN, currentlyused only for URSMRegistrationRetryTimeout, but very extensible
• Bump libs version in CN to consume required changes (also pins version)
• changing local system setup to default to 4 nodes
• new local command to update SPDelegateOwnerWallet in order to test updateDelegateOwnerWallet flow to URSM
• lots of documentation

Tests

List the manual tests and repro instructions to verify that this PR works as anticipated. Include log analysis if possible.
❗ If this change impacts clients, make sure that you have tested the clients ❗

1. additional validation in Mad Dog URSM test to confirm nodes were added by correct proposers
2. Manual regression testing - run client without URSM deployed and confirm user signup + track upload + track stream + replica set re-election
3. Manual testing - run client with URSM deployed and confirm user signup + track upload + track stream + replica set re-election

Testing steps:

• bring up 4 nodes locally (without URSM contract deployed)
• confirm maddog tests pass
• test on dapp - everything should work
• deploy URSM contract (without URSM self-registration turned off)
• confirm maddog fails bc all nodes are not registered on URSM
• test on dapp
  • new user - everything should work
  • old user from above - should work unless CN4 is primary
• enable URSM self-registration
• confirm CN4 self registers successfully
• confirm maddog tests pass
• test on dapp
  • new user - everything should work (confirmed selected CN4 to rset)
  • old user from above - everything should work
• test updateDelegateOwnerWallet flow to URSM
  • confirmed rate limits for new route work

Logs below show CN4 repeatedly trying to register on URSM until it succeeds

[2021-03-10T21:33:59.673Z]  INFO: audius_creator_node/49 on 8669656ba010: Success (requestID=wL1TLTjAy, requestMethod=GET, requestHostname=cn2_creator-node_1, requestUrl=/ursm_request_for_signature, statusCode=200, duration=438)
    requestQueryParams: spID=2&timestamp=2021-03-10T21:33:59.218Z&signature=0x742f6845195ec4bad673aae03d2dbaa49a5b599372e1c4a32ac06030847a338a1a637d0d6e7af407755baad6dc22f6e7e3d7d9dba503915890b063b93b09de5f1b
[2021-03-10T21:33:59.681Z]  INFO: audius_creator_node/49 on 8669656ba010: URSMRegistrationManager || Successfully received signature from http://cn2_creator-node_1:4001
Executing (default): SELECT 1
[2021-03-10T21:33:59.713Z]  INFO: audius_creator_node/49 on 8669656ba010: URSMRegistrationManager || Successfully received signature from http://cn1_creator-node_1:4000
[2021-03-10T21:33:59.741Z]  INFO: audius_creator_node/49 on 8669656ba010: URSMRegistrationManager || Successfully received signature from http://cn4_creator-node_1:4003
Server returned error: [500] Something caused the transaction to fail for payload 6b562ad03e844d67a609adca87f9f6919320aef387e2ae123865880e746c6f99
Retry error : Error: Request failed with status code 500
Server returned error: [500] Something caused the transaction to fail for payload 6b562ad03e844d67a609adca87f9f6919320aef387e2ae123865880e746c6f99
Retry error : Error: Request failed with status code 500
Server returned error: [500] Something caused the transaction to fail for payload 6b562ad03e844d67a609adca87f9f6919320aef387e2ae123865880e746c6f99
Retry error : Error: Request failed with status code 500
[2021-03-10T21:34:15.152Z] ERROR: audius_creator_node/49 on 8669656ba010: ServiceRegistry ERROR || RegisterNodeOnL2URSM Error
[2021-03-10T21:34:25.153Z]  INFO: audius_creator_node/49 on 8669656ba010: ServiceRegistry || Attempting to register node on L2 URSM on 10000ms interval || attempt #4 ...
[2021-03-10T21:34:25.154Z]  INFO: audius_creator_node/49 on 8669656ba010: URSMRegistrationManager || Beginning URSM registration process
Executing (default): SELECT 1
Executing (default): SELECT 1
Executing (default): SELECT 1
[2021-03-10T21:34:25.506Z]  INFO: audius_creator_node/49 on 8669656ba010: URSMRegistrationManager || Successfully received signature from http://cn1_creator-node_1:4000
[2021-03-10T21:34:25.509Z]  INFO: audius_creator_node/49 on 8669656ba010: URSMRegistrationManager || Successfully received signature from http://cn3_creator-node_1:4002
[2021-03-10T21:34:25.539Z]  INFO: audius_creator_node/49 on 8669656ba010: URSMRegistrationManager || Successfully received signature from http://cn4_creator-node_1:4003
[2021-03-10T21:34:25.753Z]  INFO: audius_creator_node/49 on 8669656ba010: URSMRegistrationManager || Successfully registered self on URSM
[2021-03-10T21:34:25.755Z]  INFO: audius_creator_node/49 on 8669656ba010: ServiceRegistry || URSM Registration completed
[2021-03-10T21:34:25.755Z]  INFO: audius_creator_node/49 on 8669656ba010: ServiceRegistry || All services that require server successfully initialized!

❗ Reminder 💡❗:
If this PR touches a critical flow (such as Indexing, Uploads, Gateway or the Filesystem), make sure to add the requires-special-attention label. Add relevant labels as necessary.

[dmanjunath]

this PR looks great. had one comment about while true. i'm basically ready to approve if you change that or feel we shouldn't change that

[SidSethi]

@dmanjunath @hareeshnagaraj addressed all comments pls re-review
also CI will bring up all these and register everything so am trusting that to test my last code change instead of running whole thing locally, lmk if you want me to test locally

[hareeshnagaraj]

just to clarify, we know the order here because ursm content nodes are ordered by ID in the route from disc prov and the requesting node aggregates the proposer responses in said order?

[SidSethi]

nah we don't know the order in which the requesting node aggregates proposer responses. this code works bc it makes no assumptions on the order of those responses

[hareeshnagaraj]

Changes look great, love the code quality and thoroughness here - comments, testing, PR description 💯

[hareeshnagaraj]

LGTM!

[dmanjunath]

YESSSSSSS