Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[AUD-185 ] IPFS 0.8.0 Upgrade #1270

Merged
merged 10 commits into from
Mar 10, 2021
Merged

[AUD-185 ] IPFS 0.8.0 Upgrade #1270

merged 10 commits into from
Mar 10, 2021

Conversation

hareeshnagaraj
Copy link
Contributor

Description

What is the purpose of this PR? What is the current behavior? New behavior? Relevant links (e.g. Trello) and/or information pertaining to PR?

Functioning upgrade to ipfs - requires python version bump to 3.8.0 + explicit use of ipfshttpclient 0.6.0 (0.6.1 is not functioning).

Multiaddr parsing works as previously expected.

Tests

List the manual tests and repro instructions to verify that this PR works as anticipated. Include log analysis if possible.
❗ If this change impacts clients, make sure that you have tested the clients ❗

Full e2e setup, peering, track uploads, photo uploads all work as expected.

Follow tests from prior validation before opening PR - #1050

❗ Reminder 💡❗:
If this PR touches a critical flow (such as Indexing, Uploads, Gateway or the Filesystem), make sure to add the requires-special-attention label. Add relevant labels as necessary.

@hareeshnagaraj hareeshnagaraj added discovery-node Discovery Node (previously known as Discovery Provider) requires-special-attention This change is risky and/or critical and requires special attention in review labels Mar 3, 2021
@hareeshnagaraj
Copy link
Contributor Author

https://linear.app/audius/issue/AUD-185/upgrade-ipfs-version-to-070

@hareeshnagaraj hareeshnagaraj changed the title IPFS 0.7.0 Upgrade [AUD-185 ] IPFS 0.7.0 Upgrade Mar 3, 2021
@hareeshnagaraj hareeshnagaraj changed the title [AUD-185 ] IPFS 0.7.0 Upgrade [AUD-185 ] IPFS 0.8.0 Upgrade Mar 4, 2021
@hareeshnagaraj
Copy link
Contributor Author

3/8/21 IPFS 0.8.0 Dry Run

  • Local cluster with ipfs 0.4.23 brought up, account created, 3 song album uploaded

    IPFS_RELEASE_VERSION=ipfs/go-ipfs:v0.4.23

  • Local ipfs nodes taken down

    node setup.js run ipfs down
node setup.js run ipfs-2 down

  • Rehydrate failure confirmed through log analysis (QmUs26qT8b754xfhY1fo1XYFFUiHDvVFNcJs5LvcXW1wxT)

    • ipfs cat single byte requests failed:
    [2021-03-09T03:37:03.804Z]  INFO: audius_creator_node/51 on 275cc16539b3: **rehydrateIpfsFromFsIfNecessary - error condition met QmUs26qT8b754xfhY1fo1XYFFUiHDvVFNcJs5LvcXW1wxT**/480x480.jpg, TimeoutError: Request timed out (requestID=CvL40ehTZ, requestMethod=GET, requestHostname=cn1_creator-node_1)
    requestUrl: /ipfs/QmUs26qT8b754xfhY1fo1XYFFUiHDvVFNcJs5LvcXW1wxT/480x480.jpg
[2021-03-09T03:37:03.804Z]  INFO: audius_creator_node/51 on 275cc16539b3: rehydrateIpfsFromFsIfNecessary - Re-adding dir QmUs26qT8b754xfhY1fo1XYFFUiHDvVFNcJs5LvcXW1wxT, stg path: /file_storage/files/1wx/QmUs26qT8b754xfhY1fo1XYFFUiHDvVFNcJs5LvcXW1wxT, filename: 480x480.jpg, ipfsPath: QmUs26qT8b754xfhY1fo1XYFFUiHDvVFNcJs5LvcXW1wxT/480x480.jpg (requestID=CvL40ehTZ, requestMethod=GET, requestHostname=cn1_creator-node_1)
    requestUrl: /ipfs/QmUs26qT8b754xfhY1fo1XYFFUiHDvVFNcJs5LvcXW1wxT/480x480.jpg
Executing (default): SELECT "fileUUID", "cnodeUserUUID", "trackBlockchainId", "multihash", "sourceFile", "fileName", "dirMultihash", "storagePath", "type", "clock", "createdAt", "updatedAt" FROM "Files" AS "File" WHERE "File"."dirMultihash" = 'QmUs26qT8b754xfhY1fo1XYFFUiHDvVFNcJs5LvcXW1wxT' AND "File"."type" = 'image';
[2021-03-09T03:37:08.821Z] ERROR: audius_creator_node/51 on 275cc16539b3: rehydrateIpfsFromFsIfNecessary - addResp **Error: getaddrinfo EAI_AGAIN local**-ipfs-node local-ipfs-node:5001, [object Object],[object Object],[object Object],[object Object] (requestID=CvL40ehTZ, requestMethod=GET, requestHostname=cn1_creator-node_1)

  • Local ipfs nodes brought up with 0.8.0

    IPFS_RELEASE_VERSION=ipfs/go-ipfs:v0.8.0

  • Previously added tracks played and successful rehydrate operations verified (logs below) - no breakage in dir structure or base CID rehydration

    • Below confirms rehydrate of QmUs26qT8b754xfhY1fo1XYFFUiHDvVFNcJs5LvcXW1wxT in 0.8.0 (no cnode restart)
Track CID:
[2021-03-09T03:41:06.629Z]  INFO: audius_creator_node/51 on 275cc16539b3: rehydrateIpfsFromFsIfNecessary - Re-adding file - QmVR7E5TDazJGBBXysGX9DudKjbQiVQZvBRZ8FLz4ZhBUc, stg path: /file_storage/files/hBU/QmVR7E5TDazJGBBXysGX9DudKjbQiVQZvBRZ8FLz4ZhBUc (requestID=aaUCfQjyJ, requestMethod=GET, requestHostname=cn1_creator-node_1)
    requestUrl: /ipfs/QmVR7E5TDazJGBBXysGX9DudKjbQiVQZvBRZ8FLz4ZhBUc
[2021-03-09T03:41:06.705Z]  INFO: audius_creator_node/51 on 275cc16539b3: rehydrateIpfsFromFsIfNecessary - Re-added file - QmVR7E5TDazJGBBXysGX9DudKjbQiVQZvBRZ8FLz4ZhBUc, stg path: /file_storage/files/hBU/QmVR7E5TDazJGBBXysGX9DudKjbQiVQZvBRZ8FLz4ZhBUc,  [{"path":"QmVR7E5TDazJGBBXysGX9DudKjbQiVQZvBRZ8FLz4ZhBUc","hash":"QmVR7E5TDazJGBBXysGX9DudKjbQiVQZvBRZ8FLz4ZhBUc","size":190458}] (requestID=aaUCfQjyJ, requestMethod=GET, requestHostname=cn1_creator-node_1)
    requestUrl: /ipfs/QmVR7E5TDazJGBBXysGX9DudKjbQiVQZvBRZ8FLz4ZhBUc
[2021-03-09T03:41:12.926Z]  INFO: audius_creator_node/51 on 275cc16539b3: Monitoring Queue: Starting || active: 1, waiting: 0, failed 0, delayed: 0, completed: 0

----

Dir verification:

(http://cn1_creator-node_1:4000/ipfs/QmUs26qT8b754xfhY1fo1XYFFUiHDvVFNcJs5LvcXW1wxT/480x480.jpg)

Image CID: 

requestUrl: /ipfs/QmUs26qT8b754xfhY1fo1XYFFUiHDvVFNcJs5LvcXW1wxT/480x480.jpg
[2021-03-09T03:39:31.593Z]  INFO: audius_creator_node/51 on 275cc16539b3: rehydrateIpfsFromFsIfNecessary - error condition met QmUs26qT8b754xfhY1fo1XYFFUiHDvVFNcJs5LvcXW1wxT/480x480.jpg, TimeoutError: Request timed out (requestID=2f5uuV6g0, requestMethod=GET, requestHostname=cn1_creator-node_1)
    requestUrl: /ipfs/QmUs26qT8b754xfhY1fo1XYFFUiHDvVFNcJs5LvcXW1wxT/480x480.jpg
[2021-03-09T03:39:31.594Z]  INFO: audius_creator_node/51 on 275cc16539b3: rehydrateIpfsFromFsIfNecessary - Re-adding dir QmUs26qT8b754xfhY1fo1XYFFUiHDvVFNcJs5LvcXW1wxT, stg path: /file_storage/files/1wx/QmUs26qT8b754xfhY1fo1XYFFUiHDvVFNcJs5LvcXW1wxT, filename: 480x480.jpg, ipfsPath: QmUs26qT8b754xfhY1fo1XYFFUiHDvVFNcJs5LvcXW1wxT/480x480.jpg (requestID=2f5uuV6g0, requestMethod=GET, requestHostname=cn1_creator-node_1)
    requestUrl: /ipfs/QmUs26qT8b754xfhY1fo1XYFFUiHDvVFNcJs5LvcXW1wxT/480x480.jpg
Executing (default): SELECT "fileUUID", "cnodeUserUUID", "trackBlockchainId", "multihash", "sourceFile", "fileName", "dirMultihash", "storagePath", "type", "clock", "createdAt", "updatedAt" FROM "Files" AS "File" WHERE "File"."dirMultihash" = 'QmUs26qT8b754xfhY1fo1XYFFUiHDvVFNcJs5LvcXW1wxT' AND "File"."type" = 'image';
[2021-03-09T03:39:31.687Z]  INFO: audius_creator_node/51 on 275cc16539b3: rehydrateIpfsFromFsIfNecessary - addResp [{"path":"blob/150x150.jpg","hash":"QmP81r2Q79LSixEPUCcWof531wBnzGTjMLkAy71SBAXPyy","size":9803},{"path":"blob/480x480.jpg","hash":"QmWTACMaGg4LDz1Yn7VRB6usbZkp1WdWtgjp62enR5mn1p","size":59129},{"path":"blob/1000x1000.jpg","hash":"QmWbyasuzUyTKHSiHkaeN55zfcnvcf87DqYQV4Mk9d7Pzr","size":190734},{"path":"blob/original.jpg","hash":"QmSXBbzNBAoNj32PzLFDXbZzfGyGT3HH8dfpAUcYWdzF43","size":492996},{"path":"blob","hash":"QmUs26qT8b754xfhY1fo1XYFFUiHDvVFNcJs5LvcXW1wxT","size":752888}] (requestID=2f5uuV6g0, requestMethod=GET, requestHostname=cn1_creator-node_1)
  • New tracks added with ipfs nodes at 0.8.0
  • Playback verified

@hareeshnagaraj
Copy link
Contributor Author

3/9/21 - Staging IPFS 0.80

  • Update cn1 ipfs to 0.8.0
"/ip4/52.12.219.251/tcp/31634/p2p/QmPCcSmd7CqMzBgGHxRoMJ7koGrP9oekJX8RWH5bAjokj5",

  • Uploaded track to staging cn1 ipfs 0.8.0, confirmed that indexing on 0.4.23 is unaffected

  • Updated all staging cns to ipfs 0.8.0

    • Evaluate process
      ◦ cn2 stage rollout after health check fix - very smooth, no issues to be found
      ◦ cn3 stage rollout similarly smooth, no issues after health check update

  • Uploaded full album, manually verified sync success

cn1 peer info:

"/ip4/52.12.219.251/tcp/31634/p2p/QmPCcSmd7CqMzBgGHxRoMJ7koGrP9oekJX8RWH5bAjokj5",

cn2 peer info:

"/ip4/34.71.153.73/tcp/31634/p2p/Qmbamzumxz93b8PJ576QSsDW7o3m8YHM4peR7qM7JTkYMn",

cn3 peer info:

"/ip4/34.121.115.207/tcp/31634/p2p/QmNWdtt5gQJdQHTrrk2NvgCAn6rSnzYG9YniQqFEeYoHU5",

all ipfs versions checked after rollout: confirmed to 0.8.0

https://creatornode.staging.audius.co/ipfs_peer_info

https://creatornode2.staging.audius.co/ipfs_peer_info

https://creatornode3.staging.audius.co/ipfs_peer_info

disc prov logs confirming successful peering after above changes (0.4.23 → 0.8.0):

dp2 → cn1 (ipfs id = QmPCcSmd7CqMzBgGHxRoMJ7koGrP9oekJX8RWH5bAjokj5)

{"levelno": 20, "level": "INFO", "msg": "index_network_peers.py | Peering with https://creatornode.staging.audius.co", "timestamp": "2021-03-10 03:59:54,712"}
{"levelno": 20, "level": "INFO", "msg": "index_network_peers.py | Successfully peered with https://usermetadata.staging.audius.co", "timestamp": "2021-03-10 03:59:55,066"}
{"levelno": 30, "level": "WARNING", "msg": "returning /ip4/52.12.219.251/tcp/31634/p2p/QmPCcSmd7CqMzBgGHxRoMJ7koGrP9oekJX8RWH5bAjokj5", "timestamp": "2021-03-10 03:59:55,095"}
{"levelno": 20, "level": "INFO", "msg": "{'Strings': ['connect QmPCcSmd7CqMzBgGHxRoMJ7koGrP9oekJX8RWH5bAjokj5 success']}", "timestamp": "2021-03-10 03:59:55,103"}
{"levelno": 20, "level": "INFO", "msg": "index_network_peers.py | Successfully peered with https://creatornode.staging.audius.co", "timestamp": "2021-03-10 03:59:55,103"}

dp2 → cn3 (ipfs id = QmNWdtt5gQJdQHTrrk2NvgCAn6rSnzYG9YniQqFEeYoHU5)


```rust
{"levelno": 30, "level": "WARNING", "msg": "returning /ip4/34.121.115.207/tcp/31634/p2p/QmNWdtt5gQJdQHTrrk2NvgCAn6rSnzYG9YniQqFEeYoHU5", "timestamp": "2021-03-10 16:45:31,470"}
{"levelno": 20, "level": "INFO", "msg": "{'Strings': ['connect QmNWdtt5gQJdQHTrrk2NvgCAn6rSnzYG9YniQqFEeYoHU5 success']}", "timestamp": "2021-03-10 16:45:31,519"}

dp2 → cn2 (ipfs id = Qmbamzumxz93b8PJ576QSsDW7o3m8YHM4peR7qM7JTkYMn)

{"levelno": 30, "level": "WARNING", "msg": "returning /ip4/34.71.153.73/tcp/31634/p2p/Qmbamzumxz93b8PJ576QSsDW7o3m8YHM4peR7qM7JTkYMn", "timestamp": "2021-03-10 16:45:31,495"}
{"levelno": 20, "level": "INFO", "msg": "index_network_peers.py | Successfully peered with https://creatornode3.staging.audius.co", "timestamp": "2021-03-10 16:45:31,519"}
{"levelno": 20, "level": "INFO", "msg": "index_network_peers.py | Peering with http://13.52.101.6:31225", "timestamp": "2021-03-10 16:45:31,520"}
{"levelno": 20, "level": "INFO", "msg": "{'Strings': ['connect Qmbamzumxz93b8PJ576QSsDW7o3m8YHM4peR7qM7JTkYMn success']}", "timestamp": "2021-03-10 16:45:31,524"}

Sync success after album upload with cn 0.8.0 ipfs and disc prov 0.4.23 ipfs
https://staging.audius.co/hntester46/album/test-album-919
User wallet:

0xd5c34c34e4e599463ad9b04aa584c5bd8d4e13dd

https://discoveryprovider3.staging.audius.co/users?wallet=0xd5c34c34e4e599463ad9b04aa584c5bd8d4e13dd

"creator_node_endpoint": "[https://creatornode.staging.audius.co](https://creatornode.staging.audius.co/),[https://creatornode2.staging.audius.co](https://creatornode2.staging.audius.co/),[https://creatornode3.staging.audius.co](https://creatornode3.staging.audius.co/)",

cn1 (primary): https://creatornode.staging.audius.co/sync_status/0xd5c34c34e4e599463ad9b04aa584c5bd8d4e13dd

"walletPublicKey": "0xd5c34c34e4e599463ad9b04aa584c5bd8d4e13dd",
    "latestBlockNumber": 19787481,
    "clockValue": 240

cn2 (s1)

https://creatornode2.staging.audius.co/sync_status/0xd5c34c34e4e599463ad9b04aa584c5bd8d4e13dd

"data": {
    "walletPublicKey": "0xd5c34c34e4e599463ad9b04aa584c5bd8d4e13dd",
    "latestBlockNumber": 19787481,
    "clockValue": 240
  },

cn3 (s2)

https://creatornode3.staging.audius.co/sync_status/0xd5c34c34e4e599463ad9b04aa584c5bd8d4e13dd

"walletPublicKey": "0xd5c34c34e4e599463ad9b04aa584c5bd8d4e13dd",
    "latestBlockNumber": 19787481,
    "clockValue": 240
  },

@hareeshnagaraj
Copy link
Contributor Author

Final tasks here - update local dev with 0.4.23 for disc prov explicitly (add a new IPFS node), investigate why this isn't breaking on local dev

@hareeshnagaraj
Copy link
Contributor Author

hareeshnagaraj commented Mar 10, 2021 •
edited

Final tasks here - update local dev with 0.4.23 for disc prov explicitly (add a new IPFS node), investigate why this isn't breaking on local dev

The reason why discovery provider local dev did not break with ipfs 0.8.0 upgrade or 0.7.0 is because there is a third node dedicated to discovery provider which was still using go-ipfs:release (not sure why this didn't resolve to a higher version and break), but we will now explicitly pin 0.4.23 for disc prov local to reflect the system - disc provs on 0.4.23 and content nodes on 0.8.0

@hareeshnagaraj hareeshnagaraj marked this pull request as ready for review March 10, 2021 20:37
dmanjunath
dmanjunath approved these changes Mar 10, 2021
View reviewed changes
Copy link
Contributor

@dmanjunath dmanjunath left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good stuff

@hareeshnagaraj hareeshnagaraj merged commit 33b4c6d into master Mar 10, 2021
@hareeshnagaraj hareeshnagaraj deleted the hn_ipfs_update_2 branch March 10, 2021 21:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dmanjunath dmanjunath dmanjunath approved these changes

Assignees
No one assigned
Labels
discovery-node Discovery Node (previously known as Discovery Provider) requires-special-attention This change is risky and/or critical and requires special attention in review
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@hareeshnagaraj @dmanjunath