Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AUD-391] In content-node track stream, add a fallback to enable streaming if track association didn't complete #1284

Merged
merged 6 commits into from
Mar 9, 2021

Conversation

dmanjunath
Copy link
Contributor

@dmanjunath dmanjunath commented Mar 9, 2021

Description

https://www.notion.so/audiusproject/Content-Node-Stream-Route-Bugfix-b585d68f89b24161a776bc70447eae7c
For a small subset of users, the track association doesn't complete. That means they don't have a record in the Tracks table. However, that doesn't mean that we can't gather enough information to stream the track. This removes the check in the Tracks table since that could fail in a small number of cases and adds fallback checks to discovery to get the metadata multihash and then attempts to lookup the copy320 file with a matching sourceFile from the trackSegments. There's also a redis cache to prevent an excessive number of discovery calls

Tests

Manually set the fileRecord to null and check that it fetches the data and does the logic successfully from discovery. Subsequent loads should use the redis cache. Attached are logs checking the regular flow and the cached flow. Look for the arrows, which are relevant comments

[2021-03-09T00:16:15.023Z]  INFO: audius_creator_node/1671 on 0edbb720c848: Begin processing request (requestID=_kSFMD92x, requestMethod=GET, requestHostname=cn3_creator-node_1, requestUrl=/tracks/stream/7eP5n)
------> [2021-03-09T00:16:15.027Z]  INFO: audius_creator_node/1671 on 0edbb720c848: Could not find a stream fallback in redis, need to go to discovery
Executing (default): SELECT "fileUUID", "cnodeUserUUID", "trackBlockchainId", "multihash", "sourceFile", "fileName", "dirMultihash", "storagePath", "type", "clock", "createdAt", "updatedAt" FROM "Files" AS "File" WHERE "File"."multihash" = 'QmZPPGjs8bfQJKoNrtJs2WfC1HyZfvwe1zUf7sxGiYgNC6' LIMIT 1;
Executing (default): SELECT "sourceFile" FROM "Files" AS "File" WHERE "File"."multihash" IN ('QmXyhD7n97McnD9L471QJPwycjYisx2cY7hLXUeS6rMoDU', 'QmfVbTd4JfznqCsN2y36AS6gGG5WprDZvNZnrNvb154th4', 'QmasHbK7YKsyBTDchp5qkeLecT2AKQynbm4NQBJqsW6AeL', 'QmWXPZcRWce5Z2jLXTj8KcpPCD5HMSZm3BCpyAJ4NxZK7y', 'QmXRqxJbTpF81V2LWgYkWPbTPNTnidjophQWXCfaBVw1KK', 'QmbGqLaaKFfZ1dxYR9b1E66NJEadtJPeswV4YYgRN6MEcA', 'Qmds2uaqfUpVMtUw2LqQ55dQqW9Knbc1p4aG9e8FuYUmfd', 'QmPA1Qyf6jiB8BtNf6mvcveuGWNE1SdU7YYLBuwSW9eRi9', 'QmTdQGjs9s7EEWLMheu5PLHuwmJDZjNCnvoYpzZ9Sc57HA', 'QmbfM8e7gVHSyXA4wojzH5ur7Qh4NtaF2hwCbcPeMPoSsL', 'QmUfkLxDFwxDncd577ACZQYKcF3zstnYrcDxukLaSMABao', 'QmPwohng8jejzeCnRnihNN1YL3FxYDrD8jhAadfWWtrSnu', 'QmPqidBAbBLsWAzNbf2iT3oGy3NamNTagkAR8e5aofPCq3', 'QmPYwQXuPJW2MzSfeL1P2L6opRPHEBxSELSksQfuGRV45N') AND "File"."cnodeUserUUID" = '5f44a6b5-7844-4226-8ffb-187440f10157';
Executing (default): SELECT "fileUUID", "cnodeUserUUID", "trackBlockchainId", "multihash", "sourceFile", "fileName", "dirMultihash", "storagePath", "type", "clock", "createdAt", "updatedAt" FROM "Files" AS "File" WHERE "File"."type" = 'copy320' AND "File"."sourceFile" = 'a247a245-729a-40dc-a7f8-01572dd91764.mp3' LIMIT 1;
[2021-03-09T00:16:15.101Z]  INFO: audius_creator_node/1671 on 0edbb720c848: Logging listen for track 1 by 0xb9819eaeb8a63a3cce101ad3682665dac2c7e0d4 (requestID=_kSFMD92x, requestMethod=GET, requestHostname=cn3_creator-node_1, requestUrl=/tracks/stream/7eP5n)
[2021-03-09T00:16:15.113Z]  INFO: audius_creator_node/1671 on 0edbb720c848: IPFS Standalone Request - QmXAnZ6PBMQqycT1yZzyAsh3BJQZy6UZh8XYxgLB6TrpX3 (requestID=_kSFMD92x, requestMethod=GET, requestHostname=cn3_creator-node_1, requestUrl=/tracks/stream/7eP5n)
[2021-03-09T00:16:15.114Z]  INFO: audius_creator_node/1671 on 0edbb720c848: IPFS Stats - Standalone Requests: 5 (requestID=_kSFMD92x, requestMethod=GET, requestHostname=cn3_creator-node_1, requestUrl=/tracks/stream/7eP5n)
[2021-03-09T00:16:15.118Z]  INFO: audius_creator_node/1671 on 0edbb720c848: Retrieving /file_storage/files/rpX/QmXAnZ6PBMQqycT1yZzyAsh3BJQZy6UZh8XYxgLB6TrpX3 directly from filesystem (requestID=_kSFMD92x, requestMethod=GET, requestHostname=cn3_creator-node_1, requestUrl=/tracks/stream/7eP5n)
(node:1671) UnhandledPromiseRejectionWarning: Error: Request failed with status code 429
    at createError (/usr/src/app/node_modules/axios/lib/core/createError.js:16:15)
    at settle (/usr/src/app/node_modules/axios/lib/core/settle.js:17:12)
    at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/axios/lib/adapters/http.js:237:11)
    at IncomingMessage.emit (events.js:203:15)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
(node:1671) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 10)
[2021-03-09T00:16:15.125Z] DEBUG: audius_creator_node/1671 on 0edbb720c848: RehydrateIpfsQueue: Adding a rehydrateIpfsFromFsIfNecessary task to the queue!, count: 0 (requestID=_kSFMD92x, requestMethod=GET, requestHostname=cn3_creator-node_1, requestUrl=/tracks/stream/7eP5n)
[2021-03-09T00:16:15.132Z] DEBUG: audius_creator_node/1671 on 0edbb720c848: RehydrateIpfsQueue: Successfully added a rehydrateIpfsFromFsIfNecessary task!, count: 0 (requestID=_kSFMD92x, requestMethod=GET, requestHostname=cn3_creator-node_1, requestUrl=/tracks/stream/7eP5n)
[2021-03-09T00:16:15.135Z] DEBUG: audius_creator_node/1671 on 0edbb720c848: RehydrateIpfsQueue: Processing a rehydrateIpfsFromFsIfNecessary task for QmXAnZ6PBMQqycT1yZzyAsh3BJQZy6UZh8XYxgLB6TrpX3, count: 0 (requestID=_kSFMD92x, requestMethod=GET, requestHostname=cn3_creator-node_1, requestUrl=/tracks/stream/7eP5n)
[2021-03-09T00:16:15.188Z]  INFO: audius_creator_node/1671 on 0edbb720c848: ipfsSingleByteCat - Retrieved QmXAnZ6PBMQqycT1yZzyAsh3BJQZy6UZh8XYxgLB6TrpX3 in 51ms (requestID=_kSFMD92x, requestMethod=GET, requestHostname=cn3_creator-node_1, requestUrl=/tracks/stream/7eP5n)
[2021-03-09T00:16:15.378Z]  INFO: audius_creator_node/1671 on 0edbb720c848: Begin processing request (requestID=W47GIaWqP, requestMethod=GET, requestHostname=cn3_creator-node_1, requestUrl=/tracks/stream/7eP5n)
------> [2021-03-09T00:16:15.382Z]  INFO: audius_creator_node/1671 on 0edbb720c848: Found the stream in redis! No need to hit discovery
[2021-03-09T00:16:15.394Z]  INFO: audius_creator_node/1671 on 0edbb720c848: Logging listen for track 1 by 0xb9819eaeb8a63a3cce101ad3682665dac2c7e0d4 (requestID=W47GIaWqP, requestMethod=GET, requestHostname=cn3_creator-node_1, requestUrl=/tracks/stream/7eP5n)
[2021-03-09T00:16:15.408Z]  INFO: audius_creator_node/1671 on 0edbb720c848: IPFS Standalone Request - QmXAnZ6PBMQqycT1yZzyAsh3BJQZy6UZh8XYxgLB6TrpX3 (requestID=W47GIaWqP, requestMethod=GET, requestHostname=cn3_creator-node_1, requestUrl=/tracks/stream/7eP5n)
[2021-03-09T00:16:15.409Z]  INFO: audius_creator_node/1671 on 0edbb720c848: IPFS Stats - Standalone Requests: 6 (requestID=W47GIaWqP, requestMethod=GET, requestHostname=cn3_creator-node_1, requestUrl=/tracks/stream/7eP5n)
[2021-03-09T00:16:15.412Z]  INFO: audius_creator_node/1671 on 0edbb720c848: Retrieving /file_storage/files/rpX/QmXAnZ6PBMQqycT1yZzyAsh3BJQZy6UZh8XYxgLB6TrpX3 directly from filesystem (requestID=W47GIaWqP, requestMethod=GET, requestHostname=cn3_creator-node_1, requestUrl=/tracks/stream/7eP5n)
(node:1671) UnhandledPromiseRejectionWarning: Error: Request failed with status code 429
    at createError (/usr/src/app/node_modules/axios/lib/core/createError.js:16:15)
    at settle (/usr/src/app/node_modules/axios/lib/core/settle.js:17:12)
    at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/axios/lib/adapters/http.js:237:11)
    at IncomingMessage.emit (events.js:203:15)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
(node:1671) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 12)
[2021-03-09T00:16:15.416Z] DEBUG: audius_creator_node/1671 on 0edbb720c848: RehydrateIpfsQueue: Adding a rehydrateIpfsFromFsIfNecessary task to the queue!, count: 0 (requestID=W47GIaWqP, requestMethod=GET, requestHostname=cn3_creator-node_1, requestUrl=/tracks/stream/7eP5n)
[2021-03-09T00:16:15.421Z] DEBUG: audius_creator_node/1671 on 0edbb720c848: RehydrateIpfsQueue: Successfully added a rehydrateIpfsFromFsIfNecessary task!, count: 0 (requestID=W47GIaWqP, requestMethod=GET, requestHostname=cn3_creator-node_1, requestUrl=/tracks/stream/7eP5n)
[2021-03-09T00:16:15.422Z] DEBUG: audius_creator_node/1671 on 0edbb720c848: RehydrateIpfsQueue: Processing a rehydrateIpfsFromFsIfNecessary task for QmXAnZ6PBMQqycT1yZzyAsh3BJQZy6UZh8XYxgLB6TrpX3, count: 0 (requestID=W47GIaWqP, requestMethod=GET, requestHostname=cn3_creator-node_1, requestUrl=/tracks/stream/7eP5n)
[2021-03-09T00:16:15.440Z]  INFO: audius_creator_node/1671 on 0edbb720c848: ipfsSingleByteCat - Retrieved QmXAnZ6PBMQqycT1yZzyAsh3BJQZy6UZh8XYxgLB6TrpX3 in 17ms (requestID=W47GIaWqP, requestMethod=GET, requestHostname=cn3_creator-node_1, requestUrl=/tracks/stream/7eP5n)
[2021-03-09T00:16:15.470Z]  INFO: audius_creator_node/1671 on 0edbb720c848: Begin processing request (requestID=bu0mOvskFV, requestMethod=GET, requestHostname=cn3_creator-node_1, requestUrl=/favicon.ico)
[2021-03-09T00:16:43.592Z]  INFO: audius_creator_node/1671 on 0edbb720c848: Begin processing request (requestID=2HQxgQmt6, requestMethod=GET, requestHostname=cn3_creator-node_1, requestUrl=/tracks/stream/7eP5n)
------> [2021-03-09T00:16:43.595Z]  INFO: audius_creator_node/1671 on 0edbb720c848: Found the stream in redis! No need to hit discovery
[2021-03-09T00:16:43.596Z]  INFO: audius_creator_node/1671 on 0edbb720c848: Logging listen for track 1 by 0xb9819eaeb8a63a3cce101ad3682665dac2c7e0d4 (requestID=2HQxgQmt6, requestMethod=GET, requestHostname=cn3_creator-node_1, requestUrl=/tracks/stream/7eP5n)
[2021-03-09T00:16:43.609Z]  INFO: audius_creator_node/1671 on 0edbb720c848: IPFS Standalone Request - QmXAnZ6PBMQqycT1yZzyAsh3BJQZy6UZh8XYxgLB6TrpX3 (requestID=2HQxgQmt6, requestMethod=GET, requestHostname=cn3_creator-node_1, requestUrl=/tracks/stream/7eP5n)
[2021-03-09T00:16:43.613Z]  INFO: audius_creator_node/1671 on 0edbb720c848: IPFS Stats - Standalone Requests: 7 (requestID=2HQxgQmt6, requestMethod=GET, requestHostname=cn3_creator-node_1, requestUrl=/tracks/stream/7eP5n)
[2021-03-09T00:16:43.617Z]  INFO: audius_creator_node/1671 on 0edbb720c848: Retrieving /file_storage/files/rpX/QmXAnZ6PBMQqycT1yZzyAsh3BJQZy6UZh8XYxgLB6TrpX3 directly from filesystem (requestID=2HQxgQmt6, requestMethod=GET, requestHostname=cn3_creator-node_1, requestUrl=/tracks/stream/7eP5n)
(node:1671) UnhandledPromiseRejectionWarning: Error: Request failed with status code 429
    at createError (/usr/src/app/node_modules/axios/lib/core/createError.js:16:15)
    at settle (/usr/src/app/node_modules/axios/lib/core/settle.js:17:12)
    at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/axios/lib/adapters/http.js:237:11)
    at IncomingMessage.emit (events.js:203:15)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
(node:1671) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 14)
[2021-03-09T00:16:43.621Z] DEBUG: audius_creator_node/1671 on 0edbb720c848: RehydrateIpfsQueue: Adding a rehydrateIpfsFromFsIfNecessary task to the queue!, count: 0 (requestID=2HQxgQmt6, requestMethod=GET, requestHostname=cn3_creator-node_1, requestUrl=/tracks/stream/7eP5n)
[2021-03-09T00:16:43.627Z] DEBUG: audius_creator_node/1671 on 0edbb720c848: RehydrateIpfsQueue: Successfully added a rehydrateIpfsFromFsIfNecessary task!, count: 0 (requestID=2HQxgQmt6, requestMethod=GET, requestHostname=cn3_creator-node_1, requestUrl=/tracks/stream/7eP5n)
[2021-03-09T00:16:43.628Z] DEBUG: audius_creator_node/1671 on 0edbb720c848: RehydrateIpfsQueue: Processing a rehydrateIpfsFromFsIfNecessary task for QmXAnZ6PBMQqycT1yZzyAsh3BJQZy6UZh8XYxgLB6TrpX3, count: 0 (requestID=2HQxgQmt6, requestMethod=GET, requestHostname=cn3_creator-node_1, requestUrl=/tracks/stream/7eP5n)
[2021-03-09T00:16:43.657Z]  INFO: audius_creator_node/1671 on 0edbb720c848: ipfsSingleByteCat - Retrieved QmXAnZ6PBMQqycT1yZzyAsh3BJQZy6UZh8XYxgLB6TrpX3 in 27ms (requestID=2HQxgQmt6, requestMethod=GET, requestHostname=cn3_creator-node_1, requestUrl=/tracks/stream/7eP5n)
[2021-03-09T00:16:43.907Z]  INFO: audius_creator_node/1671 on 0edbb720c848: Begin processing request (requestID=5-FBdZRZlo, requestMethod=GET, requestHostname=cn3_creator-node_1, requestUrl=/favicon.ico)

@cheran-senthil cheran-senthil self-assigned this Mar 9, 2021
raymondjacobson
raymondjacobson previously approved these changes Mar 9, 2021
creator-node/src/routes/tracks.js Outdated Show resolved Hide resolved
}

const { multihash } = await models.File.findOne({
let fileRecord = await models.File.findOne({
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

}

// make sure all track segments have the same sourceFile
const segments = metadataJSON.track_segments.map(segment => segment.multihash)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why all these sanity checks? it would seem maybe better to err on the side of it being ok if the segments don't line up? / does this really happen often?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it should almost never happen but this is useful bc if we pull the sourcefile off any random multihash file entry, it could map to multiple tracks (only if all uploaded by same cnodeUSerUUID) and we could end up streaming the wrong track. given that this is infrequent it seems valuable to do? like all of this should be < 1 sec only the first time request before its cached

but also can see coutnerarg

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the only case when this should happen is if the user uploads the same song multiple times and it remains unassociated after multiple failed attempts. we can just treat this as warnings and log out and continue, but the tradeoff is if there was a remix etc, there's an edge case where we could play the wrong song if a CID overlaps

Copy link
Contributor

@SidSethi SidSethi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to release, aside from ray's comment about only hitting redis if not already found from db

some comments but not a blocker for hotfix release, as long as we address before master merge

creator-node/src/routes/tracks.js Outdated Show resolved Hide resolved
creator-node/src/routes/tracks.js Outdated Show resolved Hide resolved
creator-node/src/routes/tracks.js Outdated Show resolved Hide resolved
creator-node/src/routes/tracks.js Outdated Show resolved Hide resolved
}

// make sure all track segments have the same sourceFile
const segments = metadataJSON.track_segments.map(segment => segment.multihash)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it should almost never happen but this is useful bc if we pull the sourcefile off any random multihash file entry, it could map to multiple tracks (only if all uploaded by same cnodeUSerUUID) and we could end up streaming the wrong track. given that this is infrequent it seems valuable to do? like all of this should be < 1 sec only the first time request before its cached

but also can see coutnerarg

Copy link
Contributor

@SidSethi SidSethi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for fixing the redis check. again, good to go for for hotfix, can look over again pre master merge, can prob clean some stuff up before

creator-node/src/routes/tracks.js Show resolved Hide resolved
creator-node/src/routes/tracks.js Outdated Show resolved Hide resolved
@dmanjunath
Copy link
Contributor Author

@raymondjacobson @SidSethi can. you look again? just addressed both your comments. also i realized i didn't branch off the current cn hotfix, so let me make that branch too. this we can use to merge onto master

@dmanjunath dmanjunath changed the title In content-node track stream, add a fallback to enable streaming if track association didn't complete [AUD-391] In content-node track stream, add a fallback to enable streaming if track association didn't complete Mar 9, 2021
Copy link
Contributor

@SidSethi SidSethi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

incredible, lets 🚢

@dmanjunath dmanjunath merged commit 86a5d86 into master Mar 9, 2021
@dmanjunath dmanjunath deleted the dm-vss-stream-fallback-hotfix branch March 9, 2021 20:01
@AudiusProject AudiusProject deleted a comment from linear bot Sep 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants