perf: decouple user used storage sql function #2069
Conversation
vasco-santos
force-pushed
the
perf/decouple-user-used-storage-sql-function
branch
20 times, most recently
from
736a76f
to
25cc8d3
Compare
vasco-santos
force-pushed
the
perf/decouple-user-used-storage-sql-function
branch
from
25cc8d3
to
01bb640
Compare
|
|
||
| CREATE OR REPLACE FUNCTION user_used_storage(query_user_id BIGINT) |
There was a problem hiding this comment.
keeping this function because users_by_storage_used uses it. So, just refactored to re-use partial functions created
| assert.strictEqual(data.usedStorage.uploaded, 32000) | ||
| assert.strictEqual(data.usedStorage.psaPinned, 10000) | ||
| assert.strictEqual(data.usedStorage.psaPinned, 710000) |
There was a problem hiding this comment.
Needed updated as now we are counting fixtures that were not pinned status from https://github.com/web3-storage/web3.storage/blob/main/packages/api/test/fixtures/init-data.sql#L156-L167
| // Create "Failed" Upload. It should not be counted. | ||
| const cid3 = await randomCid() | ||
| const dagSize3 = 100000 | ||
| await createUpload(client, user._id, authKey._id, cid3, { | ||
| dagSize: dagSize3, | ||
| pins: pinsError | ||
| }) | ||
|
|
||
| // Create Upload not pinned yet. It should not be counted yet. | ||
| await createUpload(client, user._id, authKey._id, await randomCid(), { | ||
| dagSize: 1000000, | ||
| pins: initialPinsNotPinned | ||
| }) | ||
|
|
||
| const usedStorageWithFailed = await client.getStorageUsed(user._id) | ||
| assert.strictEqual(usedStorageWithFailed.uploaded, dagSize1 + dagSize2, 'used storage should not count unpinned') | ||
|
|
There was a problem hiding this comment.
These would be accounted now, so really no need to add them given there is no exception anymore
| @@ -127,37 +127,13 @@ export async function createUserWithFiles (dbClient, options = {}) { | |||
| const pinRequests = 3 | |||
| const dagSize = Math.ceil(((percentStorageUsed / 100) * storageQuota) / (uploads + pinRequests)) | |||
|
|
|||
| // Create a failed upload. | |||
There was a problem hiding this comment.
Failed + Not pinned are now counted too
| /** | ||
| * @param {Number} num | ||
| */ | ||
| export function safeNumber (num) { |
There was a problem hiding this comment.
nit.
We could call this function from parseTextToNumber.
There was a problem hiding this comment.
This is not parsing anything - just ensuring it's within the safe integer range.
Maybe validateSafeInteger?
There was a problem hiding this comment.
what I meant is
export function parseTextToNumber (numberText) {
const num = Number(numberText)
validateSafeInteger(num)
return num
}
/**
* @param {Number} num
*/
export function validateSafeInteger (num) {
if (!Number.isSafeInteger(num)) {
throw new Error('Invalid integer number.')
}
return num
}
to avoid duplicating the validation in 2 functions and making sure for instance we're raising a consistent Error.
Really minor ;)
| .rpc('user_used_storage', { query_user_id: userId }) | ||
| .single() | ||
| const [userUploadedResponse, psaPinnedResponse] = await Promise.all([ | ||
| this._client |
There was a problem hiding this comment.
What is the rationale behind splitting the 2 up?
Is it to better understand where the bottleneck is?
There was a problem hiding this comment.
Goal is to decrease the request time to be less likely of getting to the 30s timeout of heroku dyno. With Alan's account that has a lot of data this was taking so much.
Also, we were talking about having a view per product usage, and we will be able to perform queries specific for each thing.
This PR aims to optimize performance for calculating user used storage. It includes:
user_used_storageinto 2 functions, one for regular uploads and other for psapsa_pin_request_search_idxshould perform better and includes the 3 columns used