fix: reduce sql queries in Share::getUsersSharingFile() #41111

DeepDiver1975 · 2023-11-20T11:48:44Z

Description

Save the 🪐 - save CPU, network, memory, energy 👯

Number of queries of Share::getUsersSharingFile() has been reduced from 17+ to 8

Motivation and Context

Be smart on sql queries

How Has This Been Tested?

🤖

Screenshots (if appropriate):

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Database schema changes (next release will require increase of minor version instead of patch)
Breaking change (fix or feature that would cause existing functionality to change)
Technical debt
Tests only (no source changes)

Checklist:

update-docs · 2023-11-20T11:48:49Z

Thanks for opening this pull request! The maintainers of this repository would appreciate it if you would create a changelog item based on your changes.

tests/lib/Share/GetUsersSharingFileTest.php

jvillafanez

Have we taken into account the error handling? I see some if (\OCP\DB::isError($result)) { code removed, but not the matching code.

jvillafanez · 2023-11-21T09:24:45Z

lib/private/Diagnostics/Query.php

+		$s = preg_replace('/\s\s+/', ' ', $this->sql);
+		foreach ($this->params as $i => $param) {
+			$param = json_encode($param);
+			$pos = strpos($s, '?');


This might be problematic if the query contains ? which aren't placeholders.

indeed - but I consider this a minor issue for the time being as this is for pure debugging purpose.
I did not find a better way. 🤷

The json-serialized data looks good to me, just return the query and the parameters separately.
Anyway, I think we should add a TODO / FIXME at least so we're aware of this issue.

The json-serialized data looks good to me, just return the query and the parameters separately.

this has a very bad DX as it requires brain power to interpolate the strings. Does not really help to analyse the sql chaos we have ....

jvillafanez · 2023-11-21T09:34:00Z

lib/private/Share/Share.php

@@ -182,125 +185,94 @@ public static function getUsersSharingFile($path, $ownerUser, $includeOwner = fa
 			}
 		}

+		// let's get the parent for the next round
+		if ($recursive === true) {
+			while ($source !== -1) {


I guess we'll have to assume that there will always be a "source == 1" entry. I think the request will crash with an "out of memory" error in case of an infinite loop, but I don't know if there is anything we could do to pinpoint where the problem is if that happens.

jvillafanez · 2023-11-21T09:36:07Z

lib/private/Share/Share.php

+					$source = (int)$meta['parent'];
+					$sources[]= $source;
+				} else {
+					break;


We should log something in this case. If we don't get the parent it means that part of the tree structure is broken, which seems serious to me.

-1 means we hit the root of the tree and we can stop.

Logic is basically unchanged it was extracted to prevent executing sql in the loop.

But we hit this break if $cache->get((int)$source) === false.
I mean, assuming we have a working tree, we'll leave the loop "naturally" because the $source will be -1 eventually. However, we enter this break if we don't get the cache info from the source file; if this happens, we might have a broken tree because we can't reach the root of the storage, so I think we should at least log something.

Will have a second look ... as stated above: this code is working ™️ this way for quite a long time this way .....
Nevertheless you might be right ..... thx

jvillafanez · 2023-11-21T09:43:56Z

lib/private/Share/Share.php

+		$qb->select('share_with', 'share_type')
+			->from('share')
+			->where($qb->expr()->in('item_source', $qb->createNamedParameter($sources, Connection::PARAM_INT_ARRAY)))
+			->andWhere($qb->expr()->in('share_type', $qb->createNamedParameter([3, 6], Connection::PARAM_INT_ARRAY)))


we should use constants here.

jvillafanez · 2023-11-21T09:49:07Z

lib/private/Share/Share.php

+			->andWhere($qb->expr()->in('share_type', $qb->createNamedParameter([3, 6], Connection::PARAM_INT_ARRAY)))
+			->andWhere($qb->expr()->in('item_type', $qb->createNamedParameter(['file', 'folder'], Connection::PARAM_STR_ARRAY)));
+		$result = $qb->execute();
+		$rows = $result->fetchAllAssociative();


I'm not sure if the query is limited enough to fetch all rows at once. We could clarify with a comment that the limiting factor is that we'll search a few item sources, so we expect 10-20 entries at most (which should be fine).

I was doing my calcs based on the fact that this can be 2 entries per folder depth - but this is wrong.
There can be by far more shares.

Will adapt this. THX for the hint 🙏

DeepDiver1975 · 2023-11-21T11:39:54Z

Have we taken into account the error handling? I see some if (\OCP\DB::isError($result)) { code removed, but not the matching code.

To my understanding in case of errors we will receive exception from dbal. These checks are pointless from my understanding. But feel free to prove me wrong.

sonarcloud · 2023-11-21T12:27:43Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
2 Code Smells

96.1% Coverage
0.0% Duplication

jvillafanez · 2023-11-21T14:14:28Z

To my understanding in case of errors we will receive exception from dbal. These checks are pointless from my understanding. But feel free to prove me wrong.

So the idea is to let the DBAL exception propagate upwards I guess. I'm ok with that, although I don't know how ownCloud will behave.

DeepDiver1975 · 2023-11-21T15:07:07Z

So the idea is to let the DBAL exception propagate upwards I guess. I'm ok with that, although I don't know how ownCloud will behave.

this was already the case for the old code from my understanding.

DeepDiver1975 · 2023-11-22T09:30:24Z

That's what you get when adding unit tests to legacy code 🙈

1) Test\Share\GetUsersSharingFileTest::testIncludeOwner

Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
-    'bob' => '/foo/welcome.txt'
+    'bob' => 'foo/welcome.txt'
 )

/drone/src/tests/lib/Share/GetUsersSharingFileTest.php:229
phpvfscomposer:///drone/src/lib/composer/phpunit/phpunit/phpunit:106

DeepDiver1975 · 2023-11-22T09:41:41Z

master is also generating the path without the leading slash .... why ever ....

DeepDiver1975 added the 2 - Developing label Nov 20, 2023

DeepDiver1975 force-pushed the fix/sql-query-count-share-getusersharingfile branch 3 times, most recently from 2f5953e to c327327 Compare November 20, 2023 16:28

phil-davis reviewed Nov 21, 2023

View reviewed changes

tests/lib/Share/GetUsersSharingFileTest.php Outdated Show resolved Hide resolved

DeepDiver1975 force-pushed the fix/sql-query-count-share-getusersharingfile branch from c327327 to 403d025 Compare November 21, 2023 08:18

jvillafanez reviewed Nov 21, 2023

View reviewed changes

DeepDiver1975 force-pushed the fix/sql-query-count-share-getusersharingfile branch from 403d025 to 39bb864 Compare November 21, 2023 11:52

DeepDiver1975 force-pushed the fix/sql-query-count-share-getusersharingfile branch from 39bb864 to a5e3026 Compare November 21, 2023 16:03

DeepDiver1975 force-pushed the fix/sql-query-count-share-getusersharingfile branch from a5e3026 to 5d72d79 Compare November 23, 2023 09:13

fix: reduce sql queries in Share::getUsersSharingFile()

ccf3b98

DeepDiver1975 force-pushed the fix/sql-query-count-share-getusersharingfile branch from 5d72d79 to ccf3b98 Compare November 23, 2023 09:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: reduce sql queries in Share::getUsersSharingFile() #41111

fix: reduce sql queries in Share::getUsersSharingFile() #41111

DeepDiver1975 commented Nov 20, 2023 •

edited

update-docs bot commented Nov 20, 2023

jvillafanez left a comment

jvillafanez Nov 21, 2023

DeepDiver1975 Nov 21, 2023

jvillafanez Nov 21, 2023

DeepDiver1975 Nov 21, 2023

jvillafanez Nov 21, 2023

jvillafanez Nov 21, 2023

DeepDiver1975 Nov 21, 2023

jvillafanez Nov 21, 2023

DeepDiver1975 Nov 21, 2023

jvillafanez Nov 21, 2023

jvillafanez Nov 21, 2023

DeepDiver1975 Nov 21, 2023

DeepDiver1975 commented Nov 21, 2023

sonarcloud bot commented Nov 21, 2023

jvillafanez commented Nov 21, 2023

DeepDiver1975 commented Nov 21, 2023

DeepDiver1975 commented Nov 22, 2023 •

edited

DeepDiver1975 commented Nov 22, 2023

fix: reduce sql queries in Share::getUsersSharingFile() #41111

Are you sure you want to change the base?

fix: reduce sql queries in Share::getUsersSharingFile() #41111

Conversation

DeepDiver1975 commented Nov 20, 2023 • edited

Description

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist:

update-docs bot commented Nov 20, 2023

jvillafanez left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DeepDiver1975 commented Nov 21, 2023

sonarcloud bot commented Nov 21, 2023

jvillafanez commented Nov 21, 2023

DeepDiver1975 commented Nov 21, 2023

DeepDiver1975 commented Nov 22, 2023 • edited

DeepDiver1975 commented Nov 22, 2023

DeepDiver1975 commented Nov 20, 2023 •

edited

DeepDiver1975 commented Nov 22, 2023 •

edited