Fix search on private worksheets #4432

AndrewJGaut · 2023-03-31T05:40:10Z

Reasons for making this change

We discovered an issue where bundles on private worksheets were not visible to cl search and would not appear in its output. Turns out there was a minor error in the search_bundles function when searching for private bundles with a non-root user (we used a JOIN instead of a LEFT OUTER JOIN in one place). This fixes that issue.

Changes

Fix issue for non-root user searching for private bundles.
Beef up tests. Add in test for searching for bundles on private worksheets. Beef up non-root user tests significantly. (Before, we had a separate section for non-root user tests for search which weren't as rigorous as the tests used for the root user. Now, we apply the exact same tests for the root user and a non-root user.)

Related issues

#4405

…ix-search-on-private-worksheets

…bundle under the group name...

…ix-search-on-private-worksheets

AndrewJGaut · 2023-03-31T06:28:30Z

tests/cli/test_cli.py

-        create_user(ctx, user_name, disk_quota='2000')
-        switch_user(user_name)
-        check_equals(uuid1, _run_command([cl, 'search', 'uuid=' + uuid1, '-u']))
+    def test_search_helper(ctx):


define a new helper function (so we can run the same tests multiple times)

AndrewJGaut · 2023-03-31T06:29:40Z

codalab/model/bundle_model.py

+            aliased_group_bundle_permission = aliased(cl_group_bundle_permission)
            add_join(
-                cl_group_bundle_permission,
-                cl_bundle.c.uuid == cl_group_bundle_permission.c.object_uuid,
+                aliased_group_bundle_permission,
+                cl_bundle.c.uuid == aliased_group_bundle_permission.c.object_uuid,
+                left_outer_join=True,
+            )
+            aliased_cl_user_group = aliased(cl_user_group)
+            add_join(
+                aliased_cl_user_group,
+                aliased_cl_user_group.c.user_id == user_id,
+                left_outer_join=True,


We use aliased here to account for the case when we've already done a join on the group_bundle_permission table or the user_group table. (If you do that twice without aliasing one of them, the query is invalid according to MySQL syntax rules.)

Does aliased automatically pick a unique name, or do you need to explicitly pick a name?

In a different PR, we should probably add validation when we do the joins to make sure that we're not joining onto the same table twice.

AndrewJGaut · 2023-03-31T19:56:01Z

tests/cli/test_cli.py

+        """Search bundles on private worksheets"""
+        wuuid = _run_command([cl, 'work', '-u'])
+        new_wuuid = _run_command([cl, 'new', random_name()])
+        _run_command([cl, 'wperm', new_wuuid, 'public', 'n'])  # make worksheet private
+        _run_command([cl, 'work', new_wuuid])  # switch to worksheet
+        uuid = _run_command([cl, 'upload', test_path('a.txt')])
+        check_equals(uuid, _run_command([cl, 'search', uuid, '-u']))
+        _run_command([cl, 'work', wuuid])


this is the new test added for search on private worksheets

AndrewJGaut · 2023-04-07T06:32:43Z

tests/cli/test_cli.py

+    test_search_helper(ctx)
+
+    # Test with non-root user.
+    if not os.getenv('CODALAB_PROTECTED_MODE'):
+        # This test does not work when protected_mode is True.
+        _, current_user_name = current_user()
+        user_name = 'non_root_user_' + random_name()
+        create_user(ctx, user_name, disk_quota='1000000')
+        switch_user(user_name)
+        new_wuuid = _run_command([cl, 'new', random_name()])
+        _run_command([cl, 'work', new_wuuid])
+        test_search_helper(ctx)


Call helper function here to actually run the tests. Note we call the same helper for the root and non-root user, so the exact same tests are performed in both cases

yifanmai

Great detective work!

yifanmai · 2023-04-10T22:44:29Z

codalab/model/bundle_model.py

+            aliased_group_bundle_permission = aliased(cl_group_bundle_permission)
            add_join(
-                cl_group_bundle_permission,
-                cl_bundle.c.uuid == cl_group_bundle_permission.c.object_uuid,
+                aliased_group_bundle_permission,
+                cl_bundle.c.uuid == aliased_group_bundle_permission.c.object_uuid,
+                left_outer_join=True,
+            )
+            aliased_cl_user_group = aliased(cl_user_group)
+            add_join(
+                aliased_cl_user_group,
+                aliased_cl_user_group.c.user_id == user_id,
+                left_outer_join=True,


Does aliased automatically pick a unique name, or do you need to explicitly pick a name?

In a different PR, we should probably add validation when we do the joins to make sure that we're not joining onto the same table twice.

yifanmai · 2023-04-10T22:46:21Z

tests/cli/test_cli.py

+            size1 + size2, float(_run_command([cl, 'search', 'name=' + name, 'data_size=.sum']))
+        )
+
+        """Search for floating bundles"""


nit: I think we use comment style # rather than docstrings for mid-method comments?

Aliased does automatically pick a unique name, so there's no issue with table name collisions or joining on the same table twice That's actually what I meant when I said the query was invalid otherwise; before I added aliased, sometimes it would join on the same table twice and MySQL actually throws an error when that happens. So we should be OK (if I'm understanding your comment correctly).

For the # comments style, I updated that.

yifanmai

So I understand, the failing case is when the bundle has no permissions, so the join is empty?

yifanmai · 2023-04-10T23:23:03Z

codalab/model/bundle_model.py

            access_via_owner = cl_bundle.c.owner_id == user_id
            access_via_group = and_(
                or_(  # Join constraint (group)
-                    cl_group_bundle_permission.c.group_uuid
+                    aliased_group_bundle_permission.c.group_uuid


Actually, is there a potential false positive case here (access_via_group is incorrectly true) if aliased_group_bundle_permission.c.group_uuid and self.public_group_uuid are both None?

Thinking about this a bit more, this logic seems incorrect?

aliased_group_bundle_permission is all the group bundle permissions for the object

aliased_cl_user_group is all the group memberships for the user

aliased_cl_user_group.c.user_id == user_id doesn't do anything because this is already enforced by the join.

So it seems like if the bundle grants access to some group A, and the user X belongs to some group B, then user X incorrectly gets access to the bundle?

Seems like what we're missing here is we need to check that aliased_group_bundle_permission.c.group_id == aliased_cl_user_group.c.group_id?

Alternatively you can push this constraint into the join condition i.e. when you in onto aliased_group_bundle_permission, join using the condition that the bundle permission group is the user's groups or the public group only?

Yes, you're totally correct. Thanks for catching this! I updated it. I believe it should be correct now.

…/codalab-worksheets into fix-search-on-private-worksheets

yifanmai · 2023-05-23T22:45:41Z

codalab/model/bundle_model.py

+                    aliased_group_bundle_permission.c.group_uuid
+                    == aliased_cl_user_group.c.group_uuid,  # Private group


I believe you have to also check that aliased_group_bundle_permission.c.group_uuid is not NULL.

Otherwise this could be triggered if the user is not in any groups, and the bundles has no group permissions. Could you check if this is the case?

OK, I added that check and some more tests to make sure that's not a problem anymore.

I think it looks good now?

…ix-search-on-private-worksheets

AndrewJGaut added 9 commits March 16, 2023 17:20

just deubgging

bd55d25

Merge branch 'master' of github.com:codalab/codalab-worksheets into f…

d50713f

…ix-search-on-private-worksheets

Got the tests passing without group search

419a965

works with root user. non-root just fails when it comes to running a …

0f3a914

…bundle under the group name...

Fixed issue with permissions

1157741

Fixed the issues.

a1df240

Merge branch 'master' of github.com:codalab/codalab-worksheets into f…

94c92dd

…ix-search-on-private-worksheets

Fix up styling

1f26d89

minor change

ec82dab

AndrewJGaut marked this pull request as ready for review March 31, 2023 06:26

AndrewJGaut requested review from percyliang and epicfaace March 31, 2023 06:27

AndrewJGaut commented Mar 31, 2023

View reviewed changes

AndrewJGaut requested review from wwwjn and yifanmai April 7, 2023 06:31

AndrewJGaut commented Apr 7, 2023

View reviewed changes

yifanmai approved these changes Apr 10, 2023

View reviewed changes

Merge branch 'master' into fix-search-on-private-worksheets

eda85dd

yifanmai requested changes Apr 10, 2023

View reviewed changes

AndrewJGaut added 3 commits April 11, 2023 14:19

Fix bug and styling

01c2602

Merge branch 'fix-search-on-private-worksheets' of github.com:codalab…

c54c06b

…/codalab-worksheets into fix-search-on-private-worksheets

Fix minor column name bug

de4f233

AndrewJGaut requested a review from yifanmai May 17, 2023 06:40

yifanmai requested changes May 24, 2023

View reviewed changes

AndrewJGaut added 4 commits September 24, 2023 11:48

Merge branch 'master' of github.com:codalab/codalab-worksheets into f…

570afce

…ix-search-on-private-worksheets

Add in a few more tests and the extra check Yifan asked for

513e416

Fix protected mode issue

9f9e68b

Fix another protected_mode issue.

7f38e55

AndrewJGaut requested a review from yifanmai September 27, 2023 05:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix search on private worksheets #4432

Fix search on private worksheets #4432

AndrewJGaut commented Mar 31, 2023 •

edited

AndrewJGaut Mar 31, 2023

AndrewJGaut Mar 31, 2023

yifanmai Apr 10, 2023

AndrewJGaut Mar 31, 2023

AndrewJGaut Apr 7, 2023

yifanmai left a comment

yifanmai Apr 10, 2023

yifanmai Apr 10, 2023

AndrewJGaut Apr 11, 2023

yifanmai left a comment

yifanmai Apr 10, 2023

yifanmai Apr 10, 2023

AndrewJGaut Apr 11, 2023

yifanmai May 23, 2023

AndrewJGaut Sep 27, 2023

		aliased_group_bundle_permission.c.group_uuid
		== aliased_cl_user_group.c.group_uuid, # Private group

Fix search on private worksheets #4432

Are you sure you want to change the base?

Fix search on private worksheets #4432

Conversation

AndrewJGaut commented Mar 31, 2023 • edited

Reasons for making this change

Changes

Related issues

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yifanmai left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yifanmai left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AndrewJGaut commented Mar 31, 2023 •

edited