[FC-0040] feat: update search index when object tags change #647

yusuf-musleh · 2024-03-24T05:19:26Z

Description

This PR implements updating search index with object tags whenever they change, based on the new CONTENT_OBJECT_TAGS_CHANGED event.

Supporting Information

Related Ticket: [Course Search] Update search index when tags are changed openedx/modular-learning#197
Depends on: [FC-0040] feat: Add CONTENT_OBJECT_TAGS_CHANGED signal openedx/openedx-events#327

Testing Instructions

Run your local devstack on this branch
Clone the openedx-events repo on this branch [FC-0040] feat: Add CONTENT_OBJECT_TAGS_CHANGED signal openedx/openedx-events#327 and make sure it is accessible from your devstack, and install it locally
- pip uninstall openedx-events
- pip install -e path/to/openedx-events/
Make sure you have meilisearch setup locally, follow the setup instructions here https://github.com/open-craft/tutor-contrib-meilisearch
Make sure you also have some sample taxonomies/tags, you can add some from here: https://github.com/open-craft/taxonomy-sample-data
Run tutor dev run cms bash and ./manage.py cms reindex_studio
View the resulting index at http://meilisearch.local.edly.io:7700/ (see tutor-contrib-meilisearch README for how to get the API key to log in)
Check and confirm that is an index with documents in it. If you already have some tags applied to blocks, they will appear there
Go back to studio, and add/update tags in a block
Go back to meilisearch dashboard, and confirm that the Tags field in the index document for the block has been updated to reflect that changes you made.
- Note: I recommend having a easily identifiable name for the block/unit that you added tags to so you can easily search for it
- NOTE: Meilisearch seems to cache queries along with their results in the frontend, so if you simply search it might show you stale data (network tab shows no requests), especially if you're searching for the same query. Make sure you refresh the Meilisearch dashboard (http://meilisearch.local.edly.io:7700/) and then perform the search.
Repeat and confirm the above for both units and blocks

Prinvate-ref: FAL-3691

yusuf-musleh · 2024-03-25T11:13:15Z

openedx/core/djangoapps/content/search/handlers.py

+    upsert_xblock_index_doc.delay(
+        content_object_tags.object_id,
+        recursive=False,
+        update_metadata=True,
+        update_tags=True,
+    )


I needed to set update_metadata=True here, because without it, it was creating new documents when tags are updated, rather than updating the existing one.

I think the new documents' issue is unrelated to this parameter.

We changed the way the ID hash was generated upstream, so the first time you tag an object (after the update), a new document is generated. Could you check if this was the case?

If so, using update_metadata=False should work.

Also, adding Tags to a LibraryComponent if passing here (and falling).

A suggestion is to refactor and create a new update function for tagging only that can handle both types of blocks calling searchable_doc_for_library_block or searchable_doc_for_course_block (and eliminate the update_metadata and update_tags parameters from the task). But it's up to you!

Could you check if this was the case?
If so, using update_metadata=False should work.

Yes -- that seemed to fix it and using update_metadata=False now works! I updated it.

I also added handling tagging for library blocks, I just opted for checking the type of the object_id for the content object and choosing the correct method based on that.

yusuf-musleh · 2024-03-25T11:14:59Z

openedx/core/djangoapps/content/search/documents.py

@@ -162,7 +162,7 @@ def _tags_for_content_object(object_id: UsageKey | LearningContextKey) -> dict:
    # tags for each component separately.
    all_tags = tagging_api.get_object_tags(object_id).all()
    if not all_tags:
-        return {}
+        return {Fields.tags: {}}


This was needed to clear out tags in the index when unselecting all tags for the block, it would remain the last value otherwise because the Fields.tags value would not be included.

Might be worth mentioning that in the code as a comment :)

openedx/core/djangoapps/content/search/tests/test_views.py

openedx/core/djangoapps/content/search/documents.py

rpenido

Hi @yusuf-musleh! Good work here!
I left some suggestions and questions in the review.

openedx/core/djangoapps/content_tagging/rest_api/v1/views.py

mariajgrimaldi · 2024-03-27T13:13:03Z

openedx/core/djangoapps/content_tagging/rest_api/v1/views.py

+            object_id = kwargs.get('object_id')
+            CONTENT_OBJECT_TAGS_CHANGED.send_event(
+                time=now(),
+                content_object=ContentObjectData(object_id=object_id)


Also I wonder, what's an object_id? What does it look like?

The object_id is the id of the course or library block as a string that is being tagged. For example:

block-v1:SampleTaxonomyOrg2+STC1+2023_1+type@vertical+block@f8de78f0897049ce997777a3a31b6ea0

Can we add that explanation to the event data docstring? Here that would address all my comments. Thank you!

Sure thing! Updated.

mariajgrimaldi

I left a few comments regarding openedx/openedx-events#327. Also, is this PR going to be open against upstream?

yusuf-musleh · 2024-03-28T07:49:47Z

Thanks @rpenido and @mariajgrimaldi for your reviews! I've addressed your comments.

Also, is this PR going to be open against upstream?

Yes, this PR should be opened against upstream master once the base of this PR gets merged.

…stones-16b82db feat: Upgrade Python dependency edx-milestones

fix: SSO self-serve tool invalid entityId parsing Commit generated by workflow `openedx/edx-platform/.github/workflows/upgrade-one-python-dependency.yml@refs/heads/master`

…thon_3.11 Make Tests Python 3.11 Compatible

…dx-enterprise-68b3753 feat: Upgrade Python dependency edx-enterprise

…lash feat!: Remove the django-splash app.

feat: re-expose deprecated fragments as passthrough Commit generated by workflow `openedx/edx-platform/.github/workflows/upgrade-one-python-dependency.yml@refs/heads/master` Co-authored-by: connorhaugh <connorhaugh@users.noreply.github.com> Co-authored-by: connorhaugh <49422820+connorhaugh@users.noreply.github.com>

Update to a Python 3.11 compatible version Commit generated by workflow `openedx/edx-platform/.github/workflows/upgrade-one-python-dependency.yml@refs/heads/master`

bradenmacdonald · 2024-04-19T18:52:15Z

@yusuf-musleh I found a small but significant bug, so I pushed a fix to your PR. And (unrelated) there are some conflicts now. Could you please rebase or merge master into this, then ping me to merge it?

bradenmacdonald · 2024-04-19T18:54:49Z

openedx/core/djangoapps/content/search/tests/test_api.py

@@ -116,11 +129,19 @@ def test_reindex_meilisearch_disabled(self, mock_meilisearch):
    @override_settings(MEILISEARCH_ENABLED=True)
    def test_reindex_meilisearch(self, mock_meilisearch):

+        # Add tags field to doc, since reindex calls with `include_tags=True` internally


This docstring is out of date - we don't have include_tags=True anymore

This reverts commit f866545.

…toring-1eb92a3 feat: Upgrade Python dependency edx-proctoring

…-1eb92a3 feat: Upgrade Python dependency edx-rbac

This takes care of updating tags data in search index for both course and library blocks.

yusuf-musleh · 2024-04-21T12:41:50Z

I found a small but significant bug, so I pushed a fix to your PR. And (unrelated) there are some conflicts now. Could you please rebase or merge master into this, then ping me to merge it?

@bradenmacdonald Got it, I've addressed your comments, rebased/fixed conflicts and created a new PR off master, thats good to go (openedx#34559). So I'm closing this PR in favor of that one.

yusuf-musleh force-pushed the yusuf-musleh/update-search-index-when-object-tags-change branch from 89d451c to b2dc9a4 Compare March 24, 2024 05:35

yusuf-musleh mentioned this pull request Mar 24, 2024

[FC-0040] feat: Add CONTENT_OBJECT_TAGS_CHANGED signal openedx/openedx-events#327

Merged

11 tasks

yusuf-musleh force-pushed the yusuf-musleh/update-search-index-when-object-tags-change branch 4 times, most recently from 0d444e4 to 21b8139 Compare March 25, 2024 11:06

yusuf-musleh changed the title ~~[WIP] feat: update search index when object tags change (temp)~~ [WIP] feat: update search index when object tags change Mar 25, 2024

yusuf-musleh commented Mar 25, 2024

View reviewed changes

yusuf-musleh force-pushed the yusuf-musleh/update-search-index-when-object-tags-change branch 6 times, most recently from 61d72bb to 4a85a61 Compare March 26, 2024 09:48

yusuf-musleh commented Mar 26, 2024

View reviewed changes

openedx/core/djangoapps/content/search/tests/test_views.py Outdated Show resolved Hide resolved

yusuf-musleh changed the title ~~[WIP] feat: update search index when object tags change~~ [FC-0040] feat: update search index when object tags change Mar 26, 2024

yusuf-musleh marked this pull request as ready for review March 26, 2024 10:27

rpenido force-pushed the rpenido/fal-3690-update-search-index-when-course-content-is-changed branch from f1a94f3 to a2e9d2a Compare March 26, 2024 18:45

rpenido reviewed Mar 26, 2024

View reviewed changes

openedx/core/djangoapps/content/search/documents.py Outdated Show resolved Hide resolved

rpenido reviewed Mar 26, 2024

View reviewed changes

rpenido force-pushed the rpenido/fal-3690-update-search-index-when-course-content-is-changed branch from a2e9d2a to 4a55c89 Compare March 26, 2024 20:57

mariajgrimaldi reviewed Mar 27, 2024

View reviewed changes

openedx/core/djangoapps/content_tagging/rest_api/v1/views.py Outdated Show resolved Hide resolved

mariajgrimaldi reviewed Mar 27, 2024

View reviewed changes

mariajgrimaldi suggested changes Mar 27, 2024

View reviewed changes

yusuf-musleh force-pushed the yusuf-musleh/update-search-index-when-object-tags-change branch 2 times, most recently from 8f39fc6 to cdde7d7 Compare March 28, 2024 04:31

rpenido force-pushed the rpenido/fal-3690-update-search-index-when-course-content-is-changed branch 2 times, most recently from b7aef89 to 45b6bf9 Compare March 28, 2024 19:34

kdmccormick and others added 9 commits April 18, 2024 15:35

feat: link py38.txt->release/quince.txt for backwards compatibility

f04532d

Merge pull request openedx#34543 from openedx/feanil/upgrade-edx-mile…

68b3753

…stones-16b82db feat: Upgrade Python dependency edx-milestones

feat: Upgrade Python dependency edx-enterprise

aabfb98

fix: SSO self-serve tool invalid entityId parsing Commit generated by workflow `openedx/edx-platform/.github/workflows/upgrade-one-python-dependency.yml@refs/heads/master`

Merge pull request openedx#34506 from openedx/feanil/fix_tests_for_py…

a08b7de

…thon_3.11 Make Tests Python 3.11 Compatible

Merge pull request openedx#34546 from openedx/hamzawaleed01/upgrade-e…

9824938

…dx-enterprise-68b3753 feat: Upgrade Python dependency edx-enterprise

Merge pull request openedx#34528 from openedx/feanil/remove_django_sp…

6ea123d

…lash feat!: Remove the django-splash app.

feat: Upgrade Python dependency edx-proctoring

fcf0461

Update to a Python 3.11 compatible version Commit generated by workflow `openedx/edx-platform/.github/workflows/upgrade-one-python-dependency.yml@refs/heads/master`

feat: Upgrade Python dependency edx-rbac

bbbc6aa

Update to a Python 3.11 compatible version Commit generated by workflow `openedx/edx-platform/.github/workflows/upgrade-one-python-dependency.yml@refs/heads/master`

bradenmacdonald reviewed Apr 19, 2024

View reviewed changes

connorhaugh and others added 12 commits April 19, 2024 12:50

Revert "temp: trace celery tasks in dd (openedx#34537)" (openedx#34553)

a7b4416

This reverts commit f866545.

Merge pull request openedx#34551 from openedx/feanil/upgrade-edx-proc…

2f4c33b

…toring-1eb92a3 feat: Upgrade Python dependency edx-proctoring

Merge pull request openedx#34552 from openedx/feanil/upgrade-edx-rbac…

0603760

…-1eb92a3 feat: Upgrade Python dependency edx-rbac

feat: Update search index when object tags updated

c549228

feat: Update index when library block tags change

7c0d41b

refactor: Remove extra params

921c504

docs: Add new event to hook events docs

28e9f58

feat: Add and use upsert_block_tags_index_docs

fe4e174

This takes care of updating tags data in search index for both course and library blocks.

chore: Update openedx-events

b4b7c7b

fix: Update tests + include course block tags in reindex

ef6b2a7

feat: Fix static-type issues + adjust tag_object

4fe4e18

fix: bug retrieving the Meilisearch API key UID (first time only)

32d7d09

yusuf-musleh force-pushed the yusuf-musleh/update-search-index-when-object-tags-change branch from b15b9d9 to 32d7d09 Compare April 21, 2024 11:29

yusuf-musleh mentioned this pull request Apr 21, 2024

[FC-0040] feat: update search index when object tags change openedx/edx-platform#34559

Merged

docs: Update comments

8fc6c45

yusuf-musleh force-pushed the yusuf-musleh/update-search-index-when-object-tags-change branch from d474406 to 8fc6c45 Compare April 21, 2024 12:13

yusuf-musleh closed this Apr 21, 2024

bradenmacdonald deleted the yusuf-musleh/update-search-index-when-object-tags-change branch April 22, 2024 17:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FC-0040] feat: update search index when object tags change #647

[FC-0040] feat: update search index when object tags change #647

yusuf-musleh commented Mar 24, 2024 •

edited by rpenido

Loading

yusuf-musleh Mar 25, 2024

rpenido Mar 26, 2024

rpenido Mar 26, 2024

yusuf-musleh Mar 28, 2024 •

edited

Loading

yusuf-musleh Mar 25, 2024

bradenmacdonald Apr 19, 2024

rpenido left a comment

mariajgrimaldi Mar 27, 2024

yusuf-musleh Mar 28, 2024

mariajgrimaldi Mar 28, 2024

yusuf-musleh Mar 31, 2024

mariajgrimaldi left a comment •

edited

Loading

yusuf-musleh commented Mar 28, 2024

bradenmacdonald commented Apr 19, 2024

bradenmacdonald Apr 19, 2024

yusuf-musleh commented Apr 21, 2024

[FC-0040] feat: update search index when object tags change #647

[FC-0040] feat: update search index when object tags change #647

Conversation

yusuf-musleh commented Mar 24, 2024 • edited by rpenido Loading

Description

Supporting Information

Testing Instructions

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yusuf-musleh Mar 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rpenido left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mariajgrimaldi left a comment • edited Loading

Choose a reason for hiding this comment

yusuf-musleh commented Mar 28, 2024

bradenmacdonald commented Apr 19, 2024

Choose a reason for hiding this comment

yusuf-musleh commented Apr 21, 2024

yusuf-musleh commented Mar 24, 2024 •

edited by rpenido

Loading

yusuf-musleh Mar 28, 2024 •

edited

Loading

mariajgrimaldi left a comment •

edited

Loading