Added support for full text search of database fields and PDF/XML galleys #2908

mauromsl · 2022-06-07T08:19:25Z

There is a feature flag to enable disable this feature (ENABLE_FULL_TEXT_SEARCH)
When the full text search is disabled, the old search interface and backend are used
Includes management commands for creating database index and populating an index of the PDF/XML files into the database.
Includes a new search interface that allows users to control what fields are included in the search.
The backend interface includes support for both MySQL and Postgres
Postgres support is enhanced in that it can weight search results and also persists only a GIN index of the full text (in MySQL we store the entirety of the text + we buid a FULLTEXT index)
closes Add support for Full Text Search #2876
closes Sort by date ascending and date descending in article search #2204
closes Add ability to search by ORCID #2728
closes Add abstract to current Articles search #2795

ajrbyers

Generally, this is great. Good work! I've added some, mostly ignorable, comments inline!

🎖️ 🎖️ 🎖️ 🎖️ 🎖️

src/core/migrations/0070_auto_20220506_1652.py

src/core/models.py

ajrbyers · 2022-06-07T09:29:38Z

src/core/models.py

+
+
+@receiver(models.signals.pre_save, sender=File)
+def update_file_index(sender, instance, **kwargs):


Should we document that when calling .update() files will not be indexed?

src/submission/models.py

src/core/janeway_global_settings.py

mauromsl added 19 commits May 19, 2022 08:02

#2876: Adds models for indexing files

bce0f01

#2876: Adds a command for loading galleys to db

4d1a042

#2876: Adds text miners

5eeeced

#2876: Adds indexing method to core.File

4e0f819

#2876: Adds FT search lookups

de50e21

#2876: Adds text parsing unit tests

bf475a6

#2876: Update requirements.txt

f40b47b

#2876: Adds an Article search manager

c97ab6b

#2876: Adds Unit Tests for new search engine

b070490

#2876: Adds new search views and forms

d667903

#2876: Adds migration and settings update

cc2c6f0

#2876: Adds full text search feature flag

29057b3

#2876: Auto-index text when FT is enabled

0f204da

#2876: Adds support for weighted full text search

3c9e7e3

#2876: Fix SearchVector not working with SearchVectorField

3c4706f

#2876: Avoid postgres migrations running on mysql

788efcb

#2876: Removes weighted search from other backends

d5c0895

#2876: Update indexes for searching with mysql

f41fcf8

#2876: Updates all themes

22856c8

mauromsl requested a review from ajrbyers June 7, 2022 08:19

ajrbyers requested changes Jun 7, 2022

View reviewed changes

mauromsl commented Jun 7, 2022

View reviewed changes

src/core/janeway_global_settings.py Show resolved Hide resolved

mauromsl added 4 commits June 7, 2022 19:40

#2876: Fixes unit tests for SQLITE

75c3194

#2876: Minor changes based on review feedback

5398ea4

2876: Ensure distinct values returned regardless of ORDER BY

e75dd0a

#2876: Updates unit tests

51cd3ef

mauromsl merged commit dce0e97 into master Jun 8, 2022

mauromsl deleted the full-text-search branch June 8, 2022 14:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added support for full text search of database fields and PDF/XML galleys #2908

Added support for full text search of database fields and PDF/XML galleys #2908

mauromsl commented Jun 7, 2022 •

edited

ajrbyers left a comment

ajrbyers Jun 7, 2022



		@receiver(models.signals.pre_save, sender=File)
		def update_file_index(sender, instance, **kwargs):

Added support for full text search of database fields and PDF/XML galleys #2908

Added support for full text search of database fields and PDF/XML galleys #2908

Conversation

mauromsl commented Jun 7, 2022 • edited

ajrbyers left a comment

Choose a reason for hiding this comment

ajrbyers Jun 7, 2022

Choose a reason for hiding this comment

mauromsl commented Jun 7, 2022 •

edited