GPS: Amélioration de l'autocomplete avec recherche plein texte by tonial · Pull Request #5365 · gip-inclusion/les-emplois

tonial · 2025-01-10T14:24:45Z

🤔 Pourquoi ?

https://www.notion.so/gip-inclusion/Am-liorations-de-l-autocomplete-Ajout-d-un-b-n-ficiaire-14b5f321b6048096bc87ea7a6bd27c50?pvs=4

Alternative à #5363

Un peu plus rapide que l'autre version avec une gestion différente : il faut taper les noms en entier (Jean ne matche pas sur Jeanne) et donc connaître l'orthographe exacte.

Niveau performances :
Juste avec la recherche plein text -> ~1 seconde
Avec la colonne GeneratedField SearchVectorField, sans index -> 0.25s
Avec la colonne + index -> 0.04 s

Le 2nd commit utilise un dictionnaire custom (simple + unaccent) pour ne pas avoir à rendre unaccent immutable
performance identiques, donc c'est un peu à nous de choisir ce qu'on préfère :)

🍰 Comment ?

Décrivez en quelques mots la solution retenue et mise en oeuvre, les difficultés ou problèmes rencontrés. Attirez l'attention sur les décisions d'architecture ou de conception importantes.

🚨 À vérifier

Mettre à jour le CHANGELOG_breaking_changes.md ?

🏝️ Comment tester

Les instructions pour reproduire le problème, les profils de test, le parcours spécifique à utiliser, etc. Si vous disposez d'une recette jetable, mettre l'URL pour tester dans cette partie.

💻 Captures d'écran

xavfernandez · 2025-01-10T16:49:49Z

itou/users/migrations/0020_make_unnaccent_immutable.py

+        migrations.RunSQL("ALTER FUNCTION unaccent(text) IMMUTABLE"),
+        migrations.RunSQL("ALTER FUNCTION unaccent(regdictionary, text) IMMUTABLE"),


En voyant ces 2 RunSQL, je me demande du coup si faire des:

CREATE TEXT SEARCH CONFIGURATION unaccent_simple (COPY=simple); ALTER TEXT SEARCH CONFIGURATION unaccent_simple ALTER MAPPING FOR hword, hword_part, word WITH unaccent,simple;

à la place ne serait pas plus propre ?

(et si l'index

GinIndex( SearchVector("first_name", "last_name", config="unaccent_simple"), name="full_name_search_idx", ),

est bien créable sans unaccent immutable.)

J'avais oublié de pusher avec le unaccent_simple.

J'ai donc 2 version dans la PR à présent (pour que tout le monde puisse tester facilement)

1er commit : unaccent en immutable

2e commit : dictionnaire unaccent_simple, sans passer unaccent en immutable
(il faut penser à faire un ./manage.py migrate users 0019 à chaque fois qu'on change de commit)

itou/www/autocomplete/views.py

francoisfreitag · 2025-01-13T08:40:23Z

itou/users/migrations/0021_user_full_name.py

+        migrations.AddField(
+            model_name="user",
+            name="full_name",
+            field=models.GeneratedField(
+                db_persist=True,
+                expression=django.contrib.postgres.search.SearchVector(
+                    "first_name", "last_name", config="simple_unaccent"
+                ),
+                output_field=django.contrib.postgres.search.SearchVectorField(),
+                verbose_name="nom complet utilisé pour rechercher un utilisateur",
+            ),
+        ),


Cette migration prend 15s à exécuter, attention au moment où on la lance.

Je la lancerais un soir tard, ou un matin tôt

itou/www/autocomplete/views.py

francoisfreitag

Les perfs sont top ! :)

itou/users/models.py

xavfernandez

Beaucoup de snapshots modifiés sans raison et j'imagine que c'est ce qui déplaît aux tests :)

tonial · 2025-01-13T12:38:31Z

J'ai fait l'erreur de mettre à jour le nom du champ avec un replace dans vscode qui en a profiter pour sournoisement reformater les fichiers en retirant les espaces les lignes avec que des espaces...

xavfernandez · 2025-01-13T20:46:13Z

itou/users/migrations/0022_user_full_name_search_idx.py

+        ("auth", "0012_alter_user_first_name_max_length"),
+        ("cities", "0001_initial"),


Ces deux dépendances me semblent étonnantes ?

Je pensais que c'était parce que cities_0001 contenait BtreeGinExtension(), mais en fait je ne le vois nulle part dans notre code, donc je me demande comment on a activé les indexes GIN 🤔
Est-ce qu'on ne devrait pas ajouter BtreeGinExtension() dans cette migration pour être safe ?

En tout cas J'ai retiré les dépendances peu utiles (on verra si les tests plantent)

Je pense que notre utilisation reste dans le natif des index Gin et ne nécessite pas d'extension :)

xavfernandez · 2025-01-13T20:47:43Z

itou/users/models.py

                name="users_user_email_upper",
-            )
+            ),
+            GinIndex("full_name_search_vector", name="full_name_search_idx"),


Suggested change

GinIndex("full_name_search_vector", name="full_name_search_idx"),

GinIndex("full_name_search_vector", name="users_user_full_name_search_idx"),

pour coller aux noms des autres index

La longueur max c'est 30 caractères, j'ai pu ajouter users_user devant, mais pour cela j'ai retiré search (et vector n'y est pas non plus)

xavfernandez · 2025-01-13T20:58:14Z

itou/static/js/gps.js

+          ajax:{
+            delay: 250 // wait 250 milliseconds before triggering the request
+          }
        },


Suggested change

ajax:{

delay: 250 // wait 250 milliseconds before triggering the request

}

},

},

ajax:{

delay: 250 // wait 250 milliseconds before triggering the request

}

Le ajax doit être au même niveau que la clef language je pense 😅

😢 j'étais pourtant sur d'être au bon niveau d'indentation

itou/static/js/gps.js

francoisfreitag · 2025-01-14T08:19:21Z

itou/static/js/gps.js

                  </a>
              </div>
-          `),
+            `),


Suggested change

`),

`),

J'ai aligné sur noResults pourtant

Est-ce que je ne devrais pas plutot décaler tout le
d'un cran vers la droite ?

Oui, effectivement.
Idéalement, on ne mixerait pas des indentations de 4 et de 2...

francoisfreitag · 2025-01-14T08:19:58Z

itou/static/js/gps.js

        },
+        ajax: {
+          delay: 250 // wait 250 milliseconds before triggering the request
+        }


Suggested change

}

},

francoisfreitag

👏

Use psql full-text with a generated search vector field and an index on it

tonial requested review from francoisfreitag and xavfernandez January 10, 2025 14:24

tonial mentioned this pull request Jan 10, 2025

GPS: Amélioration de l'autocomplete #5363

Closed

1 task

tonial force-pushed the alaurent/gps_autocomplete_full_text_search branch from 261fa63 to e65eb52 Compare January 10, 2025 14:53

xavfernandez reviewed Jan 10, 2025

View reviewed changes

tonial force-pushed the alaurent/gps_autocomplete_full_text_search branch from e65eb52 to 249a180 Compare January 10, 2025 19:36

tonial self-assigned this Jan 10, 2025

tonial added the modifié Modifié dans le changelog. label Jan 10, 2025

tonial force-pushed the alaurent/gps_autocomplete_full_text_search branch 2 times, most recently from 1668581 to 1bd5d3b Compare January 10, 2025 22:15

francoisfreitag reviewed Jan 13, 2025

View reviewed changes

itou/www/autocomplete/views.py Outdated Show resolved Hide resolved

francoisfreitag reviewed Jan 13, 2025

View reviewed changes

itou/www/autocomplete/views.py Outdated Show resolved Hide resolved

francoisfreitag reviewed Jan 13, 2025

View reviewed changes

This comment was marked as outdated.

Sign in to view

xavfernandez reviewed Jan 13, 2025

View reviewed changes

itou/users/models.py Outdated Show resolved Hide resolved

tonial force-pushed the alaurent/gps_autocomplete_full_text_search branch from 1bd5d3b to b153447 Compare January 13, 2025 12:15

tonial requested review from francoisfreitag and xavfernandez January 13, 2025 12:15

tonial force-pushed the alaurent/gps_autocomplete_full_text_search branch from b153447 to 124e8d0 Compare January 13, 2025 12:15

xavfernandez reviewed Jan 13, 2025

View reviewed changes

tonial force-pushed the alaurent/gps_autocomplete_full_text_search branch from 124e8d0 to 0d6e1d5 Compare January 13, 2025 12:37

tonial requested a review from xavfernandez January 13, 2025 14:00

xavfernandez reviewed Jan 13, 2025

View reviewed changes

tonial force-pushed the alaurent/gps_autocomplete_full_text_search branch from dbf9c56 to 4aca0fd Compare January 13, 2025 22:42

xavfernandez approved these changes Jan 14, 2025

View reviewed changes

francoisfreitag reviewed Jan 14, 2025

View reviewed changes

itou/static/js/gps.js Outdated Show resolved Hide resolved

francoisfreitag reviewed Jan 14, 2025

View reviewed changes

itou/static/js/gps.js Outdated Show resolved Hide resolved

tonial force-pushed the alaurent/gps_autocomplete_full_text_search branch from 4aca0fd to 3b80f59 Compare January 14, 2025 08:18

francoisfreitag reviewed Jan 14, 2025

View reviewed changes

tonial force-pushed the alaurent/gps_autocomplete_full_text_search branch from 3b80f59 to de70cc6 Compare January 14, 2025 08:40

francoisfreitag approved these changes Jan 14, 2025

View reviewed changes

tonial force-pushed the alaurent/gps_autocomplete_full_text_search branch from de70cc6 to 9f8ff9b Compare January 14, 2025 09:04

gps: Improve user autocomplete

d51194a

Use psql full-text with a generated search vector field and an index on it

tonial force-pushed the alaurent/gps_autocomplete_full_text_search branch from 9f8ff9b to d51194a Compare January 14, 2025 14:06

tonial added this pull request to the merge queue Jan 14, 2025

Merged via the queue into master with commit f98ff88 Jan 14, 2025

tonial deleted the alaurent/gps_autocomplete_full_text_search branch January 14, 2025 21:00

		migrations.RunSQL("ALTER FUNCTION unaccent(text) IMMUTABLE"),
		migrations.RunSQL("ALTER FUNCTION unaccent(regdictionary, text) IMMUTABLE"),

		("auth", "0012_alter_user_first_name_max_length"),
		("cities", "0001_initial"),

	GinIndex("full_name_search_vector", name="full_name_search_idx"),
	GinIndex("full_name_search_vector", name="users_user_full_name_search_idx"),

Conversation

tonial commented Jan 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🤔 Pourquoi ?

🍰 Comment ?

🚨 À vérifier

🏝️ Comment tester

💻 Captures d'écran

Uh oh!

xavfernandez Jan 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

francoisfreitag left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

xavfernandez left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tonial commented Jan 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tonial Jan 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tonial Jan 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

francoisfreitag left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tonial commented Jan 10, 2025 •

edited

Loading

xavfernandez Jan 10, 2025 •

edited

Loading

xavfernandez left a comment •

edited

Loading

tonial commented Jan 13, 2025 •

edited

Loading

tonial Jan 13, 2025 •

edited

Loading

tonial Jan 14, 2025 •

edited

Loading