x/pkgsite: change in pkgsite search for path tokenization

[ethanalee-work]

What did you do?

I encountered the following change in pkgsite search which is already in production while debugging a test: https://github.com/golang/pkgsite/blob/fd3717b20fe8bcec3503ec0238a973a719e3931b/tests/search/scripts/symbolsearch.txt#L66-L68

Expected Outcome

Searching with keywords: bee cmd command is expected to return Command github.com/beego/bee/cmd/commands

Actual Outcome

The search returns an empty path and symbol and does not match the expected result.

Root Cause

In PostgreSQL 14, there was a significant change in how text search is performed. The way that PostgreSQL tokenizes and stems words in to_tsvector('english', ...) is now different.

The pkgsite search algorithm builds a tsvector from the package path github.com/beego/bee/cmd/commands and Command symbol. Then the user query bee cmd command is parsed into a tsquery.

However, in PostgreSQL 14 there is now a mismatch. There is no overlap between the stemmed tokens in the DB and query. For example, one can run the following two queries to see the diff:

Tokens generated by the user query:

SELECT alias, description, token, dictionaries, dictionary, lexemes 
FROM ts_debug('english', 'github.com/beego/bee/cmd/commands Command');

alias   |   description   |               token               |  dictionaries  |  dictionary  |               lexemes               
-----------+-----------------+-----------------------------------+----------------+--------------+-------------------------------------
 url       | URL             | github.com/beego/bee/cmd/commands | {simple}       | simple       | {github.com/beego/bee/cmd/commands}
 host      | Host            | github.com                        | {simple}       | simple       | {github.com}
 url_path  | URL path        | /beego/bee/cmd/commands           | {simple}       | simple       | {/beego/bee/cmd/commands}
 blank     | Space symbols   |                                   | {}             |              | 
 asciiword | Word, all ASCII | Command                           | {english_stem} | english_stem | {command}
(5 rows)

What's in the DB:

SELECT alias, description, token, dictionaries, dictionary, lexemes 
FROM ts_debug('english', 'bee cmd command');
   alias   |   description   |  token  |  dictionaries  |  dictionary  |  lexemes  
-----------+-----------------+---------+----------------+--------------+-----------
 asciiword | Word, all ASCII | bee     | {english_stem} | english_stem | {bee}
 blank     | Space symbols   |         | {}             |              | 
 asciiword | Word, all ASCII | cmd     | {english_stem} | english_stem | {cmd}
 blank     | Space symbols   |         | {}             |              | 
 asciiword | Word, all ASCII | command | {english_stem} | english_stem | {command}
(5 rows)

In the first query it is clear that now PostgreSQL parses the package path as a url and url_path and doesn't split it into individual tokens that we can match on!

Is this a regression?

Yes, in the case that searching for partial path elements (like "bee") is now degraded since they are getting less or fewer accurate results.

Fix

To restore existing search behavior, we should modify search.go to replace the slashes with spaces before feeding package paths to to_tsvector. We would need to do a DB migration to repopulate all of the tsvector columns. However, it may not be worth the risk of modifying search so heavily.

[gopherbot]

Change https://go.dev/cl/750740 mentions this issue: all: update PostgreSQL version to 14

[ethanalee-work]

Per offline discussion, let's hold off on this for now given that there is not a clear degradation in UX.