Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/pkgsite: support searching by <package>.<identifier> #44142

Open
matttproud opened this issue Feb 7, 2021 · 153 comments
Open

x/pkgsite: support searching by <package>.<identifier> #44142

matttproud opened this issue Feb 7, 2021 · 153 comments

Comments

@matttproud
Copy link
Contributor

@matttproud matttproud commented Feb 7, 2021

What is the URL of the page with the issue?

https://pkg.go.dev/search?q=sql.DB

What is your user agent?

Mozilla/5.0 (X11; CrOS x86_64 13597.66.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.109 Safari/537.36

Screenshot

Screenshot 2021-02-07 13 52 26

What did you do?

I wanted to view the documentation for sql.DB, which is part of the standard distribution. I entered the search query of sql.DB into the interface, and it gave me no good results.

What did you expect to see?

Basically this but on the new web site: https://golang.org/pkg/database/sql/#DB

What did you see instead?

I don't know how the web application is implemented, but I expect either standard distribution packages are not indexed, the search query format "<package name>.<Identifier>" is not supported, or the result ranking is somehow deficient. I had been using this on old Godoc with great success since time immemorial.

@gopherbot gopherbot added this to the Unreleased milestone Feb 7, 2021
@jamalc jamalc removed this from the Unreleased milestone Feb 8, 2021
@jamalc jamalc added this to the pkgsite/golang.org-features milestone Feb 8, 2021
@julieqiu julieqiu changed the title x/pkgsite: search by <package>.<identifier> no longer possible x/pkgsite: support searching by <package>.<identifier> Feb 9, 2021
@julieqiu julieqiu removed this from the pkgsite/golang.org-features milestone Feb 9, 2021
@julieqiu julieqiu added this to the pkgsite/search milestone Feb 9, 2021
@magical
Copy link
Contributor

@magical magical commented Feb 20, 2021

I'd go further and allow searching for just <identifier>, at least within the standard library. I occasionally use golang.org's search to find, e.g., a function that i can't remember the package for https://golang.org/search?q=WriteFile#Functions, or all the types that implement some method https://golang.org/search?q=Sum#Global.

@ALTree
Copy link
Member

@ALTree ALTree commented May 12, 2021

To add another datapoint, this Go Tour page links to https://golang.org/search?q=Read#Global. The link is currently broken and I couldn't find a good pkgsite alternative to replace it.

@gopherbot
Copy link

@gopherbot gopherbot commented May 12, 2021

Change https://golang.org/cl/318651 mentions this issue: internal/postgres: refactor getPackageSymbols

@gopherbot
Copy link

@gopherbot gopherbot commented May 12, 2021

Change https://golang.org/cl/318490 mentions this issue: internal/postgres: insert search_documents path ids

@gopherbot
Copy link

@gopherbot gopherbot commented May 12, 2021

Change https://golang.org/cl/318489 mentions this issue: migrations: add search_documents package_path_id and module_path_id

@gopherbot
Copy link

@gopherbot gopherbot commented May 12, 2021

Change https://golang.org/cl/318650 mentions this issue: migrations: add symbol_search_documents

@gopherbot
Copy link

@gopherbot gopherbot commented May 12, 2021

Change https://golang.org/cl/318652 mentions this issue: internal/postgres: insert symbol search documents

gopherbot pushed a commit to golang/pkgsite that referenced this issue May 13, 2021
For golang/go#44142

Change-Id: I9d1908b07924a02153a249773d9440676a0e349a
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/318489
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
gopherbot pushed a commit to golang/pkgsite that referenced this issue May 13, 2021
The search_documents columns package_path_id and module_path_id are now
populated.

For golang/go#44142

Change-Id: I1833bfc821f3a3ebe5b71a562e616d0c6b2dc20a
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/318490
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
gopherbot pushed a commit to golang/pkgsite that referenced this issue May 13, 2021
A table is added to contain information for searching for symbols.

Build contexts are added as a text array instead of as individual rows,
since the only use case for that information is linking users to the
correct URL if the symbol is not present in the default build context.

For golang/go#44142

Change-Id: I759b22e0ced7fec547778aba11a1b5f641530fb8
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/318650
Trust: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
gopherbot pushed a commit to golang/pkgsite that referenced this issue May 13, 2021
getPackageSymbols is refactored using squirrel, so that it can be
modified for us in other similar queries.

For golang/go#44142

Change-Id: I974c51155d30c59cf61ded0386c5d3b2702188c5
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/318651
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
@gopherbot
Copy link

@gopherbot gopherbot commented May 14, 2021

Change https://golang.org/cl/318871 mentions this issue: content,internal/frontend: add symbol search

@gopherbot
Copy link

@gopherbot gopherbot commented May 14, 2021

Change https://golang.org/cl/318870 mentions this issue: internal/postgres: add symbol search

gopherbot pushed a commit to golang/pkgsite that referenced this issue May 17, 2021
Data is now inserted into symbol_search_documents.

For golang/go#44142

Change-Id: Icf00cd7c5e57a91465673d1828f961334abc3a48
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/318652
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
gopherbot pushed a commit to golang/pkgsite that referenced this issue May 17, 2021
An initial version of symbol search is added. This runs separately from
regular search and is not connected to the frontend yet.

For golang/go#44142

Change-Id: I34a21a4a743a07dcfcdfeb64b52a86bc2254f62a
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/318870
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
gopherbot pushed a commit to golang/pkgsite that referenced this issue May 17, 2021
The frontend now supports searching for symbols using the syntax
"identifier:<query>".

For golang/go#44142

Change-Id: Iae1db22091cf3921d47fbbdbdfbbf87e25792fd9
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/318871
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
@gopherbot
Copy link

@gopherbot gopherbot commented Jun 18, 2021

Change https://golang.org/cl/329497 mentions this issue: static: move symbol search template

@gopherbot
Copy link

@gopherbot gopherbot commented Jun 18, 2021

Change https://golang.org/cl/329349 mentions this issue: internal/postgres: move insertSymbols to saveModule

@gopherbot
Copy link

@gopherbot gopherbot commented Jun 18, 2021

Change https://golang.org/cl/329493 mentions this issue: internal/postgres: change logic for upsertSearchDocumentSymbols

gopherbot pushed a commit to golang/pkgsite that referenced this issue Jun 21, 2021
Logic for insertSymbols is moved out of insertUnits to saveModule. This
will allows us to run that code after the check for isLatest, which is
necessary to insert symbols where the latest unit is not a
release version.

For golang/go#44142

Change-Id: Id917b6b74020957549ad1c6594f21d3f8d27e464
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/329349
Trust: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Jun 21, 2021
upsertSearchDocumentSymbols is changed based on the new schema.

For golang/go#44142

Change-Id: I1857b04cceeb9b702e4c54fc3a8d39efa811a3ce
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/329493
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Sep 8, 2021
searchQueryAndMode now returns a string to indicate the mode, as opposed
to a boolean indicating searchSymbols.

For golang/go#44142

Change-Id: I2568902897ceb196ad8413e737a08e6372c89a90
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/348130
Trust: Julie Qiu <julie@golang.org>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Sep 8, 2021

Change https://golang.org/cl/348130 mentions this issue: internal/frontend: change searchSymbols to searchMode

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 8, 2021

Change https://golang.org/cl/348112 mentions this issue: internal/frontend: split searchQueryAndMode

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 8, 2021

Change https://golang.org/cl/348113 mentions this issue: internal/frontend: add symbol filters

gopherbot pushed a commit to golang/pkgsite that referenced this issue Sep 8, 2021
searchQueryAndMode is split into searchQuery and searchMode.

searchQuery now returns the raw query string, so that user can see the
query that they typed in.

For golang/go#44142

Change-Id: I696b8f65f732417c43a5290b43ba1887b0c1bc06
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/348112
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
@gopherbot
Copy link

@gopherbot gopherbot commented Sep 8, 2021

Change https://golang.org/cl/348374 mentions this issue: static/frontend/homepage: update examples

gopherbot pushed a commit to golang/pkgsite that referenced this issue Sep 9, 2021
The # symbol filter is updated so that it can be used with any word,
instead of only as a prefix to indicate the user wants to search in
symbol mode.

For example, "#foo bar" was understood to mean, search for the query
string 'foo bar', in symbol search mode.

This is now understood to mean, search for the query string 'foo bar',
in symbol search mode, where 'foo' MUST match a symbol name.

This syntax allows users to specify which symbol to search for.

postgres.Search will support filters in the next CL.

For golang/go#44142

Change-Id: Ifb69f0d2745d68b5e0c28d734b48e58cc02c1819
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/347610
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
@gopherbot
Copy link

@gopherbot gopherbot commented Sep 9, 2021

Change https://golang.org/cl/348793 mentions this issue: tests/search: modify test script format

gopherbot pushed a commit to golang/pkgsite that referenced this issue Sep 9, 2021
The script format in tests/search is modified such that:

* The mode that is written in the test file indicates the "m=<mode>"
  query param.
* The mode that is wanted after that request is determined by looking at
  the test cases. This can be different than the m=<mode> param if the
  user uses filters.

For golang/go#44142

Change-Id: Idff108852599a7777d4814f1f46bba94d2c6c369
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/348793
Trust: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Sep 9, 2021
postgres.Search now supports symbol and package filters.

tests/search is also updated to match this change.

For golang/go#44142

Change-Id: Icbd05b99ea3a74956d5726e03c8767ab6062d133
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/347611
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Sep 9, 2021
The search examples on the homepage are updated to reflect the updated
search heuristics.

For golang/go#44142

Change-Id: I2154485580d6b3c43c87cf1169e72429d709c95a
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/348374
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
@gopherbot
Copy link

@gopherbot gopherbot commented Sep 9, 2021

Change https://golang.org/cl/348814 mentions this issue: internal/frontend: add package tab query

gopherbot pushed a commit to golang/pkgsite that referenced this issue Sep 9, 2021
If a user searches for #DB, they will always be brought to the symbols
tab. However, if they click on the packages tab from that page, we want
them to see something useful.

As a result, the packages tab now has href set to:

/search?m=package&q=<query-stripped-of-filters.

For golang/go#44142

Change-Id: Iff7f3d8b3f8752a40855b816cc94354b0f7ea93f
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/348814
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
@julieqiu julieqiu removed this from the pkgsite/golang.org-features milestone Sep 13, 2021
@julieqiu julieqiu added this to the pkgsite/2021 milestone Sep 13, 2021
gopherbot pushed a commit to golang/pkgsite that referenced this issue Sep 13, 2021
For golang/go#44142

Change-Id: I79d57d96f033197298d9f2219f6acaeeabcc873a
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/349609
Trust: Julie Qiu <julie@golang.org>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Sep 13, 2021

Change https://golang.org/cl/349609 mentions this issue: migrations: drop unused columns

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 14, 2021

Change https://golang.org/cl/349896 mentions this issue: internal/postgres: update search_documents and symbol_search_documents simultaneously

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 14, 2021

Change https://golang.org/cl/349892 mentions this issue: internal: delete stale symbol_search_documents

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 14, 2021

Change https://golang.org/cl/349610 mentions this issue: migrations: add FK to search_documents.unit_id

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 17, 2021

Change https://golang.org/cl/350129 mentions this issue: internal/postgres/symbolsearch: rename package

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 17, 2021

Change https://golang.org/cl/350130 mentions this issue: internal/postgres/search: rename Query to SymbolQuery

gopherbot pushed a commit to golang/pkgsite that referenced this issue Sep 20, 2021
When the latest version of a module is inserted, delete
symbol_search_documents rows for symbols not in that version of the
package, so that stale rows are removed.

For golang/go#44142

Change-Id: I9a5c45eff4713cb765fd15b57530bf139385a3a8
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/349892
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Sep 20, 2021
internal/postgres/symbolsearch is renamed to internal/postgres/search,
since the package search queries will also be moved in the next CL.

For golang/go#44142

Change-Id: Ieee78787406e969ab838859526855b8f892438b1
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/350129
Reviewed-by: Jamal Carvalho <jamal@golang.org>
Trust: Julie Qiu <julie@golang.org>
gopherbot pushed a commit to golang/pkgsite that referenced this issue Sep 20, 2021
Query is renamed to SymbolQuery, since the package was renamed from
symbolsearch to search.

For golang/go#44142

Change-Id: I02c2f04305d604a3b4efc60708154bd53e07fd11
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/350130
Trust: Julie Qiu <julie@golang.org>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Sep 20, 2021

Change https://golang.org/cl/351092 mentions this issue: static/frontend/search-help: update for redesigned search

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 1, 2021

Change https://golang.org/cl/346169 mentions this issue: migrations: set symbol_search_documents.symbol_name not null

gopherbot pushed a commit to golang/pkgsite that referenced this issue Oct 1, 2021
Now that symbol_search_documents.symbol_name is populated in all
environments, a NOT NULL constraint is added.

For golang/go#44142

Change-Id: I04a852084a7fe36ff9da2725bd0f8e6ecb9d5524
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/346169
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Oct 11, 2021

Change https://golang.org/cl/355229 mentions this issue: internal/frontend: fix filter when not in first word

gopherbot pushed a commit to golang/pkgsite that referenced this issue Oct 14, 2021
Previously, the search filter # only defaulted to symbol search mode
when it was on the first word of a search. It now works regardless of
which word has the filter.

For golang/go#44142

Change-Id: Id32010f9071c7f5db82127a490372fdf89548fe5
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/355229
Trust: Julie Qiu <julie@golang.org>
Run-TryBot: Julie Qiu <julie@golang.org>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants