Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nested searches #1005

Open
sqs opened this Issue Nov 15, 2018 · 12 comments

Comments

Projects
None yet
6 participants
@sqs
Copy link
Member

sqs commented Nov 15, 2018

NOTE: See #1005 (comment) for the latest update on this issue.

Also known as subquery search. This is about improving the power of search in
Sourcegraph via adding sub-queries and logical operators (AND, OR, NOT). These
sort of queries are not uncommon via the shell. Usually done by mixing shell
pipelines, find, grep -L and grep -l. Examples of things you can achieve:

  • Make it possible to find all repositories which do not contain a Dockerfile.
  • Make it possible to find results which match X, but only if the repository contains the word Y.
  • Make it possible to find results which match X, but only if the file contains the word Y.

Here are some example queries in our proposed syntax to achieve the above:

  • type:repo f:Dockerfile
  • type:repo -f:Dockerfile
  • (type:repo langserver) cxp
  • (type:file Copyright) Redistribution
  • (type:repo (file:^license or file:readme) BSD) (type:repo (type:file -f:Dockerfile)) test

Roadmap

  • support type:repo and type:file in zoekt - done
  • search providers - done
  • nested queries in searcher - done
  • performance testing and tuning of searcher
    • add endpoint showing top recent queries #3256
    • add tool or endpoint to generate a report comparing performance of search with and without nesting enabled for non-indexed search #2879
    • make non-indexed search perform as well with nesting enabled as without #3497
  • use the nested query parser for all queries, translating them to old-style query structures behind the scenes #3495
  • use the data obtained from #3256 to ensure most common queries give the same results as before
  • feature parity for repo queries (forked, archive, repogroups)
  • backwards compatibility with diff search
  • update UI to use new syntax (eg how to specify refs )
  • update sample (saved) queries
  • update documentation

Closes:

Customers:

@sqs sqs added this to the 3.0-preview milestone Nov 15, 2018

@keegancsmith

This comment has been minimized.

Copy link
Member

keegancsmith commented Dec 3, 2018

The backwards compatibility has been the big slowdown, especially with respect to the way resolvers interact in the graphql layer. I'm busy hacking in an approach which merges the two approaches (old and new), just so we can ship this for text search and then incrementally support other backends.

@nicksnyder nicksnyder modified the milestones: 3.0-preview, 3.0 Dec 14, 2018

@keegancsmith

This comment has been minimized.

Copy link
Member

keegancsmith commented Dec 14, 2018

I've updated the issue description with the latest status (see top note) and plan (see added bullet points). There are lots of bullet points to go, but they are relatively minor compared to the work done so far.

@nicksnyder nicksnyder modified the milestones: 3.0, 3.1 Jan 7, 2019

@nicksnyder

This comment has been minimized.

Copy link
Member

nicksnyder commented Jan 7, 2019

This is still important, but not within our strict definition of what is required for 3.0, so I move this to 3.1 milestone.

@hho

This comment has been minimized.

Copy link

hho commented Jan 30, 2019

Please comment on this issue any bugs you may find.

When adding count:1000 to the end of a sub-query (as happens when you press the "show more" button at the bottom of the results list), no results are shown at all.

@sqs sqs referenced this issue Jan 30, 2019

Closed

3.2 code search UX tracking issue #2081

2 of 5 tasks complete

@sqs sqs modified the milestones: 3.1, Backlog Jan 31, 2019

@sqs

This comment has been minimized.

Copy link
Member Author

sqs commented Jan 31, 2019

Moving to backlog as we are going to prioritize search UX and bugs (#2081) in 3.1 instead of shipping this.

@ijt

This comment has been minimized.

Copy link
Contributor

ijt commented Mar 13, 2019

I'm grabbing this issue based on my understanding that I'm owning this now.

@ijt ijt assigned ijt and unassigned keegancsmith Mar 13, 2019

@ijt ijt changed the title Sub-query searches Sub-query (hierarchical) searches Mar 14, 2019

@slimsag slimsag added the customer label Mar 14, 2019

@slimsag

This comment has been minimized.

Copy link
Member

slimsag commented Mar 14, 2019

@ijt @nicksnyder @sqs I think this issue and #35 will be important for us long term. No timeline in mind, but probably important for us.

If we can get them placed into a milestone other than Backlog so we have a general idea of when we think they may be done (e.g. 6mo from now or something), that would be cool. Then if we need them sooner we can dedicate more resources to them.

@slimsag slimsag referenced this issue Mar 14, 2019

Closed

Code search 3.3 tracking issue #2740

8 of 18 tasks complete
@nicksnyder

This comment has been minimized.

Copy link
Member

nicksnyder commented Mar 14, 2019

@ijt and @ijsnow are going to start working on a design document for this from a clean slate. We won't put this into a milestone until we are reasonable confident that we know when it will ship, but the timeline we have communicated to @ijsnow and @ijt is 3-6 month timeframe. We are prioritizing quality and testing over shipping this sooner.

@slimsag

This comment has been minimized.

Copy link
Member

slimsag commented Mar 14, 2019

Great, that's perfect!

@ijt ijt changed the title Sub-query (hierarchical) searches Nested searches Mar 19, 2019

@slimsag

This comment has been minimized.

Copy link
Member

slimsag commented Apr 11, 2019

@ijt @ijsnow @nicksnyder can someone update this issue to reflect our current thinking? Is this still accurate?

the timeline we have communicated to @ijsnow and @ijt is 3-6 month timeframe. We are prioritizing quality and testing over shipping this sooner.

@nicksnyder

This comment has been minimized.

Copy link
Member

nicksnyder commented Apr 11, 2019

It is up to @ijt and @ijsnow to spec out what work needs to be done prior to actually starting to work on nested search (please do this). To a first approximation this work potentially includes (but is not limited to):

  • Customer issues with our existing search
  • Adding appropriate metrics so we can track baseline search performance and then compare it to nested search perf so we know that enabling nested search won't cause per regressions.
  • Prospective improvements to our architecture that might necessary to have a good user experience with search performance
@ijt

This comment has been minimized.

Copy link
Contributor

ijt commented Apr 16, 2019

@nicksnyder based on yesterday's conversation it looks like we are going to hold off on #3261 for now and focus on getting nested querying to work first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.