Support GitHub advanced search for repository selection #482

faec · 2018-10-22T23:20:26Z

Resolves #123.

With this change, if github.repositoryQuery is set to anything other than "public", "affiliated" or "none" it is treated as a GitHub advanced repository search.

This PR updates CHANGELOG.md

sourcegraph-bot · 2018-10-22T23:20:54Z

Thanks for the contribution, @faec!

You should receive feedback on your pull request within a few days. If you haven't already, please read through the contributing guide, and ensure that you've signed the CLA.

Did you run into any issues when creating this PR? Please describe them in an issue so we can make the experience better for the next contributor.

…pos' field

faec · 2018-10-23T21:02:20Z

Tests are still pending, but the implementation is ready for review.

nicksnyder · 2018-10-23T21:21:11Z

cmd/repo-updater/repos/github.go

+					var repos []*github.Repository
+					var rateLimitCost int
+					var err error
+					repos, hasNextPage, rateLimitCost, err = c.client.ListRepositoriesForSearch(ctx, repositoryQuery, page)


If the search endpoint has a separate rate limit then I think there is a problem here because the remaining rate limit is stored in the client. If a single client is shared between search and non-search requests, then the rate limit will keep getting overwritten.

Ah, you're right. Fundamentally the client assumes that there's only one rate limit to track, and it'll use whichever one it last processed a response from.

How fancy do we want to get to do the right thing here? On the simple end is just using a fixed delay for the search queries, since we only make them from one place so far. At the fancier end the client could maintain multiple rate limits, maybe keyed by the API being called. It might also be possible to factor rate information out of the client and get it from individual responses as needed.

My inclination is to go with a simple fixed delay for the initial checkin, but I don't have much context for the importance here, so let me know if you'd prefer something more precise.

A simple suggestion is to have two clients in the githubConnection struct. For this change you would add a new client c.searchClient that is only used right here. That way the rate limits are tracked independently.

When I made that change I realized that this also creates two independent repository caches, which is probably undesirable. I think either the client needs to be aware there are multiple rate limits, or the cache needs to be moved up to the connection struct. Or we could punt and put in a one-off delay here unless / until we have another caller for the search API.

My instinct would be to share the cache across two clients.

Done. I tried to do it with minimal exposure of client internals to the connection, see if this approach looks right to you.

faec · 2018-10-24T20:26:37Z

As for tests, I tried the existing tests but the build seems to be broken:

fae@faefriend ~/g/s/g/s/s/c/r/repos> go test
# github.com/sourcegraph/docsite/markdown
../../../../docsite/markdown/markdown.go:40:18: *blackfriday.Markdown is not a type
../../../../docsite/markdown/markdown.go:79:2: undefined: blackfriday.HTMLRenderer
# github.com/sourcegraph/sourcegraph/pkg/rcache
../../../pkg/rcache/mutex.go:37:35: cannot use pool (type *"github.com/garyburd/redigo/redis".Pool) as type redsync.Pool in array or slice literal:
        *"github.com/garyburd/redigo/redis".Pool does not implement redsync.Pool (wrong type for Get method)
                have Get() "github.com/garyburd/redigo/redis".Conn
                want Get() "github.com/gomodule/redigo/redis".Conn
FAIL    github.com/sourcegraph/sourcegraph/cmd/repo-updater/repos [build failed]

Is this a genuine problem in the current repo or am I misconfigured somehow?

nicksnyder · 2018-10-24T20:44:16Z

It looks like you have Sourcegraph checked out in GOPATH, which means Go modules are turned off by default. I filed #512 to fix our documentation.

In the meantime you can try GO111MODULE=on go test or just move your sourcegraph/sourcegraph checkout outside of gopath (I recommend the latter).

…ndependent rate limit

faec · 2018-10-25T15:52:15Z

Moving the repo out of GOPATH fixed the testing problem, thanks! There is still a broken test at HEAD though:

-- FAIL: TestUnmarshal (0.00s)
    client_test.go:53: Unexpected error message [{"FieldA": 1}]
        got:  graphql: cannot unmarshal at offset 13: before "[{\"FieldA\": 1"; after "}]": json: cannot unmarshal number into Go struct field result.FieldA of type string
        want: graphql: cannot unmarshal at offset 13: before "[{\"FieldA\": 1"; after "}]": json: cannot unmarshal number into Go value of type string
    client_test.go:53: Unexpected error message [{"FieldA": "hi", "FieldB": "bye"},{"FieldA": "hi"...
        got:  graphql: cannot unmarshal at offset 3414: before ", \"FieldB\": \"bye\"},{\"FieldA\": \"hi\", \"FieldB\": \"bye\"},{\"FieldA\": \"hi\", \"FieldB\": \"bye\"}, {\"FieldA\": 1"; after "}, {\"FieldA\": \"hi\", \"FieldB\": \"bye\"},{\"FieldA\": \"hi\", \"FieldB\": \"bye\"},{\"FieldA\": \"hi\", \"FieldB\": \"b": json: cannot unmarshal number into Go struct field result.FieldA of type string
        want: graphql: cannot unmarshal at offset 3414: before ", \"FieldB\": \"bye\"},{\"FieldA\": \"hi\", \"FieldB\": \"bye\"},{\"FieldA\": \"hi\", \"FieldB\": \"bye\"}, {\"FieldA\": 1"; after "}, {\"FieldA\": \"hi\", \"FieldB\": \"bye\"},{\"FieldA\": \"hi\", \"FieldB\": \"bye\"},{\"FieldA\": \"hi\", \"FieldB\": \"b": json: cannot unmarshal number into Go value of type string
FAIL
exit status 1
FAIL    github.com/sourcegraph/sourcegraph/cmd/repo-updater/internal/externalservice/github     0.026s

That is no longer a blocking problem for me though, so I'll proceed.

nicksnyder · 2018-10-25T17:02:23Z

I filed an issue for the failing test
#523

cmd/repo-updater/internal/externalservice/github/client.go

cmd/repo-updater/internal/externalservice/github/repos.go

…tests

faec · 2018-10-26T19:21:01Z

Addressed comments and added unit tests.

I tried adding a test around the independent rate limits too, since listAllRepositories is hard to test in its current form but that was the most subtle part of the change, but got stalled: I can only mock the Client responses from within the github package which defines it, but the multiple rate limits are only visible from the repos package that defines githubConnection. I can refactor further to make it possible, but wanted to check in (and see if you had alternate suggestions) before putting more hours into it.

nicksnyder

Thanks for adding tests!

nicksnyder · 2018-10-26T19:28:08Z

Can you update the changelong and PR description?

faec · 2018-10-26T19:39:56Z

Done on changelog, but the description looks up to date to me ("Support GitHub advanced search for repository selection" if that's what you meant), should I be looking somewhere else?

…

On Oct 26, 2018, at 3:28 PM, Nick Snyder ***@***.***> wrote: Can you update the changelong and PR description? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#482 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/APSnfczBUYmc51zVkDIHy44UDEQhJVaPks5uo2JLgaJpZM4X0YL4>.

nicksnyder · 2018-10-26T20:41:18Z

Support GitHub advanced search for repository selection

That is the title, I am referring to the first comment on this thread (which will become part of the commit message when we squash merge this into master).

For example, you can delete this part at a minimum:

This is still in progress. The initial implementation uses the "orgs" property suggested by @nicksnyder, but the plan now is to revise it to follow @sqs's ideas later in the thread.

And optionally add a short description.

faec · 2018-10-26T21:21:32Z

Ah! Sorry, I haven't done code review on GitHub before, I didn't realize which parts were included :-) Done.

…

On Oct 26, 2018, at 4:41 PM, Nick Snyder ***@***.***> wrote: Support GitHub advanced search for repository selection That is the title, I am referring to the first comment on this thread (which will become part of the commit message when we squash merge this into master). For example, you can delete this part at a minimum: This is still in progress. The initial implementation uses the "orgs" property suggested by @nicksnyder <https://github.com/nicksnyder>, but the plan now is to revise it to follow @sqs <https://github.com/sqs>'s ideas later in the thread. And optionally add a short description. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#482 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/APSnff9wtcyidEUglNVe4_xY3QGgXaxkks5uo3NwgaJpZM4X0YL4>.

nicksnyder · 2018-10-26T21:36:05Z

Thanks! Tests for listAllRepositories would be nice but seems out of scope for this change. The fact that the clients are different makes it clear that the rate limits are different.

@sqs any comments?

sqs

Looks great!

This reverts commit 9a434e6.

Co-authored-by: Renovate Bot <bot@renovateapp.com>

Add an "orgs" property parallel to "repos" in GitHubConnection.

c5a8997

faec requested review from beyang and sqs as code owners October 22, 2018 23:20

sourcegraph-bot added the new-contributor label Oct 22, 2018

Make repositoryQuery support arbitrary github queries, remove the 're…

90d0078

…pos' field

faec changed the title ~~WIP: Add organization-level repo selection for GitHub hosts~~ Support GitHub advanced search for repository selection Oct 23, 2018

nicksnyder reviewed Oct 23, 2018

View reviewed changes

nicksnyder mentioned this pull request Oct 24, 2018

Update dev instructions to recommend not checking out in GOPATH #512

Closed

Fae Charlton added 2 commits October 25, 2018 11:44

Merge branch 'master' of https://github.com/sourcegraph/sourcegraph

4d72ae5

Add a second client to githubConnection to support the search API's i…

e98d3e0

…ndependent rate limit

nicksnyder reviewed Oct 26, 2018

View reviewed changes

cmd/repo-updater/internal/externalservice/github/client.go Outdated Show resolved Hide resolved

cmd/repo-updater/internal/externalservice/github/repos.go Outdated Show resolved Hide resolved

Fae Charlton added 2 commits October 26, 2018 15:07

Break the github proxy substitution into a separate helper; add unit …

df9a6a5

…tests

Move variable declaration

783c6e1

nicksnyder approved these changes Oct 26, 2018

View reviewed changes

Update changelog

4829720

sqs approved these changes Oct 26, 2018

View reviewed changes

nicksnyder merged commit 9a434e6 into sourcegraph:master Oct 27, 2018

beyang added a commit that referenced this pull request Oct 27, 2018

Revert "Support GitHub advanced search for repository selection (#482)"

2610c11

This reverts commit 9a434e6.

nicksnyder pushed a commit that referenced this pull request Oct 30, 2018

Support GitHub advanced search for repository selection (#482)

958d78f

nicksnyder pushed a commit that referenced this pull request Oct 30, 2018

Support GitHub advanced search for repository selection (#482)

0ee5fb4

This was referenced Oct 30, 2018

2.13 test plan #607

Closed

GitHub code intelligence doesn't work after soft page load; works after hard page load #750

Closed

BolajiOlajide pushed a commit that referenced this pull request Apr 16, 2024

Update alpine Docker tag to v3.13.2 (#482)

56a8639

Co-authored-by: Renovate Bot <bot@renovateapp.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support GitHub advanced search for repository selection #482

Support GitHub advanced search for repository selection #482

faec commented Oct 22, 2018 •

edited

sourcegraph-bot commented Oct 22, 2018

faec commented Oct 23, 2018

nicksnyder Oct 23, 2018

faec Oct 23, 2018

nicksnyder Oct 23, 2018

faec Oct 24, 2018

nicksnyder Oct 24, 2018

faec Oct 25, 2018

faec commented Oct 24, 2018

nicksnyder commented Oct 24, 2018

faec commented Oct 25, 2018 •

edited

nicksnyder commented Oct 25, 2018

faec commented Oct 26, 2018

nicksnyder left a comment

nicksnyder commented Oct 26, 2018

faec commented Oct 26, 2018 via email

nicksnyder commented Oct 26, 2018

faec commented Oct 26, 2018 via email

nicksnyder commented Oct 26, 2018

sqs left a comment

Support GitHub advanced search for repository selection #482

Support GitHub advanced search for repository selection #482

Conversation

faec commented Oct 22, 2018 • edited

sourcegraph-bot commented Oct 22, 2018

faec commented Oct 23, 2018

nicksnyder Oct 23, 2018

Choose a reason for hiding this comment

faec Oct 23, 2018

Choose a reason for hiding this comment

nicksnyder Oct 23, 2018

Choose a reason for hiding this comment

faec Oct 24, 2018

Choose a reason for hiding this comment

nicksnyder Oct 24, 2018

Choose a reason for hiding this comment

faec Oct 25, 2018

Choose a reason for hiding this comment

faec commented Oct 24, 2018

nicksnyder commented Oct 24, 2018

faec commented Oct 25, 2018 • edited

nicksnyder commented Oct 25, 2018

faec commented Oct 26, 2018

nicksnyder left a comment

Choose a reason for hiding this comment

nicksnyder commented Oct 26, 2018

faec commented Oct 26, 2018 via email

nicksnyder commented Oct 26, 2018

faec commented Oct 26, 2018 via email

nicksnyder commented Oct 26, 2018

sqs left a comment

Choose a reason for hiding this comment

faec commented Oct 22, 2018 •

edited

faec commented Oct 25, 2018 •

edited