Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bitbucketserver: Document rate limit change and rationale #9048

Merged
merged 4 commits into from Mar 17, 2020
Merged
Changes from all commits
Commits
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.

Always

Just for now

@@ -34,6 +34,7 @@ All notable changes to Sourcegraph are documented in this file.
- Archived repositories are excluded from search by default. Adding `archived:yes` includes archived repositories.
- Forked repositories are excluded from search by default. Adding `fork:yes` includes forked repositories.
- CSRF and session cookies now set `SameSite=None` when Sourcegraph is running behind HTTPS and `SameSite=Lax` when Sourcegraph is running behind HTTP in order to comply with a [recent IETF proposal](https://web.dev/samesite-cookies-explained/#samesitenone-must-be-secure). As a side effect, the Sourcegraph browser extension and GitLab/Bitbucket native integrations can only connect to private instances that have HTTPS configured. If your private instance is only running behind HTTP, please configure your instance to use HTTPS in order to continue using these.
- The Bitbucket Server rate limit that Sourcegraph self-imposes has been raised from 120 req/min to 480 req/min to account for Sourcegraph instances that make use of Sourcegraphs' Bitbucket Server repository permissions and campaigns at the same time (which require a larger number of API requests aginst Bitbucket). The new number is based on us consuming roughly 8% the average API request rate against at large customers' Bitbucket Server instances. [#9048](https://github.com/sourcegraph/sourcegraph/pull/9048/files)

This comment has been minimized.

Copy link
@unknwon

unknwon Mar 17, 2020

Member

Is this a typo? s/aginst/against/?

This comment has been minimized.

Copy link
@tsenart

tsenart Mar 17, 2020

Contributor

Omg, I typed a typo while fixing a typo 🤦‍♂

- If a single, unambiguous commit SHA is used in a search query (e.g., `repo@c98f56`) and a search index exists at this commit (i.e., it is the `HEAD` commit), then the query is searched using the index. Prior to this change, unindexed search was performed for any query containing an `@commit` specifier.

### Fixed
@@ -36,8 +36,19 @@ var requestCounter = metrics.NewRequestMeter("bitbucket", "Total number of reque
//
// See https://godoc.org/golang.org/x/time/rate#Limiter for an explanation of these fields.
//
// We chose the limits here based on the fact that Sourcegraph is a heavy consumer of the Bitbucket
// Server API and that a large customer had reported to us their Bitbucket instance receives
// ~100 req/s so it seems reasonable for us to (at max) consume ~8 req/s.
//
// Note that, for comparison, Bitbucket Cloud restricts "List all repositories" requests (which are
// a good portion of our requests) to 1,000/hr, and they restrict "List a user or team's repositories"
// requests (which are roughly equal to our repository lookup requests) to 1,000/hr. We perform a list
// repositories request for every 1000 repositories on Bitbucket every 1m by default, so for someone
// with 20,000 Bitbucket repositories we need 20,000/1000 requests per minute (1200/hr) + overhead for
// repository lookup requests by users, and requests for identifying which repositories a user has
// access to (if authorization is in use) and requests for campaign synchronization if it is in use.
const (
rateLimitRequestsPerSecond = 8
rateLimitRequestsPerSecond = 8 // 480/min or 28,800/hr
RateLimitMaxBurstRequests = 500
)

ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.