Skip to content

Commit

Permalink
ARROW-7868: [Crossbow] Reduce GitHub API query parallelism
Browse files Browse the repository at this point in the history
We have triggered the abuse detection mechanism of the GitHub API on the dgx1 machine with higher bandwidth https://ci.ursalabs.org/#/builders/99/builds/261

So let set the query parallelism from an environment variable.

Closes #6439 from kszucs/ARROW-7868 and squashes the following commits:

4f5f49c <Krisztián Szűcs> limit query parallelism

Authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
Signed-off-by: Krisztián Szűcs <szucs.krisztian@gmail.com>
  • Loading branch information
kszucs committed Feb 17, 2020
1 parent 7bd02c8 commit c92cbfd
Showing 1 changed file with 8 additions and 1 deletion.
9 changes: 8 additions & 1 deletion dev/tasks/crossbow.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,10 @@

try:
import github3
_have_github3 = True
except ImportError:
github3 = object
_have_github3 = False

try:
import pygit2
Expand Down Expand Up @@ -452,6 +454,8 @@ def _parse_github_user_repo(self):
def as_github_repo(self, github_token=None):
"""Converts it to a repository object which wraps the GitHub API"""
if self._github_repo is None:
if not _have_github3:
raise ImportError('Must install github3.py')
github_token = github_token or self.github_token
username, reponame = self._parse_github_user_repo()
gh = github3.login(token=github_token)
Expand Down Expand Up @@ -904,10 +908,13 @@ def wait_until_finished(self, poll_max_minutes=120,
.format(poll_interval_minutes))
time.sleep(poll_interval_minutes * 60)

def query_assets(self, max_workers=4, ignore_prefix=None):
def query_assets(self, max_workers=None, ignore_prefix=None):
# cache the futures for later use
if not hasattr(self, '_assets'):
self._assets = []
max_workers = (
max_workers or os.environ.get('CROSSBOW_QUERY_PARALLELISM', 1)
)
with concurrent.futures.ThreadPoolExecutor(max_workers) as pool:
for task_name, task in sorted(self.tasks.items()):
# HACK: spare some queries because of the rate limit, and
Expand Down

0 comments on commit c92cbfd

Please sign in to comment.