Bug 1329685 - Do not run update_runnable_jobs in load_initial_data #2082
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Running update_runnable_jobs as part of load_initial_data causes a two minutes increase when deploying Treeherder and when running vagrant provision. The commit that introduced the regression is this one: mozilla@91f71e1 Not having an up-to-date runnable jobs table causes not to have any data in SETA's job priorities API for *Buildbot*. This does not affect TaskCluster since the date is loaded on the fly. The API that will have an empty list of builders: http://localhost:8000/api/project/mozilla-inbound/seta/job-priorities/?build_system_type=buildbot The slowest part of updating runnable jobs for Buildbot is when it runs for the first time. The waiting happens after this line shows in the output: > [treeherder.etl.runnable_jobs:117] Updating runnable jobs table with transformed allthethings.json data. After that completes, you can reload the empty API and you can see that it works properly. In any case, this is obviously not an issue in production (since the table is never empty), however, when working on your local environment you start with an empty runnable jobs table. This change ensures that we never return incomplete runnable jobs data.
edmorley
suggested changes
Jan 18, 2017
| def list_runnable_jobs(project, decision_task_id=None): | ||
| # Bug 1329685 - This guarantees that calling list_runnable_jobs always returns the correct data | ||
| # even when running Treeherder for the first time | ||
| _ensure_buildbot_data_is_loaded() |
There was a problem hiding this comment.
I'm slightly uneasy about adding this here - it will be triggered as part of API requests, which just feels wrong.
There are lots of things that aren't loaded by default in a Vagrant instance (eg bugscache), I think we need to decide a strategy for them all. eg one of:
- document additional ingestion, but no code changes
- add additional commands like ingest_push to allow easy population of an empty Vagrant instance
- ...
I'd like to make the change to load_initial_data.py in the meantime at least.
|
Closing PR for now, since this has moved to bug 1333243, and the solution will likely need to be different. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Running update_runnable_jobs as part of load_initial_data causes a two
minutes increase when deploying Treeherder and when running vagrant
provision.
The commit that introduced the regression is this one:
91f71e1
Not having an up-to-date runnable jobs table causes not to have any data in
SETA's job priorities API for Buildbot. This does not affect TaskCluster
since the date is loaded on the fly.
The API that will have an empty list of builders:
http://localhost:8000/api/project/mozilla-inbound/seta/job-priorities/?build_system_type=buildbot
The slowest part of updating runnable jobs for Buildbot is when it runs for the first time.
The waiting happens after this line shows in the output:
After that completes, you can reload the empty API and you can see that it works properly.
In any case, this is obviously not an issue in production (since the table is never empty), however, when working on your local
environment you start with an empty runnable jobs table.
This change ensures that we never return incomplete runnable jobs data.
This change is