When wikis have pending jobs, create a Kubernetes job to work through the items #603
Conversation
m90
changed the title
m90
force-pushed
the
fr/mw-job-queue
branch
3 times, most recently
from
5b892a9 to
b0bc4d9
Compare
| // Schedule site stat updates for each wiki and platform-summary | ||
| $schedule->command('schedule:stats')->daily(); | ||
|
|
||
| $schedule->job(new PollForMediaWikiJobsJob)->everyMinute(); |
There was a problem hiding this comment.
If we wanted to have this run at a higher frequency, we'd have two options:
- this package https://github.com/spatie/laravel-short-schedule
- creating a wrapper job that dispatches multiple child jobs during a minute, sleeping inbetween
I would guess the former is a bit cleaner.
|
|
||
| private function enqueueWiki (string $wikiDomain): void | ||
| { | ||
| dispatch(new ProcessMediaWikiJobsJob($wikiDomain)); |
There was a problem hiding this comment.
Do we want/need this to fan out to another job (as is) or should we rather merge the two jobs into one?
m90
force-pushed
the
fr/mw-job-queue
branch
9 times, most recently
from
413019a to
de4fe26
Compare
m90
force-pushed
the
fr/mw-job-queue
branch
3 times, most recently
from
111f98c to
a534670
Compare
m90
force-pushed
the
fr/mw-job-queue
branch
6 times, most recently
from
487b0a8 to
df08fb6
Compare
| ] | ||
| ]); | ||
|
|
||
| $job = $kubernetesClient->jobs()->apply($jobSpec); |
There was a problem hiding this comment.
Turns out that when using apply instead of create and using a unique .metadata.name per wiki, we can have Kubernetes deduplicate jobs for us and we don't have to check anything at all. Only downside is that we have to use a .spec.ttlSecondsAfterFinished of 0. Still think being able to keep this job free from any logic at all is worth it.
m90
force-pushed
the
fr/mw-job-queue
branch
2 times, most recently
from
2ca82ad to
b9939e1
Compare
deer-wmde
left a comment
deer-wmde
left a comment
There was a problem hiding this comment.
Can confirm this worked locally for me! details: https://phabricator.wikimedia.org/T330389#8895083
Follows up on #601 which my fat fingers accidentally deleted.
Ticket https://phabricator.wikimedia.org/T330389
Requires wbstack/charts#114, wmde/wbaas-deploy#885