feat(k8s): add job that can spawn one off queryservice-updater#701
Conversation
8b93045 to
403442b
Compare
403442b to
9339e77
Compare
|
After manually creating the Which created a job and a pod. The job finished as intended, logging: |
| ] | ||
| ]); | ||
|
|
||
| $job = $kubernetesClient->jobs()->apply($jobSpec); |
There was a problem hiding this comment.
Should this be create instead (making it fail when the job already exists). Not sure.
There was a problem hiding this comment.
probably depends on if we think jobs should be unique per wiki or not (e.g. in the event one fails and we try again)
There was a problem hiding this comment.
Maybe we can decide this when implementing the command. Right now I would think it doesn't matter, but then I don't know which retry mechanisms we come up with lateron.
| { | ||
| $this->wikiDomain = $wikiDomain; | ||
| $this->jobsKubernetesNamespace = env('API_JOB_NAMESPACE', 'api-jobs'); | ||
| $this->jobsKubernetesNamespace = Config::get('wbstack.api_job_namespace'); |
There was a problem hiding this comment.
This is a drive by fix for a mistake made earlier on this year. It will works as before, but we probably want to update the chart at some point as well to use WBSTACK_ env vars throughout.
There was a problem hiding this comment.
Works nicely - I encountered some error last week when trying it out but I guess that was some unlucky constellation of my local cluster, today it worked just fine.
One thing I noticed that maybe we can improve (but rather related to wbstack/queryservice-updater#143 I think) is that the Updater seems to go through 100 loops even if not necessary just had an old QS Updater image
* feat(k8s): add job that can spawn one off queryservice-updater * fix: consider entity payload for job uniqueness * refactor: read env vars through config
* feat(qs): scaffold command for rebuilding queryservice data * feat: add stub methods * feat: fetch entity ids from mediawiki api * chore: log received args in dummy job for testing * refactor: use max page size * fix: use proper command name * fix: directly query namespaces table * fix: do not fail on null lexemes setting * test: add test setup for qs rebuild command * test: add test for full command run * refactor: make sparql url configurable * test: explicitly set config values in test * refactor: source command config from cli args with defaults * test: add test case for command failure * test: empty wikis shall not dispatch jobs * feat: allow passing multiple domains * fix: explicitly request json responses from api * feat(k8s): add job that can spawn one off queryservice-updater (#701) * feat(k8s): add job that can spawn one off queryservice-updater * fix: consider entity payload for job uniqueness * refactor: read env vars through config * chore: dispatch proper job * fix: kubernetes labels cannot contain commas * docs: add changelog * chore: pass exception to extra logging args * chore: add logging to command * fix: entity payload in labels exceeds allowed length
3 participants
Ticket https://phabricator.wikimedia.org/T352657
Connects wbstack/queryservice-updater#143
Note that this PR does only provide the Job class but does not yet create such jobs. This shall happen in a dedicated PR (#704) which introduces a command for creating jobs for all entities in all wikis.