Core task manager #23555

chrisdavies · 2018-09-27T00:15:15Z

Accidentally merged. See #24356 instead

This PR adds a general purpose task manager to Kibana, allowing:

Single-run and recurring tasks
Scheduling tasks to run after a specified datetime
Basic retry logic
Recovery of stalled tasks / timeouts
Stateful tasks (tracking task state across multiple runs)
Ability to adjust performance knobs
Basic coordination to prevent the same task instance from running on more than one Kibana system at a time

Testing

In a plugin's uiExports, add a task definition:

taskDefinitions: {
  foo: {
    title: 'Foo',
    createTaskRunner(context) {
      return {
        async run() {
          const json = JSON.stringify(context.taskInstance.params);
          console.log(`Running "foo" with params: ${json}`);
        },
      };
    },
  },
},

Start kibana in REPL mode:

yarn start --repl

Then, schedule some tasks in the REPL:

var { taskManager } = kbnServer.server;

taskManager.schedule({
  taskType: 'foo',
  params: { hello: 'world!' },
});

taskManager.schedule({
  taskType: 'foo',
  params: { hello: 'world!' },
  runAt: new Date(Date.now() + 2500),
});

See the README for more info, and tweak the task definition to test various scenarios.

…-scheduler

will be useful for writing cancelable x-pack tasks.

…-scheduler

clean up, tests, comments, validations, assertions, etc.

…-scheduler

elasticmachine · 2018-10-05T19:33:00Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

tsullivan · 2018-10-10T07:33:23Z

retest

elasticmachine · 2018-10-10T09:12:41Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

…ing/task-manager

… into chrisdavies-alerting/task-manager

elasticmachine · 2018-10-19T00:03:03Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

tsullivan · 2018-10-19T20:37:16Z

x-pack/plugins/task_manager/README.md

+// https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-search-after.html
+// If scope is not specified, all tasks are returned, otherwise only tasks
+// with the given scope are returned.
+const results = await manager.find({ scope: 'my-fanci-app', searchAfter: ['ids'] });


this is outdated - it should be manager.fetch

This reverts commit 8b7c513.

* Initial stab at core task manager logic * Update task_manager readme * Update task manager readme * Add cancelable helper package, OSS, and general purpose, but will be useful for writing cancelable x-pack tasks. * Make the cancellable package promise-compliant * Update task manager readme w/ reference to cancellable * Change pool from lazy to eager, add support for sub-pools per task type. * Move cancellable to task_manager, and typescriptify it. * Working proof of concept for task manager. Still have lots to do: clean up, tests, comments, validations, assertions, etc. * Add pagination support to task manager fetch * Move task manager to OSS * Remove task manager reference from x-pack * Make task_manager a valid core plugin * Modify how task resource allocation works * Remove the special case 'max' workers occupied value * Remove x-pack package.json changes * Make taskDefinitions a part of uiExports * Make task docs saved-object compliant. * Add kbnServer to the task context. * Allow tasks to have a void / undefined return type * revert x-pack change * move cancellable to src/utils * move to src/server * use afterPluginsInit hook * task_manager.ts rename * add a wrapper with a setClient method * Add tests for task runner * Break task_pool into smaller, testable pieces * return raw task doc fields for calling code * remove todo comment * helper module for default client - setClient takes a callback fn * fix misidentified field name * fix rest args warning * flatten task_pool * remove cancellable * return raw task doc fields for calling code * remove todo comment * helper module for default client - setClient takes a callback fn * fix rest args warning * typescript fixes * roll back setClient takes a callback * createTaskRunner returns an object with run/cancel functions * Test task store, tweak task poller tests * Rename num_workers to max_workers * Tweak task pool method names * Fix cancellable test, and ts errors * Rename doc to _doc * Fix the doc type in the task manager index mappings * Make task costs configurable in kibana.yml * fix a tslint warning * TaskManager.afterPluginsInit replaces circuitous stuff * addMiddleware, implement beforeSave * wip * comment * run context stuff * pretty fix * comments * lint/pretty fixes * safeties in case they don't define both beforeRun and beforeSave * task runner test constructor opts fix * Add task_manager integration tests * FIx readme mistakes, fix schedule signature * Fix integration tests * Add blurb about middleware * make a libs directory for the small things * test for extractTaskDefinitions * unit test for middleware * Comment, refactor, and test task_manager * Touch up comments * Make task store throw error if given an unknown task type * Fix sample task definition in readme * Make task manager toggle on / off based on Elasticsearch connectivity * Prevent task manager from crashing Kibana if elasticsearch plugin is disabled * Fix task manager signature * Move the task manager into the task manager mixin, fix tests. It's an uber integration object anyway, so it seemed to make sense to keep it in one place. * Fix task manager integration tests * Update the task manager index creation to use auto_expand_replicas * Fix task manager integration tests * Fix task manager integration tests * Fix the overzealous deletion of .kibana during tests * Core task manager * Allow hardcoded ID passed in * comments * don't deconstruct type and task for payload * remove uiExport support * move task_manager to x-pack/plugins * expose as client on the plugin namespace * fix tests * move plugin api integration tests * roll back readme sample task definition example * fix sample plugin link * server.taskManager * sanitizeTaskDefinitions * fix integration tests * sanitize rename * assert unintialized and check for duplicates in registerTaskDefinitions * Remove unnecessary defaults from task_manager.test.ts * Remove task manager from OSS * Remove task manager from src * Tidy up task_manager unit tests around elasticsearch status changes * Integration test for specifying ID in scheduling * Task_manager init -> preInit to allow other plugins to consume taskManager in their init. * Remove task manager integration tests from OSS

elasticmachine · 2018-10-23T00:47:21Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-10-23T20:09:13Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

* Core task manager (elastic#23555) * Initial stab at core task manager logic * Update task_manager readme * Update task manager readme * Add cancelable helper package, OSS, and general purpose, but will be useful for writing cancelable x-pack tasks. * Make the cancellable package promise-compliant * Update task manager readme w/ reference to cancellable * Change pool from lazy to eager, add support for sub-pools per task type. * Move cancellable to task_manager, and typescriptify it. * Working proof of concept for task manager. Still have lots to do: clean up, tests, comments, validations, assertions, etc. * Add pagination support to task manager fetch * Move task manager to OSS * Remove task manager reference from x-pack * Make task_manager a valid core plugin * Modify how task resource allocation works * Remove the special case 'max' workers occupied value * Remove x-pack package.json changes * Make taskDefinitions a part of uiExports * Make task docs saved-object compliant. * Add kbnServer to the task context. * Allow tasks to have a void / undefined return type * revert x-pack change * move cancellable to src/utils * move to src/server * use afterPluginsInit hook * task_manager.ts rename * add a wrapper with a setClient method * Add tests for task runner * Break task_pool into smaller, testable pieces * return raw task doc fields for calling code * remove todo comment * helper module for default client - setClient takes a callback fn * fix misidentified field name * fix rest args warning * flatten task_pool * remove cancellable * return raw task doc fields for calling code * remove todo comment * helper module for default client - setClient takes a callback fn * fix rest args warning * typescript fixes * roll back setClient takes a callback * createTaskRunner returns an object with run/cancel functions * Test task store, tweak task poller tests * Rename num_workers to max_workers * Tweak task pool method names * Fix cancellable test, and ts errors * Rename doc to _doc * Fix the doc type in the task manager index mappings * Make task costs configurable in kibana.yml * fix a tslint warning * TaskManager.afterPluginsInit replaces circuitous stuff * addMiddleware, implement beforeSave * wip * comment * run context stuff * pretty fix * comments * lint/pretty fixes * safeties in case they don't define both beforeRun and beforeSave * task runner test constructor opts fix * Add task_manager integration tests * FIx readme mistakes, fix schedule signature * Fix integration tests * Add blurb about middleware * make a libs directory for the small things * test for extractTaskDefinitions * unit test for middleware * Comment, refactor, and test task_manager * Touch up comments * Make task store throw error if given an unknown task type * Fix sample task definition in readme * Make task manager toggle on / off based on Elasticsearch connectivity * Prevent task manager from crashing Kibana if elasticsearch plugin is disabled * Fix task manager signature * Move the task manager into the task manager mixin, fix tests. It's an uber integration object anyway, so it seemed to make sense to keep it in one place. * Fix task manager integration tests * Update the task manager index creation to use auto_expand_replicas * Fix task manager integration tests * Fix task manager integration tests * Fix the overzealous deletion of .kibana during tests * Core task manager * Allow hardcoded ID passed in * comments * don't deconstruct type and task for payload * remove uiExport support * move task_manager to x-pack/plugins * expose as client on the plugin namespace * fix tests * move plugin api integration tests * roll back readme sample task definition example * fix sample plugin link * server.taskManager * sanitizeTaskDefinitions * fix integration tests * sanitize rename * assert unintialized and check for duplicates in registerTaskDefinitions * Remove unnecessary defaults from task_manager.test.ts * Remove task manager from OSS * Remove task manager from src * Tidy up task_manager unit tests around elasticsearch status changes * Integration test for specifying ID in scheduling * Task_manager init -> preInit to allow other plugins to consume taskManager in their init. * Remove task manager integration tests from OSS * spelling * readme fix * fix test code impacted by hapi upgrade * Task Manager index creation changes (elastic#24542) * Uses putTemplate for task manager mappings and index settings. * Removes create option. * Opposite day, the test catches up to the code changes. * Ignores index unavailable during searches. * Improve taskRunner's processResult (elastic#24880) * Set task state to idle after run, and add failed state * fix tests * Test alerting demo (elastic#25136) * Ensures that task store init gets called before schedule. * Removes unused option for debugging purposes. * Fix unit tests because a second callCluster was made. * Task manager starts sanely. Does not preInit Will not allow definitions after initialization Creates store immediately. Modifies store after all plugins have initialized Adds static tasks that will be defined by plugins. * Task manager should not allow operations before initialization. * Attempts to fix runner tests. * Fixes unit test contract with APIs. * Removes unused type definitions. * Removes unused package json. * Removes unused import type. * Removes unnecessary async applied to a function. * Returns diferently if task store has already initialized. * Explains how to add static task instances to task definitions. (elastic#25880) * Tasks cannot be scheduled until task manager is initialized. * Adds task manager api integration tests to ci group4. * Context of describe test must be the test framework object. * Update src/es_archiver/lib/indices/kibana_index.js regex update to actually disallow non-kibana system indices Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com> * verify fillPool exception passing * readme update about max_workers/numWorkers * change forEach to reduce * use public interfaces in internal method * replace getMaxAttempts with public readonly maxAttempts * Update x-pack/plugins/task_manager/task_store.ts `throw new Error` and initializattion spell fix Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com> * min = 1 for max_workers Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com> * timeOut => timeout * min 1 * scope as an array * no retries Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com> * ConcreteTaskInstance is a TaskInstance * remove guard per joi logic * more return types for functions * more comments around incremental back-off Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com> * throw error instead of return undefined * poll_interval min 1000 * avoid handling err.stack directly * break up processResult * fix a few runtime issues * only fetch idle tasks * remove check for status idle * always return a state, and when there is an error return the state we had at beforeRun * check isStarted before attemptWork Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com> * ts fix

* Core task manager (#23555) * Initial stab at core task manager logic * Update task_manager readme * Update task manager readme * Add cancelable helper package, OSS, and general purpose, but will be useful for writing cancelable x-pack tasks. * Make the cancellable package promise-compliant * Update task manager readme w/ reference to cancellable * Change pool from lazy to eager, add support for sub-pools per task type. * Move cancellable to task_manager, and typescriptify it. * Working proof of concept for task manager. Still have lots to do: clean up, tests, comments, validations, assertions, etc. * Add pagination support to task manager fetch * Move task manager to OSS * Remove task manager reference from x-pack * Make task_manager a valid core plugin * Modify how task resource allocation works * Remove the special case 'max' workers occupied value * Remove x-pack package.json changes * Make taskDefinitions a part of uiExports * Make task docs saved-object compliant. * Add kbnServer to the task context. * Allow tasks to have a void / undefined return type * revert x-pack change * move cancellable to src/utils * move to src/server * use afterPluginsInit hook * task_manager.ts rename * add a wrapper with a setClient method * Add tests for task runner * Break task_pool into smaller, testable pieces * return raw task doc fields for calling code * remove todo comment * helper module for default client - setClient takes a callback fn * fix misidentified field name * fix rest args warning * flatten task_pool * remove cancellable * return raw task doc fields for calling code * remove todo comment * helper module for default client - setClient takes a callback fn * fix rest args warning * typescript fixes * roll back setClient takes a callback * createTaskRunner returns an object with run/cancel functions * Test task store, tweak task poller tests * Rename num_workers to max_workers * Tweak task pool method names * Fix cancellable test, and ts errors * Rename doc to _doc * Fix the doc type in the task manager index mappings * Make task costs configurable in kibana.yml * fix a tslint warning * TaskManager.afterPluginsInit replaces circuitous stuff * addMiddleware, implement beforeSave * wip * comment * run context stuff * pretty fix * comments * lint/pretty fixes * safeties in case they don't define both beforeRun and beforeSave * task runner test constructor opts fix * Add task_manager integration tests * FIx readme mistakes, fix schedule signature * Fix integration tests * Add blurb about middleware * make a libs directory for the small things * test for extractTaskDefinitions * unit test for middleware * Comment, refactor, and test task_manager * Touch up comments * Make task store throw error if given an unknown task type * Fix sample task definition in readme * Make task manager toggle on / off based on Elasticsearch connectivity * Prevent task manager from crashing Kibana if elasticsearch plugin is disabled * Fix task manager signature * Move the task manager into the task manager mixin, fix tests. It's an uber integration object anyway, so it seemed to make sense to keep it in one place. * Fix task manager integration tests * Update the task manager index creation to use auto_expand_replicas * Fix task manager integration tests * Fix task manager integration tests * Fix the overzealous deletion of .kibana during tests * Core task manager * Allow hardcoded ID passed in * comments * don't deconstruct type and task for payload * remove uiExport support * move task_manager to x-pack/plugins * expose as client on the plugin namespace * fix tests * move plugin api integration tests * roll back readme sample task definition example * fix sample plugin link * server.taskManager * sanitizeTaskDefinitions * fix integration tests * sanitize rename * assert unintialized and check for duplicates in registerTaskDefinitions * Remove unnecessary defaults from task_manager.test.ts * Remove task manager from OSS * Remove task manager from src * Tidy up task_manager unit tests around elasticsearch status changes * Integration test for specifying ID in scheduling * Task_manager init -> preInit to allow other plugins to consume taskManager in their init. * Remove task manager integration tests from OSS * spelling * readme fix * fix test code impacted by hapi upgrade * Task Manager index creation changes (#24542) * Uses putTemplate for task manager mappings and index settings. * Removes create option. * Opposite day, the test catches up to the code changes. * Ignores index unavailable during searches. * Improve taskRunner's processResult (#24880) * Set task state to idle after run, and add failed state * fix tests * Test alerting demo (#25136) * Ensures that task store init gets called before schedule. * Removes unused option for debugging purposes. * Fix unit tests because a second callCluster was made. * Task manager starts sanely. Does not preInit Will not allow definitions after initialization Creates store immediately. Modifies store after all plugins have initialized Adds static tasks that will be defined by plugins. * Task manager should not allow operations before initialization. * Attempts to fix runner tests. * Fixes unit test contract with APIs. * Removes unused type definitions. * Removes unused package json. * Removes unused import type. * Removes unnecessary async applied to a function. * Returns diferently if task store has already initialized. * Explains how to add static task instances to task definitions. (#25880) * Tasks cannot be scheduled until task manager is initialized. * Adds task manager api integration tests to ci group4. * Context of describe test must be the test framework object. * Update src/es_archiver/lib/indices/kibana_index.js regex update to actually disallow non-kibana system indices Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com> * verify fillPool exception passing * readme update about max_workers/numWorkers * change forEach to reduce * use public interfaces in internal method * replace getMaxAttempts with public readonly maxAttempts * Update x-pack/plugins/task_manager/task_store.ts `throw new Error` and initializattion spell fix Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com> * min = 1 for max_workers Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com> * timeOut => timeout * min 1 * scope as an array * no retries Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com> * ConcreteTaskInstance is a TaskInstance * remove guard per joi logic * more return types for functions * more comments around incremental back-off Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com> * throw error instead of return undefined * poll_interval min 1000 * avoid handling err.stack directly * break up processResult * fix a few runtime issues * only fetch idle tasks * remove check for status idle * always return a state, and when there is an error return the state we had at beforeRun * check isStarted before attemptWork Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com> * ts fix

chrisdavies and others added 30 commits June 22, 2018 16:35

Initial stab at core task manager logic

bfb3cbc

Merge master

c567e44

Merge branch 'master' of github.com:elastic/kibana into alerting/task…

d9c89fc

…-scheduler

Update task_manager readme

9e22ae1

Update task manager readme

b8a345c

Add cancelable helper package, OSS, and general purpose, but

1d2e4a4

will be useful for writing cancelable x-pack tasks.

Make the cancellable package promise-compliant

8f5e1e3

Update task manager readme w/ reference to cancellable

63f643f

Change pool from lazy to eager, add support for sub-pools per task type.

b9f2af8

Merge branch 'master' of github.com:elastic/kibana into alerting/task…

da940c9

…-scheduler

Move cancellable to task_manager, and typescriptify it.

f887d43

Working proof of concept for task manager. Still have lots to do:

9597456

clean up, tests, comments, validations, assertions, etc.

Add pagination support to task manager fetch

3516c6b

Merge branch 'master' of github.com:elastic/kibana into alerting/task…

5350ba7

…-scheduler

Move task manager to OSS

4681eb2

Merge branch 'master' of github.com:elastic/kibana into alerting/task…

6904d62

…-scheduler

Remove task manager reference from x-pack

f239ef0

Make task_manager a valid core plugin

5ed3df9

Modify how task resource allocation works

ae9d517

Remove the special case 'max' workers occupied value

f038a31

Merge branch 'master' of github.com:elastic/kibana into alerting/task…

3799252

…-scheduler

Remove x-pack package.json changes

e60e8dc

Make taskDefinitions a part of uiExports

063f6f7

Make task docs saved-object compliant.

6fea07d

Add kbnServer to the task context.

274d6b5

Allow tasks to have a void / undefined return type

e740eb0

revert x-pack change

6012f31

move cancellable to src/utils

b2a014f

move to src/server

6452315

use afterPluginsInit hook

3630e8c

elastic deleted a comment from elasticmachine Oct 5, 2018

pickypg self-requested a review October 10, 2018 16:43

tsullivan added 2 commits October 18, 2018 13:49

Merge remote-tracking branch 'upstream/master' into chrisdavies-alert…

869f619

…ing/task-manager

Merge branch 'alerting/task-manager' of github.com:chrisdavies/kibana…

ebcc15b

… into chrisdavies-alerting/task-manager

tsullivan reviewed Oct 19, 2018

View reviewed changes

Merge branch 'master' into chrisdavies-alerting/task-manager

7139fe9

tsullivan added the v7.0.0 label Oct 22, 2018

tsullivan merged commit 8b7c513 into elastic:master Oct 22, 2018

tsullivan deleted the alerting/task-manager branch October 22, 2018 18:25

tsullivan added a commit that referenced this pull request Oct 22, 2018

Revert "Core task manager (#23555)"

0f50053

This reverts commit 8b7c513.

tsullivan mentioned this pull request Oct 22, 2018

Revert "Core task manager" #24352

Merged

tsullivan added a commit that referenced this pull request Oct 22, 2018

Revert "Core task manager (#23555)" (#24352)

a239ca2

This reverts commit 8b7c513.

tsullivan mentioned this pull request Oct 22, 2018

Core task manager #24356

Merged

epixa added reverted and removed v6.5.0 v7.0.0 labels Oct 22, 2018

epixa removed the request for review from pickypg October 22, 2018 20:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Core task manager #23555

Core task manager #23555

chrisdavies commented Sep 27, 2018 •

edited by epixa

Loading

elasticmachine commented Oct 5, 2018

tsullivan commented Oct 10, 2018

elasticmachine commented Oct 10, 2018

elasticmachine commented Oct 19, 2018

tsullivan Oct 19, 2018

elasticmachine commented Oct 23, 2018

elasticmachine commented Oct 23, 2018

Core task manager #23555

Core task manager #23555

Conversation

chrisdavies commented Sep 27, 2018 • edited by epixa Loading

Accidentally merged. See #24356 instead

Testing

elasticmachine commented Oct 5, 2018

💔 Build Failed

tsullivan commented Oct 10, 2018

elasticmachine commented Oct 10, 2018

💔 Build Failed

elasticmachine commented Oct 19, 2018

💔 Build Failed

tsullivan Oct 19, 2018

Choose a reason for hiding this comment

elasticmachine commented Oct 23, 2018

💚 Build Succeeded

elasticmachine commented Oct 23, 2018

💔 Build Failed

chrisdavies commented Sep 27, 2018 •

edited by epixa

Loading