Skip to content

Commit

Permalink
Add description of new task features
Browse files Browse the repository at this point in the history
  • Loading branch information
mederly committed Nov 10, 2021
1 parent df37346 commit 6ee9016
Showing 1 changed file with 55 additions and 2 deletions.
57 changes: 55 additions & 2 deletions docs/tasks/activities/index.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -108,10 +108,63 @@ of each other.

#TODO#

== Migration from 4.3 and before
== Migration from 4.3 and Before

#TODO#

== GUI support
== GUI Support

#TODO#

== New Features

=== Cluster Auto-Scaling Capability

Distributed - a.k.a. multi-node - activities can be scaled up or down by starting or stopping worker tasks.
This scaling is appropriate when the distribution definition changes, e.g. number of worker tasks is increased
or decreased. It should be done also when the cluster configuration changes, e.g. nodes are added or deleted.

This up/down-scaling was present in midPoint since 3.8 (as an experimental feature) in the form of _workers
reconciliation_ process. In 4.4 the following changes were done:

1. The workers reconciliation process was improved and fixed:
- It no longer closes superfluous workers - it suspends them instead, allowing resuming them when needed.
- When suspending the workers, any unprocessed buckets are returned to the coordinator to process them
immediately by other workers. The same is true when the task stops for any other reason.
This eliminates lengthy "scavenging" process at the end of the processing.
- When in scavenging phase, non-scavenger workers are not created. Only the scavengers are.
- The whole reconciliation is now driven by node.operationalState (i.e. from repo) instead of
node.executionState (determined dynamically). So in case of unstable clusters or transition situations
the results should be more predictable.

2. New auto-scaling activity was created. It periodically scans for tasks that are autoscaling-enabled
and invokes workers reconciliation process on them, if the cluster state has changed in the meanwhile.
(Note that we currently do not support auto-scaling triggered by changes in activity distribution
definition. The user is responsible for reconciling the workers manually in that cases.)

=== Cluster-wide Thresholds

The xref:/midpoint/reference/tasks/thresholds/[threshold mechanism] before 4.4 was limited to a single node,
and the related policy rules counters were transient, so they did not survive midPoint node restarts.
There were other limitations as well.

Starting with 4.4, counters for policy rules are stored persistently in the respective task objects,
providing support for cluster-wide watching of policy rules thresholds, across task and node restarts.

=== Task Statistics Improvements

* Flexible xref:reporting/execution-reports/[activity reports] were introduced:

- buckets processed,
- items processed,
- ConnId operations executed,
- internal operations executed.

Also, bucket size analysis mode is now available. It allows determining size of buckets before
real processing starts: either all buckets, or a (random or regular) sample of specified size.

* The following information is now collected per individual activities (not per tasks):

- synchronization statistics,
- actions executed,
- work bucket management statistics.

0 comments on commit 6ee9016

Please sign in to comment.