Skip to content

Commit

Permalink
Add description of (almost) all missing tasks
Browse files Browse the repository at this point in the history 
Plus updated shadow integrity check description.
  • Loading branch information
@mederly
mederly committed Feb 4, 2022
1 parent 502b3d4 commit cfc101b
Show file tree
Hide file tree
Showing 2 changed files with 149 additions and 87 deletions.
105 changes: 54 additions & 51 deletions docs/resources/shadow/integrity-check.adoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,94 +7,97 @@
:page-wiki-metadata-modify-date: 2021-01-22T15:26:59.446+01:00
:page-upkeep-status: yellow

In order to diagnose and fix specific problems with shadow objects in repository, a new task type was created: Shadow Integrity Check Task.
In order to diagnose and fix specific problems with shadow objects in repository, a new task (activity) type was created: Shadow Integrity Check.

How to use it: just import the following task (taken from link:https://github.com/Evolveum/midpoint-samples/blob/master/samples/tasks/task-check-shadow-integrity.xml[samples/tasks/task-check-shadow-integrity.xml]).
== How to use it

Just import the following task:

[source,xml]
----
<task xmlns="http://midpoint.evolveum.com/xml/ns/public/common/common-3">
<name>Shadow Integrity Check</name>
<extension xmlns:mext="http://midpoint.evolveum.com/xml/ns/public/model/extension-3">
<mext:objectQuery>
<q:paging xmlns:q="http://prism.evolveum.com/xml/ns/public/query-3">
<q:orderBy>name</q:orderBy> <!-- in order to work with repositories that implement iteration by paging (so the order is not changed via fix operations) -->
</q:paging>
</mext:objectQuery>
<mext:fix>normalization</mext:fix>
</extension>
<ownerRef oid="00000000-0000-0000-0000-000000000002" type="UserType"/>
<executionStatus>runnable</executionStatus>
<handlerUri>http://midpoint.evolveum.com/xml/ns/public/model/shadow-integrity-check/handler-3</handlerUri>
<recurrence>single</recurrence>
<executionState>runnable</executionState>
<activity>
<work>
<shadowIntegrityCheck/>
</work>
</activity>
</task>
----

There are the following configuration options, in task extension container:
== Configuration Parameters

. *objectQuery*: Specifies which shadows to process.
The default is all shadows.
Note that for databases that implement iterative searches via paging (taking e.g. records 1-100, then 101-200, etc) it is advisable to define an order-by criterion, which is independent on fixes carried out.
E.g. if the fix applied is the normalization of attributes, then the shadow object name is OK, because it is not changed in the process.
Such databases are H2 and MySQL.
[%autowidth]
|===
| Parameter | Description | Default
| `shadows`
| What xref:/midpoint/reference/tasks/activities/object-set-specification/[shadows] to analyze.
| All shadows.
| `aspectsToDiagnose`
| Aspects that should be diagnosed.
| All.
| `aspectsToFix`
| Aspects that should be fixed.
| None.
| `duplicateShadowsResolver`
| Class name of the duplicate shadows resolver to be used.
| Built-in one.
| `checkDuplicatesOnPrimaryIdentifiersOnly`
| Whether to check uniqueness on primary identifiers (or on secondary ones as well).
| `false` (meaning all identifiers)
|===

. *diagnose* and *fix*: Specify which problems to diagnose (default = everything except "fetch") and which to fix (default = none).
== Aspects being Diagnosed and Fixed

Currently supported checks are:

[%autowidth]
|===
| Keyword | Description | Support

| normalization
| Checks whether all identifiers in a shadow are in a normalized form with regards to matching rule specified in the schemaHandling section.
| `normalization`
| Checks whether all identifiers in a shadow are in a normalized form with regard to matching rule specified in the `schemaHandling` section.
(Note: before running this, please check that you have consistently specified matching rule for all intents of a given kind of objects.
In future, we'll check that setting automatically, but currently we do not.)
In the future, we'll check that setting automatically, but currently we do not.)
| diagnose + fix


| uniqueness
| `uniqueness`
| Checks whether there are more shadows (of given resource+kind) bound to specific identifier value, e.g. icfs:name or icfs:uid or whatever.
| diagnose + fix (++*++)


| intents
| Checks whether shadow has intent specified.
Issues a warning if it has not.
When fixing, fetches the resource object and tries to determine correct intent.
| `intents`
| Checks whether shadow has intent specified. When fixing, fetches the resource object and tries to determine correct intent.
| diagnose + fix


| extraData
| `extraData`
| Checks for extra data in shadow activation container.
As of midPoint 3.3, we store only enableTimestamp, disableTimestamp, archiveTimestamp and disableReason there.
All other data (e.g. administrativeStatus) have to be fetched from the resource on the fly.
This check ensures that the superfluous data are not stored in the shadows, and removes those data that are there (e.g. after migration from pre-3.3 versions of midPoint).
Currently, we store only `enableTimestamp`, `disableTimestamp`, `archiveTimestamp` and `disableReason` there.
All other data (e.g. `administrativeStatus`) have to be fetched from the resource on the fly.
This check ensures that the superfluous data are not stored in the shadows, and removes those data
that are there.
| diagnose + fix


| owners
| Checks whether shadow has at least one owner + if the synchronization situation (LINKED) is consistent with owners found - LINKED shadow has to have at least one owner, not-LINKED shadow has to have 0 owners.
| `owners`
| Checks whether shadow has at least one owner + if the synchronization situation (`LINKED`) is consistent
with owners found. The `LINKED` shadow has to have at least one owner, not-`LINKED` shadow has to have 0 owners.
| diagnose only


| fetch
| `existenceOnResource`
| Checks whether the corresponding object can be fetched from the resource.
Because of the performance implications, this check is not carried out by default.
| diagnose only


| resourceRef
| Checks whether resource pointed to by the shadow really exists, including whether resource reference is set at all.
(Actually, these checks cannot be turned off.) *Fix is done by removing the shadow, so please use with care.*
| `resourceRef`
| Checks whether resource pointed to by the shadow really exists, including whether resource
reference is set at all. (Actually, these checks cannot be turned off.) *Fix is done by removing the shadow, so please use with care.*
| diagnose + fix


|===

(++*++) Note that the test checking for uniqueness is implemented purely in RAM, so it works for shadow sets of reasonable size (e.g. 100K entries, depending on amount of your heap).
It is faster and simpler to implement.
If necessary, just log a JIRA issue to implement more general solution that would work with shadow sets of any size.
(++*++) Note that the test checking for uniqueness is implemented purely in RAM, so it works for shadow
sets of reasonable size (e.g. 100K entries, depending on amount of your heap). It is faster and simpler to implement.

== Output

The task produces a report that is written to midPoint log file.
It looks like this:
Expand Down Expand Up @@ -144,4 +147,4 @@ Duplicates for resource: .............. (OID:..............), kind = ENTITLEMENT
--> deleted redundant shadow (skipped because of dry run) shadow: CN=.............. (OID:16fc34ab-5acc-4320-8db1-6f707d2ee9ac)
2015-09-03 12:39:23,464 [] [midPointScheduler_Worker-6] INFO (com.evolveum.midpoint.model.impl.util.AbstractSearchIterativeTaskHandler): Finished Shadow integrity check (Task(id:1441204963732-0-1, name:A Shadow Integrity Check, oid:db2ce226-c13f-4731-ab2e-9bad11223fe0)). Processed 3020 objects in 180 seconds, got 3 errors. Average time for one object: 56.21755 milliseconds (wall clock time average: 59.927483 ms).
----
----
131 changes: 95 additions & 36 deletions docs/tasks/activities/work.adoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -206,48 +206,107 @@ appended to the default time-scanning query.
| How "stale" must be a shadow to be deleted.
| Obligatory.

| `objectIntegrityCheck`
| x
| Simple
| N
| N
| x
| x
| x
.2+| `objectIntegrityCheck`
.2+| The purpose of this activity is to detect and optionally fix anomalies in repository objects.
However, currently its only function is to display information about objects size.
.2+| Simple
.2+| N
.2+| N
| `objects`
| What xref:../object-set-specification/[objects] to analyze.
| All objects in repository.
| `histogramColumns`
| Number of columns in the histograms of object sizes.
| 80

| `shadowIntegrityCheck`
| x
| Simple
| N
| N
| x
| x
| x
.5+| `shadowIntegrityCheck`
.5+| The purpose of this activity is to detect and optionally anomalies in resource object shadows
stored in the repository. See xref:/midpoint/reference/resources/shadow/integrity-check/[the details].
.5+| Simple
.5+| N
.5+| N
| `shadows`
| What xref:../object-set-specification/[shadows] to analyze.
| All shadows.
| `aspectsToDiagnose`
| Aspects that should be diagnosed.
| All.
| `aspectsToFix`
| Aspects that should be fixed.
| None.
| `duplicateShadowsResolver`
| Class name of the duplicate shadows resolver to be used.
| Built-in one.
| `checkDuplicatesOnPrimaryIdentifiersOnly`
| Whether to check uniqueness on primary identifiers (or on secondary ones as well).
| `false` (meaning all identifiers)

| `activityAutoScaling`
| x
.4+| `activityAutoScaling`
.4+| Adapts running activities to the current cluster configuration by executing "reconcile workers" operation
when the cluster state changes (i.e. a node goes up or down).
.4+| Simple
.4+| Y
.4+| Y
| `tasks`
| xref:../object-set-specification/[Tasks] that should be considered for auto-scaling.
(Any task must be multi-node, running, and the auto-scaling for it must not be disabled.)
| All eligible tasks.
| `minReconciliationInterval`
| Minimal interval between reconciliations, i.e. it will not reconcile workers if this time has not elapsed since the last one.
| 0
| `maxReconciliationInterval`
| Maximal interval between reconciliations, i.e. it will _always_ reconcile workers after this time has elapsed since the last one.
| infinity
| `skipInitialReconciliation`
| Whether to skip initial reconciliation of the workers on the activity start.
| `false`

| `propagation`
| TODO
| Simple
| Y
| Y
| x
| x
| x
|
|
|
|
|

| `noOp`
| x
| `multiPropagation`
| TODO
| Simple
| Y
| Y
| x
| x
| x
|
|
|
|
|

.3+| `noOp`
.3+| "No operation" activity to be used for demonstration and testing purposes. It consists of a set of numbered steps,
each taking a given time.
.3+| Simple
.3+| Y
.3+| Y
| `delay`
| Duration of each step in milliseconds.
| 0
| `steps`
| Number of steps.
| 1
| `stepInterruptibility`
| Should a step of this activity be interruptible? Value of `none` means that
the activity is not interruptible, except at step boundaries. If `hard` is used,
the activity can be interrupted only in "hard" way, i.e. by sending the Java interrupt signal.
Finally, `full` means that the activity can be interrupted any time.
| `none`

| `extension`
| x
| This is a place for customer-provided activities.
| Simple
| -
| -
| x
| x
| x
|
|
|
|
|
|===

Customer-provided activities can have any configuration items and distribution (multi-threading,
clustering) properties.

0 comments on commit cfc101b

Please sign in to comment.