Skip to content

Commit

Permalink
leftovers and archive .... are we "done"?
Browse files Browse the repository at this point in the history
  • Loading branch information
@semancik
semancik committed Apr 21, 2021
1 parent da41136 commit 9e79f09
Show file tree
Hide file tree
Showing 30 changed files with 3,648 additions and 2 deletions.
87 changes: 87 additions & 0 deletions docs/concepts/clockwork/clockwork-and-projector.adoc
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
= Clockwork and Projector
:page-wiki-name: Clockwork and Projector
:page-upkeep-status: orange

Clockwork and Projector are two sophisticated component that work together to compute the changes to objects that midPoint manages and to control the execution of these changes.
These two components form a "brain" of the midPoint.

TODO: picture


== Projector

Projector is responsible for computing the changes.
It is triggered by user changes in GUI, received by service or detected by wiki:Synchronization[synchronization]. Projector recomputes all the wiki:Synchronization+Policies[synchronization policies], applies all wiki:Mappings+and+Expressions[mappings and expressions], processes wiki:Advanced+Hybrid+RBAC[RBAC] role assignments and so on.
However, projector is just computing the values.
It is not yet executing them.
The result of projector computations are stored in wiki:Model+Context[model context].

Projector distinguished objects into two categories: wiki:Focus+and+Projections[Focus and Projections]

* *Focus* which is typically wiki:UserType[user]. Focus is the object which is central to the recomputation.
The object that links to all the projections.

* *Projection* which is typically an account (represented by wiki:Shadow+Objects[shadow]).

Projector reflects data from the focus to projections (using wiki:Outbound+Mapping[outbound mappings]) and also from the projections to the focus (using wiki:Inbound+Mapping[inbound mappings]). This happens as many times as necessary to get a complete computation result.


== Clockwork

Clockwork is a controller.
Clockwork invokes projector to recompute everything that needs to be recomputed.
Clockwork invokes wiki:Hooks[hooks] to further process the request (e.g. to route it through a wiki:Workflows+(midPoint+3.x)[workflow]). This happens several times.
Clockwork then proceeds to execute the changes.
This may also happen several times if wiki:Provisioning+Dependencies[dependencies] are used.
Clockwork handles execution errors.
And at the end of execution it once again invokes hooks (e.g. to handle wiki:Notifications[notifications]).


== Clockwork States

The clockwork process goes through several stages:

* *INITIAL* state.
Nothing has been done yet.
The wiki:Model+Context[model context] is filled with a request.
But no recomputation has yet taken place.
The hooks can use this state to inspect or change the request even before any serious action takes place.

* *PRIMARY* state.
Request is recomputed.
The wiki:Model+Context[Model Context] was recomputed and contains the preview of changes.
The hooks can use this state to change the original request.
Nothing has been executed yet.
This is often used to influence assignments by routing the request through approval workflow.
E.g. the workflow can remove unapproved role assignments from the request.

* *SECONDARY* state.
Request is recomputed once again.
The wiki:Model+Context[model context] is recomputed to reflect the changes from the primary state.
The changes are executed after this state is completed.
The hooks can use this state to inspect the values of projections.
E.g. it is sometimes used to let system administrators approve the actual changes of account attributes before they are executed.
The hooks can be invoked several times as this state may be processed several times when wiki:Provisioning+Dependencies[provisioning dependencies] are used.

* *FINAL* state.
All the changes are executed.
The wiki:Model+Context[model context] reflects the state after the execution.
This state can be used by the hooks to inspect the results.
E.g. this is often used to send out the notifications about the completed operations.


== TODO

TODO: more details


== See Also

* wiki:Model+Context[Model Context]

* wiki:Focus+and+Projections[Focus and Projections]

* wiki:Hooks[Hooks]

* wiki:Projector+and+clockwork+internals[Projector and clockwork internals] (only some sketches there - yet)

Binary file added docs/concepts/clockwork/model-context/clockwork-projector-model-context.png
View file
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
75 changes: 75 additions & 0 deletions docs/concepts/clockwork/model-context/index.adoc
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
= Model Context
:page-wiki-name: Model Context
:page-upkeep-status: yellow

Model context is a complex data structure that describes the context of computation in wiki:Clockwork+and+Projector[Clockwork and Projector] components.
It is also used in wiki:Hooks[Hooks], wiki:Scripting+Hooks[Scripting Hooks], wiki:Workflows+(midPoint+3.x)[Workflows (midPoint 3.x)] and other components that take part in the computation.

image::clockwork-projector-model-context.png[]



The model context is putting together the wiki:Focus+and+Projections["focus" and "projections"] that belong together.
Focus is usually a user, projections are accounts.
In such a case the model context groups together a user with all the accounts are associated with that accounts.
These are usually accounts that are linked to the user.
But it also may be a new account that was not yet created, an old account that was recently deleted, etc.
MidPoint groups all these objects together to allow efficient computation of assignments and mappings and other policies.

Model Context has three parts (see wiki:Clockwork+and+Projector[Clockwork and Projector] page for explanation of terms used here):

* The *context itself* contains information about the entire computation (such as computation state).
These can be accessed by methods invoked directly on the ModelContext.
* *Focus* part which contains information about focal object.
There is at most one focus (see wiki:Focus+and+Projections[Focus and Projections]).
* *Projection* part which contains information about each projection.
There may be multiple projections (see wiki:Focus+and+Projections[Focus and Projections]).
Focus and projection parts have similar structure.
Both of these parts contain:

* *Old object*: the object (focus or projection) as it was before the computation.
This means really the beginning of computation.
Please note that the computation can take several days e.g. if the request waits for approval.
* *Current object*: the object as it was last time the Projector loaded the object.
This is usually quite recent information (at most few seconds old).
* *New object*: the expected form of new object after the computation.
This item is here mostly for informational purposes and for diagnostics.
The actual value of the result may be slightly different (e.g. if two operations are carried out over the same object in parallel).
* *Primary delta*: The request delta.
This is the delta that was explicitly entered in the GUI, supplied to the web service or otherwise specified in wiki:IDM+Model+Interface[IDM Model Interface] invocation.
This is the "command" that midPoint should execute.
This defines what user _wants_. This delta will be executed exactly as it was specified.
* *Secondary delta*: The computed delta.
Secondary delta originates from execution of mappings or hooks or other automated mechanisms.
This describes what midPoint has _computed_. This delta will be executed, but it can be recomputed several times during the process.
* *Synchronization delta*: The detected delta.
The delta that was detected by synchronization.
MidPoint assumes that this delta was already executed and all it can do is to react to this.
It is used as an input to the computation.
This delta will *not* be executed again.
All the objects and deltas are in wiki:Prism+Objects[prism representation].


== Implementation Notes

"Model Context" is a public name of this data structure as it is exposed by the wiki:IDM+Model+Interface[IDM Model Interface] (midPoint Model API).
It is internally implemented by a very rich LensContext data structure.
It is designed to be serializable therefore it can be used in long-running asynchronous processes that include human tasks (such as workflows).
It is serialized in an upgradeable data format (currently XML) which allows running processes (such as workflows waiting for approval) to survive system upgrades.


== See Also

* wiki:Clockwork+and+Projector[Clockwork and Projector]

* wiki:Focus+and+Projections[Focus and Projections]

154 changes: 154 additions & 0 deletions docs/concepts/clockwork/model-context/serialization.adoc
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
= Model context serialization
:page-wiki-name: Model context serialization
:page-upkeep-status: yellow

Although in most cases the model context lives in memory only, there are situations where it has to be stored persistently.
A typical example is an wiki:Workflows+(midPoint+3.x)[approval workflow]: a user requests an operation that has to be made, but the operation (or some parts of it) may be subject of an approval.
Therefore, the whole model operation context has to be stored until the approval is obtained.
Then it is revived, and the operation is allowed to continue.

Originally the storage of model context was implemented using java serialization.
From midPoint 2.2 onwards, we have developed an XML serialization format for the context, in order to be able to easily upgrade midPoint even with some of workflows being in-progress.

The serialized context in principle reflects the structure of native Java objects and looks like this:

[source,xml]
----
<m:modelContext xmlns:m="http://midpoint.evolveum.com/xml/ns/public/model/model-context-2">
<m:state>primary</m:state>
<m:channel>http://midpoint.evolveum.com/xml/ns/public/gui/channels-2#user</m:channel>
<m:focusContext>
<m:objectOld xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
oid="c168470c-bfef-414f-88b5-5d144f4f3d6c"
version="2"
xsi:type="UserType">
<name>
<orig xmlns="http://prism.evolveum.com/xml/ns/public/types-2">security</orig>
<norm xmlns="http://prism.evolveum.com/xml/ns/public/types-2">security</norm>
</name>
<activation>
<c:administrativeStatus>enabled</c:administrativeStatus>
</activation>
<fullName>
<orig xmlns="http://prism.evolveum.com/xml/ns/public/types-2">Security Manager</orig>
<norm xmlns="http://prism.evolveum.com/xml/ns/public/types-2">security manager</norm>
</fullName>
<credentials>
<password>
<c:value xmlns:ds="http://www.w3.org/2000/09/xmldsig#"
xmlns:enc="http://www.w3.org/2001/04/xmlenc#"
xsi:type="c:ProtectedStringType">
<enc:EncryptedData Type="http://www.w3.org/2001/04/xmlenc#Element">
<enc:EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#aes128-cbc"/>
<ds:KeyInfo>
<ds:KeyName>4HXeUejV93Vd3JuIZz7sbs5bVko=</ds:KeyName>
</ds:KeyInfo>
<enc:CipherData>
<enc:CipherValue>iimZ2lYWAEatFmD8C3ZLp0Kw+Bp0+de6vQTfNiE5aK6A1CVDx1homddXQzc7edYL</enc:CipherValue>
</enc:CipherData>
</enc:EncryptedData>
</c:value>
</password>
<allowedIdmAdminGuiAccess>true</allowedIdmAdminGuiAccess>
</credentials>
</m:objectOld>
<m:objectNew xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
oid="c168470c-bfef-414f-88b5-5d144f4f3d6c"
version="2"
xsi:type="UserType">
<name>
<orig xmlns="http://prism.evolveum.com/xml/ns/public/types-2">security</orig>
<norm xmlns="http://prism.evolveum.com/xml/ns/public/types-2">security</norm>
</name>
<assignment>
<targetRef xmlns:c="http://midpoint.evolveum.com/xml/ns/public/common/common-2a"
oid="12345678-d34d-b33f-f00d-000187987988"
type="c:RoleType"/>
</assignment>
<activation>
<c:administrativeStatus>enabled</c:administrativeStatus>
<c:effectiveStatus>enabled</c:effectiveStatus>
<enableTimestamp>2013-06-25T17:32:10.091+02:00</enableTimestamp>
</activation>
<fullName>
<orig xmlns="http://prism.evolveum.com/xml/ns/public/types-2">Security Manager</orig>
<norm xmlns="http://prism.evolveum.com/xml/ns/public/types-2">security manager</norm>
</fullName>
<credentials>
<password>
<c:value xmlns:ds="http://www.w3.org/2000/09/xmldsig#"
xmlns:enc="http://www.w3.org/2001/04/xmlenc#"
xsi:type="c:ProtectedStringType">
<enc:EncryptedData Type="http://www.w3.org/2001/04/xmlenc#Element">
<enc:EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#aes128-cbc"/>
<ds:KeyInfo>
<ds:KeyName>4HXeUejV93Vd3JuIZz7sbs5bVko=</ds:KeyName>
</ds:KeyInfo>
<enc:CipherData>
<enc:CipherValue>iimZ2lYWAEatFmD8C3ZLp0Kw+Bp0+de6vQTfNiE5aK6A1CVDx1homddXQzc7edYL</enc:CipherValue>
</enc:CipherData>
</enc:EncryptedData>
</c:value>
</password>
<allowedIdmAdminGuiAccess>true</allowedIdmAdminGuiAccess>
</credentials>
</m:objectNew>
<m:primaryDelta xmlns:icfs="http://midpoint.evolveum.com/xml/ns/public/connector/icf-1/resource-schema-2"
xmlns:t="http://prism.evolveum.com/xml/ns/public/types-2"
xmlns:icfc="http://midpoint.evolveum.com/xml/ns/public/connector/icf-1/connector-schema-2"
xmlns:c="http://midpoint.evolveum.com/xml/ns/public/common/common-2a"
xmlns:q="http://prism.evolveum.com/xml/ns/public/query-2"
xmlns:cap="http://midpoint.evolveum.com/xml/ns/public/resource/capabilities-2"
xmlns:apti="http://midpoint.evolveum.com/xml/ns/public/common/api-types-2"
xmlns:wfcf="http://midpoint.evolveum.com/xml/ns/model/workflow/common-forms-2"
xmlns:ds="http://www.w3.org/2000/09/xmldsig#"
xmlns:enc="http://www.w3.org/2001/04/xmlenc#">
<t:changeType>modify</t:changeType>
<t:objectType>c:UserType</t:objectType>
<t:oid>c168470c-bfef-414f-88b5-5d144f4f3d6c</t:oid>
</m:primaryDelta>
<m:objectTypeClass>com.evolveum.midpoint.xml.ns._public.common.common_2a.UserType</m:objectTypeClass>
<m:oid>c168470c-bfef-414f-88b5-5d144f4f3d6c</m:oid>
<m:secondaryDeltas xmlns:icfs="http://midpoint.evolveum.com/xml/ns/public/connector/icf-1/resource-schema-2"
xmlns:t="http://prism.evolveum.com/xml/ns/public/types-2"
xmlns:icfc="http://midpoint.evolveum.com/xml/ns/public/connector/icf-1/connector-schema-2"
xmlns:c="http://midpoint.evolveum.com/xml/ns/public/common/common-2a"
xmlns:q="http://prism.evolveum.com/xml/ns/public/query-2"
xmlns:cap="http://midpoint.evolveum.com/xml/ns/public/resource/capabilities-2"
xmlns:apti="http://midpoint.evolveum.com/xml/ns/public/common/api-types-2"
xmlns:wfcf="http://midpoint.evolveum.com/xml/ns/model/workflow/common-forms-2"
xmlns:ds="http://www.w3.org/2000/09/xmldsig#"
xmlns:enc="http://www.w3.org/2001/04/xmlenc#">
<m:wave number="0">
<m:delta>
<t:changeType>modify</t:changeType>
<t:objectType>c:UserType</t:objectType>
<t:oid>c168470c-bfef-414f-88b5-5d144f4f3d6c</t:oid>
<t:modification>
<t:modificationType>replace</t:modificationType>
<t:path>c:activation</t:path>
<t:value>
<c:effectiveStatus>enabled</c:effectiveStatus>
</t:value>
</t:modification>
<t:modification>
<t:modificationType>replace</t:modificationType>
<t:path>c:activation</t:path>
<t:value>
<c:enableTimestamp xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:type="xsd:dateTime">2013-06-25T17:32:10.091+02:00</c:enableTimestamp>
</t:value>
</t:modification>
</m:delta>
</m:wave>
</m:secondaryDeltas>
</m:focusContext>
<m:focusClass>com.evolveum.midpoint.xml.ns._public.common.common_2a.UserType</m:focusClass>
<m:projectionClass>com.evolveum.midpoint.xml.ns._public.common.common_2a.ShadowType</m:projectionClass>
<m:doReconciliationForAllProjections>false</m:doReconciliationForAllProjections>
<m:projectionWave>2</m:projectionWave>
<m:executionWave>0</m:executionWave>
</m:modelContext>
----

0 comments on commit 9e79f09

Please sign in to comment.