Skip to content

Commit

Permalink
ninja: more docs
Browse files Browse the repository at this point in the history
  • Loading branch information
@1azyman
1azyman committed Feb 29, 2024
1 parent 583a6fd commit 7b8931c
Show file tree
Hide file tree
Showing 4 changed files with 128 additions and 127 deletions.
75 changes: 74 additions & 1 deletion docs/deployment/ninja/command/initial-objects.adoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,77 @@
[NOTE]
Experimental feature.

#TODO#
This is the last step of upgrade procedure while midPoint is still down.
It is necessary to update initial objects to make sure that they are compatible with new version of midPoint and midPoint can start and work properly.

IMPORTANT: When upgrading LTS (from 4.4 -> 4.8) it is necessary to upgrade at least securityPolicy object.
Reason for this is that there were changes in authentication/authorization processing for different channels and configuration related to it, hence securityPolicy.
See <<Important updates>> for more details.

Ninja contains command `initial-objects` to help you update set of initial objects (see `ninja.sh help initial-objects` for more details).
Initial objects command will help add, update or replace existing objects in repository.

All necessary objects are bundled in directly in ninja, however, it is possible to use custom set of new initial objects using `--file` option.
`--file` options can be used multiple times or have multiple values separated by comma to specify multiple directories/files.

Ninja also supports filtered processing using `--type`, `--oid` and `--reverse-oid-filter`/`--reverse-type-filter`.
Such options give you possibility to process only subset of objects using this command.
E.g. replace objects without custom changes without any merge, reports or dry-run while reviewing merged objects with custom changes.

This command also supports dry-run mode and reporting.
Dry-run mode can be used to see what changes will be done in repository without actually doing them (option `--dry-run`).
Reporting can be turned on using `-r, --report` options.
Report option can be coupled with `--report-style [DELTA|FULL_OBJECT]` to specify whether XML output should contain only deltas or full objects after update.
There's also standard set of options to send output to file (`-o, --output`) and overwrite (`-O, --overwrite`).

NOTE: Initial objects that are new in upgraded version of midPoint are not added to repository by ninja automatically.
By default, these objects will be properly imported during first start of upgraded midPoint.
If one wants to import them before first start, it is possible to use `--force-add` option.

WARNING: Merging algorithm used in this ninja command is experimental and might not work properly in all cases.
Please review all changes proposed by ninja using `--dry-run` before applying them to repository.

There are two main scenarios that can be followed:

* <<No changes in initial objects>>
* <<Custom changes in initial objects>>
==== No changes in initial objects

This one is the simplest case.
If there are no custom changes in initial objects made by users, then ninja can update initial objects automatically.
Update can be done by overwriting existing ones without merge.
Following command can be used (please use `--dry-run` if you want to see what will be done without actually doing it):

[source,bash]
----
./bin/ninja.sh initial-objects --no-merge
----

==== Custom changes in initial objects

If there are custom changes in initial objects made by users, then ninja can help you merge them.
First, dry run is recommended to see what needs to be done, optionally with report of changes:

[source,bash]
----
./bin/ninja.sh initial-objects --dry-run --report --report-style FULL_OBJECT -o report-initial-objects.xml
----

If you're satisfied with changes proposed by ninja, you can execute command without `--dry-run` and report related options.
If there are objects (filtered by types or oids) that can be handled without merge, you can run:

[source,bash]
----
./bin/ninja.sh initial-objects --no-merge [--type securityPolicy,valuePolicy,...] [--oid 93ae3cd3-b34d-4093-ad49-adba573a95ba]
----

==== Important updates

There is at least one object - securityPolicy that needs to be updated when upgrading LTS from 4.4 to 4.8.
If there are no custom changes in securityPolicy, following command can be used:

[source,bash]
----
./bin/ninja.sh initial-objects --no-merge --type securityPolicy
----
47 changes: 46 additions & 1 deletion docs/deployment/ninja/command/run-sql.adoc
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,48 @@
= Run SQL

#TODO#
`run-sql` command can be used to execute SQL scripts on database.

Database schema changes are implemented in SQL scripts that are bundled with distribution in folder `doc/config/sql/native/*.sql`.
User can use `--mode [repository|audit]` switch together with `--upgrade` to run proper scripts automatically or use `--scripts` option to specify custom set of scripts.

Run-sql command can be used also to create database schema from scratch using switch `--create`.

Ninja by default uses repository/audit configuration from _midpoint-home/config.xml_ to connect to database.
This behaviour can be changed via `--jdbc-url`, `--jdbc-username` and `--jdbc-password` options.
With these options, ninja will switch to _raw_ mode, create custom JDBC connection and execute scripts on it.

If `--result` switch is used, ninja will print results for each query in script to SYSOUT, otherwise results are ignored.

.Example runs custom-upgrade-database.sql script on JDBC connection specified by url/username/password
[source,bash]
----
./bin/ninja.sh run-sql \
--jdbc-url jdbc:postgresql://localhost:5432/postgres \
--jdbc-username postgres \
--jdbc-password postgres \
--scripts ./custom-upgrade-database.sql
----

.This example runs upgrade scripts for repository on database defined in midpoint-home/config.xml
[source,bash]
----
./bin/ninja.sh run-sql \
--mode repository \
--upgrade
----

.Following example runs upgrade scripts for repository on database defined in midpoint-home/config.xml
[source,bash]
----
./bin/ninja.sh run-sql \
--mode audit \
--upgrade
----

.This example runs custom defined scripts for audit on database defined in midpoint-home/config.xml
[source,bash]
----
./bin/ninja.sh run-sql \
--mode repository \
--scripts ./upgrade/new-distribution/doc/config/sql/native/postgres-upgrade.sql
----
6 changes: 5 additions & 1 deletion docs/deployment/ninja/command/upgrade-installation.adoc
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
= Upgrade installation

#TODO#
Upgrade installation command will copy and replace files in midPoint installation directory using files from distribution.
By default, installation directory is computed as a parent of midpoint-home directory.
This behaviour can be changed using `--installation-directory <PATH>` option.

All files that would be replaced can be backed up if necessary using `--backup-installation-directory <path>` option.
127 changes: 3 additions & 124 deletions docs/deployment/ninja/use-case/upgrade-with-ninja.adoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -605,6 +605,8 @@ Please review logs of midPoint containers to make sure there are no errors.





== Upgrade procedure details

=== Upgrade tasks to use activities
Expand Down Expand Up @@ -675,134 +677,11 @@ Objects can be upgraded by using CSV report from verify:
--verification-file verify-output.csv --report-style csv
----





=== Upgrade DB schema

Database schema changes are handled by SQL scripts that are bundled with distribution in folder `doc/config/sql/native/*.sql`.
These changes can be applied using ninja command `run-sql`.
User can use `--mode [repository|audit]` switch together with `--upgrade` to run proper scripts automatically or use `--scripts` option to specify custom set of scripts.

Run-sql command can be used also to create database schema from scratch using switch `--create`.

Ninja by default uses repository/audit configuration from _midpoint-home/config.xml_ to connect to database.
This behaviour can be changed via `--jdbc-url`, `--jdbc-username` and `--jdbc-password` options.
With these options, ninja will switch to _raw_ mode, create custom JDBC connection and execute scripts on it.

If `--result` switch is used, ninja will print results for each query in script to SYSOUT, otherwise results are ignored.

.Example uses of run-sql command
[source,bash]
----
# runs custom-upgrade-database.sql script on JDBC connection specified by url/username/password
./bin/ninja.sh run-sql \
--jdbc-url jdbc:postgresql://localhost:5432/postgres \
--jdbc-username postgres \
--jdbc-password postgres \
--scripts ./custom-upgrade-database.sql
# runs upgrade scripts for repository on database defined in midpoint-home/config.xml
./bin/ninja.sh run-sql \
--mode repository \
--upgrade
# runs upgrade scripts for repository on database defined in midpoint-home/config.xml
./bin/ninja.sh run-sql \
--mode audit \
--upgrade
# runs custom defined scripts for audit on database defined in midpoint-home/config.xml
./bin/ninja.sh run-sql \
--mode repository \
--scripts ./upgrade/new-distribution/doc/config/sql/native/postgres-upgrade.sql
----

=== Upgrade installation

Upgrade installation step will copy and replace files in midPoint installation directory using files from distribution.
By default, installation directory is computed as a parent of midpoint-home directory.
This behavoiour can be changed using `--installation-directory <PATH>` option.

All files that would be replaced can be backed up if necessary using `--backup-installation-directory <path>` option.

=== Initial objects

This is the last step of upgrade procedure while midPoint is still down.
It is necessary to update initial objects to make sure that they are compatible with new version of midPoint and midPoint can start and work properly.

IMPORTANT: When upgrading LTS (from 4.4 -> 4.8) it is necessary to upgrade at least securityPolicy object.
Reason for this is that there were changes in authentication/authorization processing for different channels and configuration related to it, hence securityPolicy.
See <<Important updates>> for more details.

Ninja contains command `initial-objects` to help you update set of initial objects (see `ninja.sh help initial-objects` for more details).
Initial objects command will help add, update or replace existing objects in repository.

All necessary objects are bundled in directly in ninja, however, it is possible to use custom set of new initial objects using `--file` option.
`--file` options can be used multiple times or have multiple values separated by comma to specify multiple directories/files.

Ninja also supports filtered processing using `--type`, `--oid` and `--reverse-oid-filter`/`--reverse-type-filter`.
Such options give you possibility to process only subset of objects using this command.
E.g. replace objects without custom changes without any merge, reports or dry-run while reviewing merged objects with custom changes.

This command also supports dry-run mode and reporting.
Dry-run mode can be used to see what changes will be done in repository without actually doing them (option `--dry-run`).
Reporting can be turned on using `-r, --report` options.
Report option can be coupled with `--report-style [DELTA|FULL_OBJECT]` to specify whether XML output should contain only deltas or full objects after update.
There's also standard set of options to send output to file (`-o, --output`) and overwrite (`-O, --overwrite`).

NOTE: Initial objects that are new in upgraded version of midPoint are not added to repository by ninja automatically.
By default, these objects will be properly imported during first start of upgraded midPoint.
If one wants to import them before first start, it is possible to use `--force-add` option.

WARNING: Merging algorithm used in this ninja command is experimental and might not work properly in all cases.
Please review all changes proposed by ninja using `--dry-run` before applying them to repository.

There are two main scenarios that can be followed:

* <<No changes in initial objects>>
* <<Custom changes in initial objects>>

==== No changes in initial objects

This one is the simplest case.
If there are no custom changes in initial objects made by users, then ninja can update initial objects automatically.
Update can be done by overwriting existing ones without merge.
Following command can be used (please use `--dry-run` if you want to see what will be done without actually doing it):

[source,bash]
----
./bin/ninja.sh initial-objects --no-merge
----

==== Custom changes in initial objects

If there are custom changes in initial objects made by users, then ninja can help you merge them.
First, dry run is recommended to see what needs to be done, optionally with report of changes:

[source,bash]
----
./bin/ninja.sh initial-objects --dry-run --report --report-style FULL_OBJECT -o report-initial-objects.xml
----

If you're satisfied with changes proposed by ninja, you can execute command without `--dry-run` and report related options.
If there are objects (filtered by types or oids) that can be handled without merge, you can run:

[source,bash]
----
./bin/ninja.sh initial-objects --no-merge [--type securityPolicy,valuePolicy,...] [--oid 93ae3cd3-b34d-4093-ad49-adba573a95ba]
----

==== Important updates

There is at least one object - securityPolicy that needs to be updated when upgrading LTS from 4.4 to 4.8.
If there are no custom changes in securityPolicy, following command can be used:

[source,bash]
----
./bin/ninja.sh initial-objects --no-merge --type securityPolicy
----
For more information see xref:../index.adoc#run-sql[run-sql] command documentation.

// TODO TODO TODO what if I was using other than Default Security Policy?
// TODO TODO TODO GUI auth works, but REST (Studio) will not work
Expand Down

0 comments on commit 7b8931c

Please sign in to comment.