Skip to content

Commit

Permalink
For overwrite, do all deletes, then all adds.
Browse files Browse the repository at this point in the history 
Minor UI enhancements.
Fix 2 JPA bugs.
General cleanup.
Documentation.
  • Loading branch information
@ssilvert
ssilvert committed Jan 8, 2016
1 parent 979205c commit fbff61b
Show file tree
Hide file tree
Showing 20 changed files with 302 additions and 528 deletions.
10 changes: 5 additions & 5 deletions core/src/main/java/org/keycloak/representations/idm/PartialImportRepresentation.java
View file
@@ -1,5 +1,5 @@
/*
* Copyright 2015 Red Hat Inc. and/or its affiliates and other contributors
* Copyright 2016 Red Hat Inc. and/or its affiliates and other contributors
* as indicated by the @author tags. All rights reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License"); you may not
Expand All @@ -21,9 +21,9 @@
import org.codehaus.jackson.annotate.JsonIgnoreProperties;

/**
* Used for partial import of users, clients, and identity providers.
* Used for partial import of users, clients, roles, and identity providers.
*
* @author Stan Silvert ssilvert@redhat.com (C) 2015 Red Hat Inc.
* @author Stan Silvert ssilvert@redhat.com (C) 2016 Red Hat Inc.
*/
@JsonIgnoreProperties(ignoreUnknown=true)
public class PartialImportRepresentation {
Expand All @@ -49,11 +49,11 @@ public boolean hasIdps() {
}

public boolean hasRealmRoles() {
return (roles.getRealm() != null) && (!roles.getRealm().isEmpty());
return (roles != null) && (roles.getRealm() != null) && (!roles.getRealm().isEmpty());
}

public boolean hasClientRoles() {
return (roles.getClient() != null) && (!roles.getClient().isEmpty());
return (roles != null) && (roles.getClient() != null) && (!roles.getClient().isEmpty());
}

public String getIfResourceExists() {
Expand Down
220 changes: 120 additions & 100 deletions docbook/auth-server-docs/reference/en/en-US/modules/export-import.xml
View file
@@ -1,114 +1,134 @@
<chapter id="export-import">
<title>Export and Import</title>
<para>
Export/import is useful especially if you want to migrate your whole Keycloak database from one environment to another or migrate to different database (For example from MySQL to Oracle).
You can trigger export/import at startup of Keycloak server and it's configurable with System properties right now. The fact it's done at server startup means that no-one can access Keycloak UI or REST endpoints
and edit Keycloak database on the fly when export or import is in progress. Otherwise it could lead to inconsistent results.
</para>
<para>
You can export/import your database either to:
<itemizedlist>
<listitem>Directory on local filesystem</listitem>
<listitem>Single JSON file on your filesystem</listitem>
</itemizedlist>
<section>
<title>Startup export/import</title>
<para>
Export/import is useful especially if you want to migrate your whole Keycloak database from one environment to another or migrate to different database (For example from MySQL to Oracle).
You can trigger export/import at startup of Keycloak server and it's configurable with System properties right now. The fact it's done at server startup means that no-one can access Keycloak UI or REST endpoints
and edit Keycloak database on the fly when export or import is in progress. Otherwise it could lead to inconsistent results.
</para>
<para>
You can export/import your database either to:
<itemizedlist>
<listitem>Directory on local filesystem</listitem>
<listitem>Single JSON file on your filesystem</listitem>
</itemizedlist>

When importing using the "dir" strategy, note that the files need to follow the naming convention specified below.
If you are importing files which were previously exported, the files already follow this convention.
<itemizedlist>
<listitem>{REALM_NAME}-realm.json, such as "acme-roadrunner-affairs-realm.json" for the realm named "acme-roadrunner-affairs"</listitem>
<listitem>{REALM_NAME}-users-{INDEX}.json, such as "acme-roadrunner-affairs-users-0.json" for the first users file of the realm named "acme-roadrunner-affairs"</listitem>
</itemizedlist>
</para>
<para>
If you import to Directory, you can specify also the number of users to be stored in each JSON file. So if you have
very large amount of users in your database, you likely don't want to import them into single file as the file might be very big.
Processing of each file is done in separate transaction as exporting/importing all users at once could also lead to memory issues.
</para>
<para>
To export into unencrypted directory you can use:
<programlisting><![CDATA[
When importing using the "dir" strategy, note that the files need to follow the naming convention specified below.
If you are importing files which were previously exported, the files already follow this convention.
<itemizedlist>
<listitem>{REALM_NAME}-realm.json, such as "acme-roadrunner-affairs-realm.json" for the realm named "acme-roadrunner-affairs"</listitem>
<listitem>{REALM_NAME}-users-{INDEX}.json, such as "acme-roadrunner-affairs-users-0.json" for the first users file of the realm named "acme-roadrunner-affairs"</listitem>
</itemizedlist>
</para>
<para>
If you import to Directory, you can specify also the number of users to be stored in each JSON file. So if you have
very large amount of users in your database, you likely don't want to import them into single file as the file might be very big.
Processing of each file is done in separate transaction as exporting/importing all users at once could also lead to memory issues.
</para>
<para>
To export into unencrypted directory you can use:
<programlisting><![CDATA[
bin/standalone.sh -Dkeycloak.migration.action=export
-Dkeycloak.migration.provider=dir -Dkeycloak.migration.dir=<DIR TO EXPORT TO>
]]></programlisting>
And similarly for import just use <literal>-Dkeycloak.migration.action=import</literal> instead of <literal>export</literal> .
</para>
<para>
To export into single JSON file you can use:
<programlisting><![CDATA[
And similarly for import just use <literal>-Dkeycloak.migration.action=import</literal> instead of <literal>export</literal> .
</para>
<para>
To export into single JSON file you can use:
<programlisting><![CDATA[
bin/standalone.sh -Dkeycloak.migration.action=export
-Dkeycloak.migration.provider=singleFile -Dkeycloak.migration.file=<FILE TO EXPORT TO>
]]></programlisting>
</para>
<para>
Here's an example of importing:
<programlisting><![CDATA[
</para>
<para>
Here's an example of importing:
<programlisting><![CDATA[
bin/standalone.sh -Dkeycloak.migration.action=import
-Dkeycloak.migration.provider=singleFile -Dkeycloak.migration.file=<FILE TO IMPORT>
-Dkeycloak.migration.strategy=OVERWRITE_EXISTING
]]></programlisting>
</para>
<para>
Other available options are:
<variablelist>
<varlistentry>
<term>-Dkeycloak.migration.realmName</term>
<listitem>
<para>
can be used if you want to export just one specified realm instead of all.
If not specified, then all realms will be exported.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-Dkeycloak.migration.usersExportStrategy</term>
<listitem>
<para>
can be used to specify for Directory providers to specify where to import users.
Possible values are:
<itemizedlist>
<listitem>DIFFERENT_FILES - Users will be exported into more different files according to maximum number of users per file. This is default value</listitem>
<listitem>SKIP - exporting of users will be skipped completely</listitem>
<listitem>REALM_FILE - All users will be exported to same file with realm (So file like "foo-realm.json" with both realm data and users)</listitem>
<listitem>SAME_FILE - All users will be exported to same file but different than realm (So file like "foo-realm.json" with realm data and "foo-users.json" with users)</listitem>
</itemizedlist>
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-Dkeycloak.migration.usersPerFile</term>
<listitem>
<para>
can be used to specify number of users per file (and also per DB transaction).
It's 5000 by default. It's used only if usersExportStrategy is DIFFERENT_FILES
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-Dkeycloak.migration.strategy</term>
<listitem>
<para>
is used during import. It can be used to specify how to proceed if realm with same name
already exists in the database where you are going to import data. Possible values are:
<itemizedlist>
<listitem>IGNORE_EXISTING - Ignore importing if realm of this name already exists</listitem>
<listitem>OVERWRITE_EXISTING - Remove existing realm and import it again with new data from JSON file.
If you want to fully migrate one environment to another and ensure that the new environment will contain same data
like the old one, you can specify this.
</listitem>
</itemizedlist>
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
<para>
When importing realm files that weren't exported before, the option <literal>keycloak.import</literal> can be used. If more than one realm
file needs to be imported, a comma separated list of file names can be specified. This is more appropriate than the cases before, as this
will happen only after the master realm has been initialized. Examples:
<itemizedlist>
<listitem>-Dkeycloak.import=/tmp/realm1.json</listitem>
<listitem>-Dkeycloak.import=/tmp/realm1.json,/tmp/realm2.json</listitem>
</itemizedlist>
</para>

</para>
<para>
Other available options are:
<variablelist>
<varlistentry>
<term>-Dkeycloak.migration.realmName</term>
<listitem>
<para>
can be used if you want to export just one specified realm instead of all.
If not specified, then all realms will be exported.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-Dkeycloak.migration.usersExportStrategy</term>
<listitem>
<para>
can be used to specify for Directory providers to specify where to import users.
Possible values are:
<itemizedlist>
<listitem>DIFFERENT_FILES - Users will be exported into more different files according to maximum number of users per file. This is default value</listitem>
<listitem>SKIP - exporting of users will be skipped completely</listitem>
<listitem>REALM_FILE - All users will be exported to same file with realm (So file like "foo-realm.json" with both realm data and users)</listitem>
<listitem>SAME_FILE - All users will be exported to same file but different than realm (So file like "foo-realm.json" with realm data and "foo-users.json" with users)</listitem>
</itemizedlist>
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-Dkeycloak.migration.usersPerFile</term>
<listitem>
<para>
can be used to specify number of users per file (and also per DB transaction).
It's 5000 by default. It's used only if usersExportStrategy is DIFFERENT_FILES
</para>
</listitem>
</varlistentry>
<varlistentry>
<term>-Dkeycloak.migration.strategy</term>
<listitem>
<para>
is used during import. It can be used to specify how to proceed if realm with same name
already exists in the database where you are going to import data. Possible values are:
<itemizedlist>
<listitem>IGNORE_EXISTING - Ignore importing if realm of this name already exists</listitem>
<listitem>OVERWRITE_EXISTING - Remove existing realm and import it again with new data from JSON file.
If you want to fully migrate one environment to another and ensure that the new environment will contain same data
like the old one, you can specify this.
</listitem>
</itemizedlist>
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
<para>
When importing realm files that weren't exported before, the option <literal>keycloak.import</literal> can be used. If more than one realm
file needs to be imported, a comma separated list of file names can be specified. This is more appropriate than the cases before, as this
will happen only after the master realm has been initialized. Examples:
<itemizedlist>
<listitem>-Dkeycloak.import=/tmp/realm1.json</listitem>
<listitem>-Dkeycloak.import=/tmp/realm1.json,/tmp/realm2.json</listitem>
</itemizedlist>
</para>
</section>
<section>
<title>Admin console export/import</title>
<para>
Import of most resources can be performed from the admin console.
Exporting resources will be supported in future versions.
</para>
<para>
The files created during a "startup" export can be used to import from
the admin UI. This way, you can export from one realm and import to
another realm. Or, you can export from one server and import to another.
</para>
<warning>
<para>
The admin console import allows you to "overwrite" resources if you choose.
Use this feature with caution, especially on a production system.
</para>
</warning>
</section>
</chapter>
20 changes: 12 additions & 8 deletions forms/common-themes/src/main/resources/theme/base/admin/resources/js/controllers/realm.js
View file
Expand Up @@ -2229,6 +2229,17 @@ module.controller('RealmImportCtrl', function($scope, realm, $route,
}
}, true);

$scope.successMessage = function() {
var message = $scope.results.added + ' records added. ';
if ($scope.ifResourceExists === 'SKIP') {
message += $scope.results.skipped + ' records skipped.'
}
if ($scope.ifResourceExists === 'OVERWRITE') {
message += $scope.results.overwritten + ' records overwritten.';
}
return message;
}

$scope.save = function() {
var json = angular.copy($scope.fileContent);
json.ifResourceExists = $scope.ifResourceExists;
Expand All @@ -2243,14 +2254,7 @@ module.controller('RealmImportCtrl', function($scope, realm, $route,

var importFile = $resource(authUrl + '/admin/realms/' + realm.realm + '/partialImport');
$scope.results = importFile.save(json, function() {
var message = $scope.results.added + ' records added. ';
if ($scope.ifResourceExists === 'SKIP') {
message += $scope.results.skipped + ' records skipped.'
}
if ($scope.ifResourceExists === 'OVERWRITE') {
message += $scope.results.overwritten + ' records overwritten.';
}
Notifications.success(message);
Notifications.success($scope.successMessage());
}, function(error) {
if (error.data.errorMessage) {
Notifications.error(error.data.errorMessage);
Expand Down
22 changes: 10 additions & 12 deletions .../common-themes/src/main/resources/theme/base/admin/resources/partials/partial-import.html
View file
Expand Up @@ -86,6 +86,7 @@ <h1>Partial Import</h1>
</div>

<div class="form-group" data-ng-show="hasResults()">
{{successMessage()}}
<table class="table table-striped table-bordered">
<thead>
<tr>
Expand All @@ -95,26 +96,23 @@ <h1>Partial Import</h1>
<th>Id</th>
</tr>
</thead>
<tfoot>
<tr>
<td>
<div class="table-nav">
<button data-ng-click="setFirstPage()" class="first" ng-disabled="">First page</button>
<button data-ng-click="setPreviousPage()" class="prev" ng-disabled="!hasPrevious()">Previous page</button>
<button data-ng-click="setNextPage()" class="next" ng-disabled="!hasNext()">Next page</button>
</div>
</td>
</tr>
</tfoot>
<tbody>
<tr ng-repeat="result in resultsPage()" >
<td>{{result.action}}</td>
<td ng-show="result.action == 'OVERWRITTEN'"><span class="label label-danger">{{result.action}}</span></td>
<td ng-show="result.action == 'SKIPPED'"><span class="label label-warning">{{result.action}}</span></td>
<td ng-show="result.action == 'ADDED'"><span class="label label-success">{{result.action}}</span></td>
<td>{{result.resourceType}}</td>
<td>{{result.resourceName}}</td>
<td>{{result.id}}</td>
</tr>
</tbody>
</table>

<div class="table-nav">
<button data-ng-click="setFirstPage()" class="first" ng-disabled="">First page</button>
<button data-ng-click="setPreviousPage()" class="prev" ng-disabled="!hasPrevious()">Previous page</button>
<button data-ng-click="setNextPage()" class="next" ng-disabled="!hasNext()">Next page</button>
</div>
</div>
</form>
</div>
Expand Down
9 changes: 6 additions & 3 deletions model/jpa/src/main/java/org/keycloak/models/jpa/JpaUserProvider.java
View file
Expand Up @@ -76,9 +76,12 @@ public UserModel addUser(RealmModel realm, String id, String username, boolean a
userModel.joinGroup(g);
}
}
for (RequiredActionProviderModel r : realm.getRequiredActionProviders()) {
if (r.isEnabled() && r.isDefaultAction()) {
userModel.addRequiredAction(r.getAlias());

if (addDefaultRequiredActions){
for (RequiredActionProviderModel r : realm.getRequiredActionProviders()) {
if (r.isEnabled() && r.isDefaultAction()) {
userModel.addRequiredAction(r.getAlias());
}
}
}

Expand Down

0 comments on commit fbff61b

Please sign in to comment.