CIAM-6389 Pagination for groups/subgroups in Admin Console groups user groups

[eatikrh]

This PR is a proof-of-concept to help with the discussion of the approach to solve the performance issues with many groups/subgroups in searches. The main idea is not to populate the full subgroups but to only populate a single line of ancestry from the groups in the search result set to the top-level group. Fine-grained permissions are checked on the result set. Below are the changes involved with this POC.

GroupDialogPicker.tsx

• switched places of null and filter on line 83: isSearching ? null : { search: filter },
• made it a non-top-level group search on line 95: count = (await adminClient.groups.count({ search: filter, top: true }))

GroupTree.tsx

• made it a non-toplevel group search on line 95: const count = (await adminClient.groups.count({ search, top: true }))

RealmCacheSession.java

• added searchForGroupByNameNoAncestryStream method

JpaRealmProvider.java

• Changed the definition of getGroupsCountByNameContaining method this method was using searchForGroupByNameStream to count the results. Which creates a hierarchy to top level groups resulting in counting the top level groups and in my dataset this was always 1. By only counting the group ids using the name query "getGroupIdsByNameContaining" we are avoiding database hits that was used create the group representation objects.
• added searchForGroupByNameNoAncestryStream method

GroupStorageManager.java

• added searchForGroupByNameNoAncestryStream method

MapGroupProvider.java

• Made the same performance improvement in getGroupsCountByNameContaining() method as in JpaRealmProvider
• added searchForGroupByNameNoAncestryStream method

GroupsResource.java

• in listGroups method we are now calling the searchForGroupByNameNoAncestryStream method. we are applying the fine grained permissions on the groups in the result set, then converting to single ancestor hierarchy using the newly introduced GroupUtils.toAncestorsLine() method.

GroupLookupProvider.java

• added searchForGroupByNameNoAncestryStream method to the interface

ModelToRepresentation.java

• added searchForGroupByNameNoAncestryStream method

GroupsResource.java

• modified getGroups endpoint method to use searchForGroupByNameNoAncestryStream if a search term is used. (this needs more tests in different usages. since this is part of the group API we may not be aware of all the usages.)

GroupUtils.java

• This is where the main logic of single-line-ancestry is implemented: toAncestorsLine method uses a static class GroupRepresentationExtended based on GroupRepresentation class to keep track of the parent ids. it also uses a HashMap to keep track of processed groups which eliminates repeated database hits.
  List tree keeps the top-level groups with only the subgroups
  linking the search results are populated.

HardcodedGroupStorageProvider.java

• added searchForGroupByNameNoAncestryStream method

[cypress]

1 failed and 1 flaky tests on run #8538 ↗︎

1

526

48

0

1

Details:

Merge 00ccd4f into 080b53f...
Project: Keycloak Admin UI	Commit: e0160b3054 ℹ️
Status: Failed	Duration: 20:38 💡
Started: Aug 21, 2023 1:06 PM	Ended: Aug 21, 2023 1:27 PM

  cypress/e2e/clients_test.spec.ts • 1 failed test • chrome

View Output Video

Test		Artifacts
Clients test > Service account tab test > Assign		Screenshots Video

  cypress/e2e/realm_settings_tabs_test.spec.ts • 1 flaky test • chrome

View Output Video

Test		Artifacts
Realm settings tabs tests > Accessibility tests for realm settings > Check a11y violations on client policies tab/ creating policy		Output Screenshots

_{This comment has been generated by cypress-bot as a result of this project's GitHub integration settings.}

[ghost]

Unreported flaky test detected, please review

[ghost]

Unreported flaky test detected

If the below flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.actions.AppInitiatedActionTotpSetupTest#setupOtpPolicyChangedHotp

Keycloak CI - Base IT (1)

org.openqa.selenium.TimeoutException: 
java.net.SocketTimeoutException: Read timed out
Build info: version: '3.14.0', revision: 'aacccce0', time: '2018-08-02T20:19:58.91Z'
System info: host: 'fv-az573-208', ip: '10.1.0.134', os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.0-1042-azure', java.version: '17.0.8'
Driver info: driver.version: HtmlUnitDriver
...

Report flaky test

org.keycloak.testsuite.actions.AppInitiatedActionTotpSetupTest#setupTotpRegisterManual

Keycloak CI - Base IT (1)

java.lang.AssertionError
	at org.junit.Assert.fail(Assert.java:87)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at org.junit.Assert.assertTrue(Assert.java:53)
	at org.keycloak.testsuite.actions.AppInitiatedActionTotpSetupTest.setupTotpRegisterManual(AppInitiatedActionTotpSetupTest.java:197)
...

Report flaky test

org.keycloak.testsuite.actions.AppInitiatedActionTotpSetupTest#setupTotpExisting

Keycloak CI - Base IT (1)

java.lang.AssertionError: Event expected
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at org.junit.Assert.assertNotNull(Assert.java:713)
	at org.keycloak.testsuite.AssertEvents.poll(AssertEvents.java:82)
...

Report flaky test

[ghost]

Unreported flaky test detected, please review

[ghost]

Unreported flaky test detected

If the below flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.forms.VerifyProfileTest#testAttributeRequiredButNotSelectedByScopeIsNotRenderedOnVerificationScreenForcedByAnotherAttribute

Keycloak CI - Forms IT (chrome)

java.lang.IllegalArgumentException: No enum constant org.keycloak.testsuite.pages.AppPage.RequestType.
	at java.base/java.lang.Enum.valueOf(Enum.java:273)
	at org.keycloak.testsuite.pages.AppPage$RequestType.valueOf(AppPage.java:56)
	at org.keycloak.testsuite.pages.AppPage.getRequestType(AppPage.java:49)
	at jdk.internal.reflect.GeneratedMethodAccessor554.invoke(Unknown Source)
...

Report flaky test

org.keycloak.testsuite.forms.RecoveryAuthnCodesAuthenticatorTest#test03AuthenticateRecoveryAuthnCodes

Keycloak CI - Forms IT (firefox)

org.openqa.selenium.TimeoutException: 
Navigation timed out after 10000 ms
Build info: version: '3.14.0', revision: 'aacccce0', time: '2018-08-02T20:19:58.91Z'
System info: host: 'fv-az162-859', ip: '10.1.0.82', os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.0-1041-azure', java.version: '17.0.8'
Driver info: org.openqa.selenium.firefox.FirefoxDriver
...

Report flaky test

[mhajas]

@eatikrh org.keycloak.testsuite.forms.RecoveryAuthnCodesAuthenticatorTest#test03AuthenticateRecoveryAuthnCodes
is unrelated to your changes, it also failed here: #22463 I reported it as a flaky test here: #22691

[eatikrh]

@mhajas thank you for lookinng into it.

[ghost]

Unreported flaky test detected

If the below flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.forms.RecoveryAuthnCodesAuthenticatorTest#test02SetupRecoveryAuthnCodesLogoutOtherSessionsNotChecked

Keycloak CI - Forms IT (chrome)

java.lang.IllegalArgumentException: No enum constant org.keycloak.testsuite.pages.AppPage.RequestType.
	at java.base/java.lang.Enum.valueOf(Enum.java:273)
	at org.keycloak.testsuite.pages.AppPage$RequestType.valueOf(AppPage.java:56)
	at org.keycloak.testsuite.pages.AppPage.getRequestType(AppPage.java:49)
	at jdk.internal.reflect.GeneratedMethodAccessor554.invoke(Unknown Source)
...

Report flaky test

[ghost]

Unreported flaky test detected, please review

[eatikrh]

This has been superseded by PR PR#22700