Keycloak being able to handle 1000+ identity providers in a realm

[mposolda]

Description

This includes various tasks needed for Keycloak to be able to handle 1000+ identity providers in a realm (might be even 10k or more).

Discussion

#21076

Report from the community users about the issues with 4000 identity providers in a realm: https://issues.redhat.com/browse/KEYCLOAK-17860

Issues

Tasks

Beta Give feedback

1. Add possibility to limit field length in legacy event store #14888
   area/storage kind/bug team/store +3
   
2. Doublecheck dataset in keycloak-benchmark can be used to add thousands of identity providers in a realm keycloak-benchmark#382
   +0
3. Make sure identity providers are not send in realm GET requests and PUT requests used in "Realm settings" #21072
   kind/enhancement status/triage +2
4. Authentication flow screens in admin console with many identity providers #27053
   area/identity-brokering kind/enhancement kind/performance status/triage +4
5. Identity providers: pagination in admin REST API #21073
   kind/enhancement +1
   
6. Identity providers: pagination in model #21954
   kind/task +1
   
7. Identity providers: pagination in admin console #21074
   kind/enhancement release/24.0.0 +2
8. Identity providers: Pagination in account console (and account REST API) #21261
   kind/enhancement status/triage +2
9. Do we need to handle 1000+ identity providers on login page? #21075
   kind/enhancement status/triage +2

Relevant Open PRs

• KEYCLOAK-18061 On Linked account, paged listing of IdPs #8022
• Significant improvement on Identity Providers UI data loading #7960

Motivation

No response

[bonnm]

Hi,
Just a comment to the area of handling thousands of IdPs: Automated import/processing of IdPs from a SAML federation would also be a nice feature. There already was a discussion about this: #8608, and there is a fork https://github.com/eosc-kc/keycloak which works with 1000+ federated SAML-IdPs. Maybe it is worth to have a look at it.

[cgeorgilakis]

I believe also account console with many IdPs issue must be added to this epic.
Without fixing it, linked-accounts page will be unresponsible and user can not link his IdPs (accounts).

There is a PR for resolving this issue. Could you review it?

[cgeorgilakis]

Hi, Just a comment to the area of handling thousands of IdPs: Automated import/processing of IdPs from a SAML federation would also be a nice feature. There already was a discussion about this: #8608, and there is a fork https://github.com/eosc-kc/keycloak which works with 1000+ federated SAML-IdPs. Maybe it is worth to have a look at it.

We have implemented the SAML Federation and we could discuss in a github discussion/ github issue. SAML federation is based in supporting section 2.3.1 of Metadata for the OASIS Security Assertion Markup Language (SAML) V2.0 and it gives a way to manage 1000+ SAML IdPs - like in our use case. SAML federation support needs all that mentioned in this epic.

I propose add SAML federation as an issue in this epic and discuss details in github discussion.

[rmartinc]

Starting looking at this one. I think that the steps to make this happen are more or less the following:

• Create a new API to manage the providers. API should contain basic CRUD and a search/list with pagination enabled.
• Change the UIs (admin, login page,...) to manage the new endpoints and not the list in the realm. This step will probably be translated into different issues.
• Remove/deprecate the list in the realm (avoid loading all of them in the realm representation).
• Subsequent enhancements (like import, sync or other things that are commented in the different discussions...). Those can be managed as separated issues as we have the base to cover thousands of providers.

The first step is always having the new endpoints to manage the providers. I think that #21072 is the one to cover that point and the starting point. So I'm starting thinking about this. I will present a skeleton to see what everybody thinks about it.

[jonkoops]

@rmartinc if we're building new APIs I'd really like to get started on a proper v2 endpoint with proper Open API annotations.

[rmartinc]

@jonkoops I'm starting to look what we have now and how far we are from the goal. Probably we are almost there and we don't need a lot of things, making v2 not really worthy. We have openapi annotations now, so we will use them for sure not matter what we decide.

My idea is presenting a skeleton first so you and everybody can scold me. 😄

[jonkoops]

Hahah, sounds fair! Let me know when I can pitch in.

[rmartinc]

@jonkoops and anyone interested, I updated #21072 with the rest API changes (as expected they are minimal).