feat: Add SCIM scenarios; Fix findings by gtema · Pull Request #1136 · openstack-experimental/keystone

gtema · 2026-07-31T17:50:48Z

Add ScimRealmRead and ScimSync loadtest scenarios covering raft-backed
federation/SCIM (ADR 0021, 0024): realm CRUD reads/updates via admin
token, and the real IdP-connector lifecycle over /SCIM/v2 with a
per-VU API Key bearer (provision, reconcile, attribute drift,
offboard). Seed setup is now idempotent against a pre-existing SCIM
realm/IdP (no delete endpoint exists, so an interrupted prior run
leaves them behind; 409 now reuses them instead of aborting seed).

While chasing intermittent 401s surfaced by ScimSync under load,
found and fixed two real bugs in the raft-backed auth path:

Timing oracle in ApiKeyAuth: the IP-not-allowed rejection branch
returned immediately while every other rejection branch (not-found,
inactive, wrong-secret) burns an Argon2id hash first, letting an
attacker distinguish a valid-but-IP-blocked lookup_hash from a
nonexistent one by response latency (ADR 0021 Invariant 7). Now
burns the same dummy-hash cost.
Two linearizability gaps in the ReadIndex forward-to-leader path
(get_by_key/prefix/prefix_index): the leader-side gRPC handlers
served forwarded reads without re-confirming current leadership,
and follower-side callers silently fell back to an unguarded local
read whenever the forward RPC failed. Both fixed.

Signed-off-by: gtema artem.goncharov@gmail.com

github-actions · 2026-07-31T18:18:27Z

🦢 Load Test Results

Goose Attack Report

Plan Overview

Action	Started	Stopped	Elapsed	Users
Increasing	26-08-03 17:40:22	26-08-03 17:40:56	00:00:34	0 → 45
Maintaining	26-08-03 17:40:56	26-08-03 17:41:27	00:00:31	45
Decreasing	26-08-03 17:41:27	26-08-03 17:41:27	00:00:00	0 ← 45

Request Metrics

Method	Name	# Requests	# Fails	Average (ms)	Min (ms)	Max (ms)	RPS	Failures/s
DELETE	DELETE /v3/auth/tokens	71	0	333.94	43	417	2.37	0.00
DELETE	DELETE /v3/projects/:id (teardown)	1	0	145.00	145	145	0.03	0.00
DELETE	DELETE /v3/roles/:id (teardown)	1	0	80.00	80	80	0.03	0.00
DELETE	DELETE /v3/users/:id (teardown)	2	0	95.00	35	155	0.07	0.00
GET		992	0	357.64	183	528	33.07	0.00
GET	GET /v3/auth/tokens (validate new)	71	0	340.77	211	424	2.37	0.00
GET	GET /v3/auth/tokens (validate revoked)	72	0	310.62	10	389	2.40	0.00
GET	GET /v3/projects/:id	91	0	338.51	267	428	3.03	0.00
GET	GET /v3/projects/:id (catalog)	135	0	341.19	226	428	4.50	0.00
GET	GET /v3/role_assignments	71	0	328.17	243	404	2.37	0.00
GET	GET /v3/role_assignments?role.id	70	0	330.94	245	412	2.33	0.00
GET	GET /v3/role_assignments?scope.domain.id	71	0	321.99	230	406	2.37	0.00
GET	GET /v3/role_assignments?user.id	71	0	319.52	249	403	2.37	0.00
GET	GET /v3/roles	181	0	338.34	267	435	6.03	0.00
GET	GET /v3/roles/:id	91	0	335.91	224	432	3.03	0.00
GET	GET /v3/users/:id	174	0	354.28	170	465	5.80	0.00
GET	GET /v3/users/:id (catalog)	172	0	350.88	183	471	5.73	0.00
GET	GET /v4/api-keys	101	101	304.89	200	388	3.37	3.37
GET	GET /v4/mappings/rulesets	101	101	305.40	194	391	3.37	3.37
GET	GET /v4/oauth2/:domain_id/.well-known/openid-configuration	11572	11572	3.33	1	53	385.73	385.73
GET	GET /v4/oauth2/:domain_id/clients	100	100	305.74	189	374	3.33	3.33
GET	GET /v4/oauth2/:domain_id/jwks	11575	11575	3.34	1	49	385.83	385.83
GET	GET /v4/scim_realms	101	101	304.96	193	390	3.37	3.37
POST	POST /v3/auth/tokens	70	0	316.17	250	392	2.33	0.00
POST	POST /v3/auth/tokens (password, invalid)	29	0	1056.76	609	1470	0.97	0.00
POST	POST /v3/auth/tokens (password, project-scoped)	52	0	1331.71	711	1902	1.73	0.00
POST	POST /v3/auth/tokens (password, unscoped)	54	0	1035.15	395	1640	1.80	0.00
POST	POST /v3/auth/tokens (rescope to system)	160	0	384.23	300	473	5.33	0.00
POST	POST /v3/auth/tokens (rescope, invalid project)	96	0	319.08	246	424	3.20	0.00
POST	POST /v3/auth/tokens (system-scoped, bootstrap admin)	51	0	1218.96	787	1902	1.70	0.00
	Aggregated	26399	23550	50.78	1	1902	879.97	785.00

Response Time Metrics

Method	Name	50%ile (ms)	60%ile (ms)	70%ile (ms)	80%ile (ms)	90%ile (ms)	95%ile (ms)	99%ile (ms)	100%ile (ms)
DELETE	DELETE /v3/auth/tokens	340	340	360	370	380	400	410	417
DELETE	DELETE /v3/projects/:id (teardown)	145	145	145	145	145	145	145	145
DELETE	DELETE /v3/roles/:id (teardown)	80	80	80	80	80	80	80	80
DELETE	DELETE /v3/users/:id (teardown)	35	35	35	155	155	155	155	155
GET		360	370	380	390	410	430	470	500
GET	GET /v3/auth/tokens (validate new)	340	350	360	370	380	390	410	420
GET	GET /v3/auth/tokens (validate revoked)	320	330	330	350	360	370	380	389
GET	GET /v3/projects/:id	340	340	350	370	380	400	410	428
GET	GET /v3/projects/:id (catalog)	340	350	360	370	380	400	410	428
GET	GET /v3/role_assignments	330	340	350	360	380	390	400	400
GET	GET /v3/role_assignments?role.id	330	340	350	360	370	380	390	410
GET	GET /v3/role_assignments?scope.domain.id	320	330	350	350	370	370	380	406
GET	GET /v3/role_assignments?user.id	320	330	330	350	360	370	380	400
GET	GET /v3/roles	340	340	360	370	380	390	420	435
GET	GET /v3/roles/:id	340	350	350	360	380	390	430	430
GET	GET /v3/users/:id	350	360	370	380	400	410	440	465
GET	GET /v3/users/:id (catalog)	350	360	370	380	390	400	440	470
GET	GET /v4/api-keys	310	320	320	330	350	350	370	388
GET	GET /v4/mappings/rulesets	310	310	320	330	340	360	370	390
GET	GET /v4/oauth2/:domain_id/.well-known/openid-configuration	2	3	4	5	7	9	15	53
GET	GET /v4/oauth2/:domain_id/clients	300	310	320	330	360	370	370	370
GET	GET /v4/oauth2/:domain_id/jwks	2	3	4	5	7	9	15	49
GET	GET /v4/scim_realms	300	310	320	330	350	360	390	390
POST	POST /v3/auth/tokens	310	330	330	340	360	380	390	390
POST	POST /v3/auth/tokens (password, invalid)	1,000	1,000	1,000	1,000	1,000	1,000	1,000	1,000
POST	POST /v3/auth/tokens (password, project-scoped)	1,000	1,000	1,000	1,902	1,902	1,902	1,902	1,902
POST	POST /v3/auth/tokens (password, unscoped)	1,000	1,000	1,000	1,000	1,000	1,000	1,000	1,640
POST	POST /v3/auth/tokens (rescope to system)	380	390	400	410	430	450	470	470
POST	POST /v3/auth/tokens (rescope, invalid project)	320	320	330	350	360	390	420	420
POST	POST /v3/auth/tokens (system-scoped, bootstrap admin)	1,000	1,000	1,000	1,000	1,902	1,902	1,902	1,902
	Aggregated	3	4	5	8	310	350	430	1,902

Status Code Metrics

Method	Name	Status Codes
DELETE	DELETE /v3/auth/tokens	71 [204]
DELETE	DELETE /v3/projects/:id (teardown)	1 [204]
DELETE	DELETE /v3/roles/:id (teardown)	1 [204]
DELETE	DELETE /v3/users/:id (teardown)	2 [204]
GET		992 [200]
GET	GET /v3/auth/tokens (validate new)	71 [200]
GET	GET /v3/auth/tokens (validate revoked)	72 [404]
GET	GET /v3/projects/:id	91 [200]
GET	GET /v3/projects/:id (catalog)	135 [200]
GET	GET /v3/role_assignments	71 [200]
GET	GET /v3/role_assignments?role.id	70 [200]
GET	GET /v3/role_assignments?scope.domain.id	71 [200]
GET	GET /v3/role_assignments?user.id	71 [200]
GET	GET /v3/roles	181 [200]
GET	GET /v3/roles/:id	91 [200]
GET	GET /v3/users/:id	174 [200]
GET	GET /v3/users/:id (catalog)	172 [200]
GET	GET /v4/api-keys	101 [500]
GET	GET /v4/mappings/rulesets	101 [500]
GET	GET /v4/oauth2/:domain_id/.well-known/openid-configuration	11,572 [500]
GET	GET /v4/oauth2/:domain_id/clients	100 [500]
GET	GET /v4/oauth2/:domain_id/jwks	11,575 [500]
GET	GET /v4/scim_realms	101 [500]
POST	POST /v3/auth/tokens	70 [201]
POST	POST /v3/auth/tokens (password, invalid)	29 [401]
POST	POST /v3/auth/tokens (password, project-scoped)	52 [201]
POST	POST /v3/auth/tokens (password, unscoped)	54 [201]
POST	POST /v3/auth/tokens (rescope to system)	160 [201]
POST	POST /v3/auth/tokens (rescope, invalid project)	96 [401]
POST	POST /v3/auth/tokens (system-scoped, bootstrap admin)	51 [201]
	Aggregated	125 [401], 72 [404], 2,190 [200], 387 [201], 75 [204], 23,550 [500]

Transaction Metrics

Transaction	# Times Run	# Fails	Average (ms)	Min (ms)	Max (ms)	RPS	Failures/s
ReadHeavy
0.0	0	0	0.00	0	0	0.00	0.00
0.1	177	0	361.11	203	442	5.90	0.00
0.2	177	0	338.94	263	435	5.90	0.00
0.3	176	0	342.34	267	467	5.87	0.00
TokenLifecycle
1.0	0	0	0.00	0	0	0.00	0.00
1.1	72	0	1302.17	594	1570	2.40	0.00
ValidateToken
2.0	0	0	0.00	0	0	0.00	0.00
2.1	153	0	404.50	202	528	5.10	0.00
UserCRUD
3.0	0	0	0.00	0	0	0.00	0.00
3.1	0	0	0.00	0	0	0.00	0.00
3.2	174	0	354.49	170	466	5.80	0.00
3.3	2	0	95.00	35	155	0.07	0.00
ProjectCRUD
4.0	0	0	0.00	0	0	0.00	0.00
4.1	0	0	0.00	0	0	0.00	0.00
4.2	91	0	338.58	267	428	3.03	0.00
4.3	1	0	145.00	145	145	0.03	0.00
RoleCRUD
5.0	0	0	0.00	0	0	0.00	0.00
5.1	0	0	0.00	0	0	0.00	0.00
5.2	91	0	335.93	224	432	3.03	0.00
5.3	1	0	80.00	80	80	0.03	0.00
RoleList
6.0	0	0	0.00	0	0	0.00	0.00
6.1	181	0	338.42	267	435	6.03	0.00
RoleAssignmentList
7.0	0	0	0.00	0	0	0.00	0.00
7.1	71	0	328.25	243	404	2.37	0.00
7.2	71	0	319.59	249	403	2.37	0.00
7.3	71	0	322.04	230	406	2.37	0.00
7.4	70	0	330.96	245	412	2.33	0.00
UserRead
8.0	0	0	0.00	0	0	0.00	0.00
8.1	172	0	364.69	183	440	5.73	0.00
8.2	172	0	351.01	184	471	5.73	0.00
ProjectRead
9.0	0	0	0.00	0	0	0.00	0.00
9.1	137	0	336.78	202	429	4.57	0.00
9.2	135	0	341.28	226	428	4.50	0.00
PasswordAuth
10.0	54	0	1035.15	395	1640	1.80	0.00
10.1	52	0	1331.94	711	1902	1.73	0.00
PasswordAuthNegative
11.0	29	0	1056.97	609	1470	0.97	0.00
Rescope
12.0	0	0	0.00	0	0	0.00	0.00
12.1	160	0	384.40	300	473	5.33	0.00
RescopeNegative
13.0	0	0	0.00	0	0	0.00	0.00
13.1	96	0	319.18	247	425	3.20	0.00
SystemScopeAuth
14.0	51	0	1219.00	787	1902	1.70	0.00
ApiKeyCRUD
15.0	0	0	0.00	0	0	0.00	0.00
15.1	0	0	0.00	0	0	0.00	0.00
15.2	100	0	0.00	0	0	3.33	0.00
15.3	101	0	304.93	200	388	3.37	0.00
15.4	1	0	0.00	0	0	0.03	0.00
MappingCRUD
16.0	0	0	0.00	0	0	0.00	0.00
16.1	0	0	0.00	0	0	0.00	0.00
16.2	100	0	0.00	0	0	3.33	0.00
16.3	101	0	305.61	194	391	3.37	0.00
16.4	1	0	0.00	0	0	0.03	0.00
OAuth2ClientCRUD
17.0	0	0	0.00	0	0	0.00	0.00
17.1	0	0	0.00	0	0	0.00	0.00
17.2	99	0	0.00	0	0	3.30	0.00
17.3	100	0	305.76	189	374	3.33	0.00
17.4	1	0	0.00	0	0	0.03	0.00
OAuth2Discovery
18.0	11575	0	3.45	3	49	385.83	0.00
18.1	11572	0	3.46	2	53	385.73	0.00
ScimRealmRead
19.0	0	0	0.00	0	0	0.00	0.00
19.1	100	0	0.00	0	0	3.33	0.00
19.2	101	0	304.99	193	390	3.37	0.00
19.3	100	0	0.00	0	0	3.33	0.00
ScimSync
20.0	0	0	0.00	0	0	0.00	0.00
20.1	0	0	0.00	0	0	0.00	0.00
20.2	0	0	0.00	0	0	0.00	0.00
20.3	6194809	0	0.00	0	46	206493.64	0.00
20.4	6194809	0	0.00	4	34	206493.64	0.00
20.5	6194809	0	0.00	0	24	206493.64	0.00
20.6	1	0	0.00	0	0	0.03	0.00
Aggregated	18611117	0	0.07	2	1902	620370.56	0.00

Scenario Metrics

Transaction	# Users	# Times Run	Average (ms)	Min (ms)	Max (ms)	Scenarios/s	Iterations
ReadHeavy	6	174	1043.82	907	1223	5.80	29.00
TokenLifecycle	3	69	1321.32	1159	1570	2.30	23.00
ValidateToken	2	151	405.82	327	528	5.03	75.50
UserCRUD	2	172	355.53	282	466	5.73	86.00
ProjectCRUD	1	90	338.83	267	428	3.00	90.00
RoleCRUD	1	90	337.19	253	432	3.00	90.00
RoleList	2	179	338.74	267	435	5.97	89.50
RoleAssignmentList	3	70	1303.70	1149	1523	2.33	23.33
UserRead	4	169	718.16	572	869	5.63	42.25
ProjectRead	3	134	680.77	552	815	4.47	44.67
PasswordAuth	4	52	2392.71	1579	3205	1.73	13.00
PasswordAuthNegative	1	28	1064.18	609	1470	0.93	28.00
Rescope	2	158	384.41	300	473	5.27	79.00
RescopeNegative	1	95	319.80	247	425	3.17	95.00
SystemScopeAuth	2	49	1229.61	787	1902	1.63	24.50
ApiKeyCRUD	1	100	305.98	237	388	3.33	100.00
MappingCRUD	1	100	306.73	236	391	3.33	100.00
OAuth2ClientCRUD	1	99	306.94	233	374	3.30	99.00
OAuth2Discovery	3	11572	7.41	1	82	385.73	3857.33
ScimRealmRead	1	100	306.11	248	390	3.33	100.00
ScimSync	1	6194809	0.00	0	46	206493.64	6194809.00
Aggregated	45	6208460	0.22	1	3205	206948.67	6199898.00

Error Metrics

Method	Name	#	Error
POST	POST /v3/auth/tokens (password, invalid)	80	401 Unauthorized: POST /v3/auth/tokens (password, invalid)
POST	POST /v3/auth/tokens (rescope, invalid project)	298	401 Unauthorized: POST /v3/auth/tokens (rescope, invalid project)
GET	GET /v3/auth/tokens (validate revoked)	304	404 Not Found: GET /v3/auth/tokens (validate revoked)
GET	GET /v4/api-keys	163	500 Internal Server Error: GET /v4/api-keys
GET	GET /v4/mappings/rulesets	167	500 Internal Server Error: GET /v4/mappings/rulesets
GET	GET /v4/oauth2/:domain_id/.well-known/openid-configuration	25295	500 Internal Server Error: GET /v4/oauth2/:domain_id/.well-known/openid-configuration
GET	GET /v4/oauth2/:domain_id/clients	171	500 Internal Server Error: GET /v4/oauth2/:domain_id/clients
GET	GET /v4/oauth2/:domain_id/jwks	25298	500 Internal Server Error: GET /v4/oauth2/:domain_id/jwks
GET	GET /v4/scim_realms	188	500 Internal Server Error: GET /v4/scim_realms
POST	POST /v4/api-keys (setup)	1	500 Internal Server Error: POST /v4/api-keys (setup)
POST	POST /v4/mappings/rulesets (setup)	1	500 Internal Server Error: POST /v4/mappings/rulesets (setup)
POST	POST /v4/oauth2/:domain_id/clients (setup)	1	500 Internal Server Error: POST /v4/oauth2/:domain_id/clients (setup)

View full report

github-actions · 2026-07-31T18:22:29Z

🧪 Tempest Identity Compatibility Results (advisory, non-blocking)

rust

Totals
======
Ran: 133 tests in 34.4180 sec.
 - Passed: 63
 - Skipped: 4
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 66
Sum of execute time for each test: 19.9093 sec.

Failed test IDs

===== FAILED TESTS (target=rust) =====
tempest.api.identity.admin.v3.test_application_credentials.ApplicationCredentialsV3AdminTest.test_create_application_credential_with_roles[id-3b3dd48f-3388-406a-a9e6-4d078a552d0e]
tempest.api.identity.admin.v3.test_domains_negative.DomainsNegativeTestJSON.test_create_domain_with_name_length_over_64[id-37b1bbf2-d664-4785-9a11-333438586eae,negative]
tempest.api.identity.admin.v3.test_domains_negative.DomainsNegativeTestJSON.test_delete_active_domain[gate,id-1f3fbff5-4e44-400d-9ca1-d953f05f609b,negative]
tempest.api.identity.admin.v3.test_domains.DomainsTestJSON.test_create_domain_without_description[id-2abf8764-309a-4fa9-bc58-201b799817ad]
tempest.api.identity.admin.v3.test_domains.DomainsTestJSON.test_create_update_delete_domain[id-f2f5b44a-82e8-4dad-8084-0661ea3b18cf,smoke]
tempest.api.identity.admin.v3.test_default_project_id.TestDefaultProjectId.test_default_project_id[id-d6110661-6a71-49a7-a453-b5e26640ff6d]
tempest.api.identity.admin.v3.test_domains.DomainsTestJSON.test_domain_delete_cascades_content[id-d8d318b7-d1b3-4c37-94c5-3c5ba0b121ea]
tempest.api.identity.admin.v3.test_domains.DomainsTestJSON.test_list_domains_filter_by_enabled[id-3fd19840-65c1-43f8-b48c-51bdd066dff9]
tempest.api.identity.admin.v3.test_credentials.CredentialsTestJSON.test_credentials_create_get_update_delete[id-7cd59bf9-bda4-4c72-9467-d21cab278355,smoke]
setUpClass (tempest.api.identity.admin.v3.test_endpoint_groups.EndPointGroupsTest)
tempest.api.identity.admin.v3.test_list_projects.ListProjectsStaticTestJSON.test_list_projects[id-1d830662-22ad-427c-8c3e-4ec854b0af44]
tempest.api.identity.admin.v3.test_domain_configuration.DomainConfigurationTestJSON.test_create_domain_config_and_show_config_groups_and_options[id-9e3ff13c-f597-4f01-9377-d6c06c2a1477]
tempest.api.identity.admin.v3.test_domain_configuration.DomainConfigurationTestJSON.test_create_update_and_delete_domain_config[id-7161023e-5dd0-4612-9da0-1bac6ac30b63]
tempest.api.identity.admin.v3.test_domain_configuration.DomainConfigurationTestJSON.test_create_update_and_delete_domain_config_groups_and_opts[id-c7510fa2-6661-4170-9c6b-4783a80651e9]
tempest.api.identity.admin.v3.test_domain_configuration.DomainConfigurationTestJSON.test_show_default_group_config_and_options[id-11a02bf0-6f94-4380-b3b0-c8dc18fc0d22]
tempest.api.identity.admin.v3.test_list_projects.ListProjectsTestJSON.test_list_projects_with_enabled[id-0fe7a334-675a-4509-b00e-1c4b95d5dae8]
tempest.api.identity.admin.v3.test_list_projects.ListProjectsTestJSON.test_list_projects_with_parent[id-6edc66f5-2941-4a17-9526-4073311c1fac]
tempest.api.identity.admin.v3.test_groups.GroupsV3TestJSON.test_group_users_add_list_delete[id-1598521a-2f36-4606-8df9-30772bd51339,smoke]
tempest.api.identity.admin.v3.test_inherits.InheritsV3TestJSON.test_inherit_assign_check_revoke_roles_on_projects_group[id-26021436-d5a4-4256-943c-ded01e0d4b45]
tempest.api.identity.admin.v3.test_inherits.InheritsV3TestJSON.test_inherit_assign_check_revoke_roles_on_projects_user[id-18b70e45-7687-4b72-8277-b8f1a47d7591]
tempest.api.identity.admin.v3.test_inherits.InheritsV3TestJSON.test_inherit_assign_list_check_revoke_roles_on_domains_group[id-c7a8dda2-be50-4fb4-9a9c-e830771078b1]
tempest.api.identity.admin.v3.test_inherits.InheritsV3TestJSON.test_inherit_assign_list_check_revoke_roles_on_domains_user[id-4e6f0366-97c8-423c-b2be-41eae6ac91c8]
tempest.api.identity.admin.v3.test_inherits.InheritsV3TestJSON.test_inherit_assign_list_revoke_user_roles_on_domain[id-3acf666e-5354-42ac-8e17-8b68893bcd36]
tempest.api.identity.admin.v3.test_inherits.InheritsV3TestJSON.test_inherit_assign_list_revoke_user_roles_on_project_tree[id-9f02ccd9-9b57-46b4-8f77-dd5a736f3a06]
tempest.api.identity.admin.v3.test_groups.GroupsV3TestJSON.test_list_user_groups[id-64573281-d26a-4a52-b899-503cb0f4e4ec]
tempest.api.identity.admin.v3.test_users_negative.UsersNegativeTest.test_create_user_for_non_existent_domain[id-e75f006c-89cc-477b-874d-588e4eab4b17,negative]
tempest.api.identity.admin.v3.test_oauth_consumers.OAUTHConsumersV3Test.test_create_and_show_consumer[id-c8307ea6-a86c-47fd-ae7b-5b3b2caca76d]
tempest.api.identity.admin.v3.test_oauth_consumers.OAUTHConsumersV3Test.test_delete_consumer[id-fdfa1b7f-2a31-4354-b2c7-f6ae20554f93]
tempest.api.identity.admin.v3.test_oauth_consumers.OAUTHConsumersV3Test.test_list_consumers[id-09ca50de-78f2-4ffb-ac71-f2254036b2b8]
tempest.api.identity.admin.v3.test_oauth_consumers.OAUTHConsumersV3Test.test_update_consumer[id-080a9b1a-c009-47c0-9979-5305bf72e3dc]
tempest.api.identity.admin.v3.test_list_users.UsersV3TestJSON.test_list_users_with_not_enabled[id-bff8bf2f-9408-4ef5-b63a-753c8c2124eb]
tempest.api.identity.admin.v3.test_policies.PoliciesTestJSON.test_create_update_delete_policy[id-e544703a-2f03-4cf2-9b0f-350782fdb0d3,smoke]
tempest.api.identity.admin.v3.test_policies.PoliciesTestJSON.test_list_policies[id-1a0ad286-2d06-4123-ab0d-728893a76201]
tempest.api.identity.v3.test_projects.IdentityV3ProjectsTest.test_list_projects_returns_only_authorized_projects[id-86128d46-e170-4644-866a-cc487f699e1d]
tempest.api.identity.admin.v3.test_project_tags.IdentityV3ProjectTagsTest.test_list_update_delete_project_tags[id-7c123aac-999d-416a-a0fb-84b915ab10de]
tempest.api.identity.admin.v3.test_projects_negative.ProjectsNegativeStaticTestJSON.test_create_projects_name_length_over_64[id-502b6ceb-b0c8-4422-bf53-f08fdb21e2f0,negative]
tempest.api.identity.v3.test_tokens.TokensV3Test.test_validate_token[id-a9512ac3-3909-48a4-b395-11f438e16260]
tempest.api.identity.v3.test_api_discovery.TestApiDiscovery.test_api_media_types[id-657c1970-4722-4189-8831-7325f3bc4265,smoke]
tempest.api.identity.v3.test_api_discovery.TestApiDiscovery.test_api_version_resources[id-b9232f5e-d9e5-4d97-b96c-28d3db4de1bd,smoke]
tempest.api.identity.v3.test_api_discovery.TestApiDiscovery.test_identity_v3_existence[id-79aec9ae-710f-4c54-a4fc-3aa25b4feac3]
tempest.api.identity.v3.test_api_discovery.TestApiDiscovery.test_list_api_versions[id-721f480f-35b6-46c7-846e-047e6acea0dc,smoke]
tempest.api.identity.v3.test_catalog.IdentityCatalogTest.test_catalog_standardization[id-56b57ced-22b8-4127-9b8a-565dfb0207e2]
tempest.api.identity.admin.v3.test_projects.ProjectsTestJSON.test_create_is_domain_project[id-a7eb9416-6f9b-4dbb-b71b-7f73aaef59d5]
tempest.api.identity.admin.v3.test_roles.RolesV3TestJSON.test_assignments_for_domain_roles[id-3859df7e-5b78-4e4d-b10e-214c8953842a]
tempest.api.identity.admin.v3.test_projects.ProjectsTestJSON.test_project_create_with_parent[id-1854f9c0-70bc-4d11-a08a-1c789d339e3d]
tempest.api.identity.admin.v3.test_projects.ProjectsTestJSON.test_project_get_equals_list[id-d1db68b6-aebe-4fa0-b79d-d724d2e21162]
tempest.api.identity.admin.v3.test_roles.RolesV3TestJSON.test_assignments_for_implied_roles_create_delete[id-c8828027-df48-4021-95df-b65b92c7429e]
tempest.api.identity.admin.v3.test_roles.RolesV3TestJSON.test_grant_list_revoke_role_to_group_on_domain[id-4bf8a70b-e785-413a-ad53-9f91ce02faa7]
tempest.api.identity.admin.v3.test_roles.RolesV3TestJSON.test_grant_list_revoke_role_to_group_on_project[id-cbf11737-1904-4690-9613-97bcbb3df1c4]
tempest.api.identity.admin.v3.test_roles.RolesV3TestJSON.test_grant_list_revoke_role_to_group_on_system[id-c888fe4f-8018-48db-b959-542225c1b4b6]
tempest.api.identity.admin.v3.test_roles.RolesV3TestJSON.test_grant_list_revoke_role_to_user_on_domain[id-6c9a2940-3625-43a3-ac02-5dcec62ef3bd]
tempest.api.identity.admin.v3.test_roles.RolesV3TestJSON.test_implied_domain_roles[id-eb1e1c24-1bc4-4d47-9748-e127a1852c82]
tempest.api.identity.admin.v3.test_roles.RolesV3TestJSON.test_implied_roles_create_check_show_delete[id-c90c316c-d706-4728-bcba-eb1912081b69]
tempest.api.identity.admin.v3.test_roles.RolesV3TestJSON.test_role_create_update_show_list[id-18afc6c0-46cf-4911-824e-9989cc056c3a,smoke]
tempest.api.identity.admin.v3.test_tokens.TokensV3TestJSON.test_get_available_domain_scopes[id-ec5ecb05-af64-4c04-ac86-4d9f6f12f185]
tempest.api.identity.admin.v3.test_trusts.TrustsV3TestJSON.test_get_trusts_all[id-4773ebd5-ecbf-4255-b8d8-b63e6f72b65d,smoke]
tempest.api.identity.admin.v3.test_tokens.TokensV3TestJSON.test_rescope_token[id-565fa210-1da1-4563-999b-f7b5b67cf112]
tempest.api.identity.admin.v3.test_trusts.TrustsV3TestJSON.test_get_trusts_query[id-6268b345-87ca-47c0-9ce3-37792b43403a]
tempest.api.identity.v3.test_application_credentials.ApplicationCredentialsV3Test.test_create_application_credential[id-8080c75c-eddc-4786-941a-c2da7039ae61]
tempest.api.identity.v3.test_application_credentials.ApplicationCredentialsV3Test.test_create_application_credential_expires[id-852daf0c-42b5-4239-8466-d193d0543ed3]
tempest.api.identity.v3.test_application_credentials.ApplicationCredentialsV3Test.test_list_application_credentials[id-ff0cd457-6224-46e7-b79e-0ada4964a8a6]
tempest.api.identity.v3.test_application_credentials.ApplicationCredentialsV3Test.test_query_application_credentials[id-9bb5e5cc-5250-493a-8869-8b665f6aa5f6]
tempest.api.identity.admin.v3.test_trusts.TrustsV3TestJSON.test_trust_expire[id-0ed14b66-cefd-4b5c-a964-65759453e292]
tempest.api.identity.admin.v3.test_trusts.TrustsV3TestJSON.test_trust_impersonate[id-5a0a91a4-baef-4a14-baba-59bf4d7fcace]
tempest.api.identity.admin.v3.test_trusts.TrustsV3TestJSON.test_trust_noimpersonate[id-ed2a8779-a7ac-49dc-afd7-30f32f936ed2]
tempest.api.identity.admin.v3.test_users.UsersV3TestJSON.test_list_user_projects[id-a831e70c-e35b-430b-92ed-81ebbc5437b8]
===== END FAILED TESTS (target=rust) =====

View full run

github-actions · 2026-07-31T19:10:47Z

Bencher Report

Project	keystone
Branch	claude/loadtest-expansion-stress-u8i0j4
Testbed	ubuntu-latest

Click to view all benchmark results

Benchmark	Latency	Benchmark Result nanoseconds (ns) (Result Δ%)	Upper Boundary nanoseconds (ns) (Limit %)
Command_Serde/apply/remove	📈 view plot 🚷 view threshold	118,880.00 ns (-62.77%) Baseline: 319,272.06 ns	1,634,101.99 ns (7.27%)
Command_Serde/apply/set	📈 view plot 🚷 view threshold	124,040.00 ns (-64.49%) Baseline: 349,281.86 ns	1,679,671.23 ns (7.38%)
Command_Serde/pack/delete	📈 view plot 🚷 view threshold	123.94 ns (+4.40%) Baseline: 118.71 ns	157.53 ns (78.68%)
Command_Serde/pack/delete_index	📈 view plot 🚷 view threshold	113.57 ns (+8.01%) Baseline: 105.15 ns	139.12 ns (81.63%)
Command_Serde/pack/set	📈 view plot 🚷 view threshold	201.57 ns (+6.39%) Baseline: 189.47 ns	248.75 ns (81.03%)
Command_Serde/pack/set_index	📈 view plot 🚷 view threshold	109.56 ns (+4.18%) Baseline: 105.17 ns	139.18 ns (78.72%)
Command_Serde/unpack/delete	📈 view plot 🚷 view threshold	190.20 ns (+3.03%) Baseline: 184.61 ns	267.96 ns (70.98%)
Command_Serde/unpack/delete_index	📈 view plot 🚷 view threshold	175.28 ns (+13.12%) Baseline: 154.95 ns	218.78 ns (80.12%)
Command_Serde/unpack/set	📈 view plot 🚷 view threshold	288.31 ns (+12.36%) Baseline: 256.60 ns	349.57 ns (82.48%)
Command_Serde/unpack/set_index	📈 view plot 🚷 view threshold	159.99 ns (+5.66%) Baseline: 151.42 ns	212.26 ns (75.37%)
Payload_encryption/pack/remove_cmd	📈 view plot 🚷 view threshold	119.84 ns (+5.59%) Baseline: 113.50 ns	151.03 ns (79.35%)
Payload_encryption/pack/set_cmd	📈 view plot 🚷 view threshold	196.94 ns (+0.80%) Baseline: 195.39 ns	263.37 ns (74.78%)
Payload_encryption/unpack/remove_cmd	📈 view plot 🚷 view threshold	199.16 ns (+3.12%) Baseline: 193.13 ns	281.45 ns (70.76%)
Payload_encryption/unpack/set_cmd	📈 view plot 🚷 view threshold	314.84 ns (+16.56%) Baseline: 270.11 ns	374.42 ns (84.09%)
Raft_1Node_Latency/prefix/1node	📈 view plot 🚷 view threshold	3,427,200.00 ns (+27.68%) Baseline: 2,684,171.25 ns	6,171,231.98 ns (55.54%)
Raft_1Node_Latency/read/1node	📈 view plot 🚷 view threshold	42,163.00 ns (+10.75%) Baseline: 38,068.81 ns	53,319.60 ns (79.08%)
Raft_1Node_Latency/remove/1node	📈 view plot 🚷 view threshold	351,450.00 ns (-57.04%) Baseline: 818,054.53 ns	4,098,046.95 ns (8.58%)
Raft_1Node_Latency/write/1node	📈 view plot 🚷 view threshold	362,060.00 ns (-62.72%) Baseline: 971,300.16 ns	5,492,419.70 ns (6.59%)
build_snapshot/default	📈 view plot 🚷 view threshold	107,910.00 ns (-13.77%) Baseline: 125,140.23 ns	219,356.50 ns (49.19%)
fernet token/project	📈 view plot 🚷 view threshold	1,457.70 ns (+2.23%) Baseline: 1,425.89 ns	1,811.00 ns (80.49%)
get_data_keyspace	📈 view plot 🚷 view threshold	0.32 ns (+1.91%) Baseline: 0.32 ns	0.39 ns (82.85%)
get_db	📈 view plot 🚷 view threshold	0.32 ns (+1.56%) Baseline: 0.32 ns	0.39 ns (82.69%)
get_fernet_token_timestamp/project	📈 view plot 🚷 view threshold	162.96 ns (+16.03%) Baseline: 140.44 ns	179.00 ns (91.04%)
get_keyspace	📈 view plot 🚷 view threshold	4.32 ns (-29.96%) Baseline: 6.16 ns	14.58 ns (29.61%)

🐰 View full continuous benchmarking report in Bencher

Add ScimRealmRead and ScimSync loadtest scenarios covering raft-backed federation/SCIM (ADR 0021, 0024): realm CRUD reads/updates via admin token, and the real IdP-connector lifecycle over /SCIM/v2 with a per-VU API Key bearer (provision, reconcile, attribute drift, offboard). Seed setup is now idempotent against a pre-existing SCIM realm/IdP (no delete endpoint exists, so an interrupted prior run leaves them behind; 409 now reuses them instead of aborting seed). While chasing intermittent 401s surfaced by ScimSync under load, found and fixed two real bugs in the raft-backed auth path: - Timing oracle in ApiKeyAuth: the IP-not-allowed rejection branch returned immediately while every other rejection branch (not-found, inactive, wrong-secret) burns an Argon2id hash first, letting an attacker distinguish a valid-but-IP-blocked lookup_hash from a nonexistent one by response latency (ADR 0021 Invariant 7). Now burns the same dummy-hash cost. - Two linearizability gaps in the ReadIndex forward-to-leader path (get_by_key/prefix/prefix_index): the leader-side gRPC handlers served forwarded reads without re-confirming current leadership, and follower-side callers silently fell back to an unguarded local read whenever the forward RPC failed. Both fixed. fix(scim): map raft revision conflict to 409 instead of 500 The RaftBackend only matched StoreError::Conflict to convert ApiStoreError::Conflict to ScimRealmProviderError::Conflict. However, the the Conflict arm never matched -- concurrent PUT /v4/scim_realms returned 500 Internal Server Error instead of 409 Conflict. Also restructure update_impl error handling to properly separate Conflict, NotFound, and raft storage errors. fix(storage): Fall back to local read when forward fails When a forwarded gRPC read fails (e.g., a removed node has stale leader cache pointing to an old leader), fall back to local read from the FjallDB instead of hard-failing. This allows removed nodes to serve their last committed state. Also add test_prefix_index_leader_race_concurrent regression test for GitHub #1135 to verify prefix_index works correctly under concurrent load on the leader. Signed-off-by: Artem Goncharov <artem.goncharov@gmail.com>

gtema force-pushed the claude/loadtest-expansion-stress-u8i0j4 branch 3 times, most recently from 10007b9 to b5d0c3d Compare August 3, 2026 14:20

gtema force-pushed the claude/loadtest-expansion-stress-u8i0j4 branch from b5d0c3d to 4acc50e Compare August 3, 2026 17:22

gtema added this pull request to the merge queue Aug 3, 2026

Merged via the queue into main with commit 58b82b9 Aug 3, 2026
36 checks passed

openstack-experimental-release-plz Bot mentioned this pull request Aug 3, 2026

chore: release #754

Open

Navigation Menu

Uh oh!

feat: Add SCIM scenarios; Fix findings - #1136

feat: Add SCIM scenarios; Fix findings#1136
gtema merged 1 commit into
mainfrom
claude/loadtest-expansion-stress-u8i0j4

gtema commented Jul 31, 2026

Uh oh!

github-actions Bot commented Jul 31, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 31, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 31, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

gtema commented Jul 31, 2026

Uh oh!

github-actions Bot commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Goose Attack Report

Plan Overview

Request Metrics

Response Time Metrics

Status Code Metrics

Transaction Metrics

Scenario Metrics

Error Metrics

Uh oh!

github-actions Bot commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bencher Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions Bot commented Jul 31, 2026 •

edited

Loading

github-actions Bot commented Jul 31, 2026 •

edited

Loading

github-actions Bot commented Jul 31, 2026 •

edited

Loading