SSO OIDC Activation

[rmsferreira]

Dear all.
I'm facing some problems when activate the SSO OIDC.
When this is activated, the admin user lost access to some capabilities like see users, investigations, etc...

Couldn't load the queue
The backend didn't return queue data. Try again, or check the API logs if this keeps happening.

API 403 — /api/v1/alerts/queue

The user list could not be loaded.

API 403 — /api/v1/admin/users?sort=created&order=desc&limit=20&offset=0

[rmsferreira]

026-10-30T20:33:18.711Z [aisoc-api] events_of_interest ClickHouse fallback: clickhouse query failed: Code: 47.
2026-10-30T20:33:18.711Z [aisoc-api] DB::Exception: Missing columns: 'ingested_at' while processing query: 'SELECT count() AS cnt FROM aisoc.raw_events WHERE ((ingested_at >= toDateTime('2026-10-30 19:33:18')) AND (ingested_at < toDateTime('2026-10-30 20:33:18'))) AND (tenant_id = '00000000-0000-0000-0000-000000000001') LIMIT 1', required columns: 'ingested_at' 'tenant_id', maybe you meant: 'tenant_id'. Stack trace:
2026-10-30T20:33:18.711Z [aisoc-api] 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c6251d7 in /usr/bin/clickhouse
2026-10-30T20:33:18.711Z [aisoc-api] 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000715b6b1 in /usr/bin/clickhouse
2026-10-30T20:33:18.711Z [aisoc-api] 2. DB::TreeRewriterResult::collectUsedColumns(std::shared_ptrDB::IAST const&, bool, bool) @ 0x00000000122245f9 in /usr/bin/clickhouse
2026-10-30T20:33:18.711Z [aisoc-api] 3. DB::TreeRewriter::analyzeSelect(std::shared_ptrDB::IAST&, DB::TreeRewriterResult&&, DB::SelectQueryOptions const&, std::vector<DB::TableWithColumnNamesAndTypes, std::allocatorDB::TableWithColumnNamesAndTypes> const&, std::vector<String, std::allocator> const&, std::shared_ptrDB::TableJoin) const @ 0x0000000012229261 in /usr/bin/clickhouse
2026-10-30T20:33:18.711Z [aisoc-api] 4. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptrDB::IAST const&, std::shared_ptrDB::Context const&, std::optionalDB::Pipe, std::shared_ptrDB::IStorage const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptrDB::PreparedSets)::$_0::operator()(bool) const @ 0x0000000011eda5bc in /usr/bin/clickhouse
2026-10-30T20:33:18.711Z [aisoc-api] 5. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptrDB::IAST const&, std::shared_ptrDB::Context const&, std::optionalDB::Pipe, std::shared_ptrDB::IStorage const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptrDB::PreparedSets) @ 0x0000000011ece615 in /usr/bin/clickhouse
2026-10-30T20:33:18.711Z [aisoc-api] 6. DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptrDB::IAST const&, std::shared_ptrDB::Context, DB::SelectQueryOptions const&, std::vector<String, std::allocator> const&) @ 0x0000000011f7e3a8 in /usr/bin/clickhouse
2026-10-30T20:33:18.711Z [aisoc-api] 7. DB::InterpreterFactory::get(std::shared_ptrDB::IAST&, std::shared_ptrDB::Context, DB::SelectQueryOptions const&) @ 0x0000000011e845be in /usr/bin/clickhouse
2026-10-30T20:33:18.711Z [aisoc-api] 8. DB::executeQueryImpl(char const*, char const*, std::shared_ptrDB::Context, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000122c902a in /usr/bin/clickhouse
2026-10-30T20:33:18.711Z [aisoc-api] 9. DB::executeQuery(String const&, std::shared_ptrDB::Context, bool, DB::QueryProcessingStage::Enum) @ 0x00000000122c5015 in /usr/bin/clickhouse
2026-10-30T20:33:18.711Z [aisoc-api] 10. DB::TCPHandler::runImpl() @ 0x0000000013140ff9 in /usr/bin/clickhouse
2026-10-30T20:33:18.711Z [aisoc-api] 11. DB::TCPHandler::run() @ 0x00000000131533d9 in /usr/bin/clickhouse
2026-10-30T20:33:18.711Z [aisoc-api] 12. Poco::Net::TCPServerConnection::start() @ 0x0000000015b4dc74 in /usr/bin/clickhouse
2026-10-30T20:33:18.711Z [aisoc-api] 13. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b4ee71 in /usr/bin/clickhouse
2026-10-30T20:33:18.711Z [aisoc-api] 14. Poco::PooledThread::run() @ 0x0000000015c85aa7 in /usr/bin/clickhouse
2026-10-30T20:33:18.711Z [aisoc-api] 15. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c83d7c in /usr/bin/clickhouse
2026-10-30T20:33:18.711Z [aisoc-api] 16. ? @ 0x00007bebdb2f9609 in ?
2026-10-30T20:33:18.711Z [aisoc-api] 17. ? @ 0x00007bebdb21e353 in ?
2026-10-30T20:33:18.829Z [aisoc-api] events_of_interest ClickHouse fallback: clickhouse query failed: Simultaneous queries on single connection detected
2026-10-30T20:33:18.829Z [aisoc-api] events_of_interest ClickHouse fallback: clickhouse query failed: [Errno 9] Bad file descriptor

[beenuar]

Thanks — you found two real bugs, and both are fixed in #1569.

1. The admin lockout (the 403s). You're right that activating SSO is what did it, and nothing about the admin account changed — a different user signing in triggered it. The mechanism: permissions resolve from the static role map only while a tenant has zero role grants. Your bootstrap admin was created with role = admin but no row in the grant table (it's created after the seed that would have backfilled one). That's harmless until the first SSO user signs in — provisioning grants that user a role, which flips the whole tenant to database-backed RBAC, and from that moment your admin (with no grant row) resolves to zero permissions. Hence 403 on users, investigations, the alert queue, everything at once.

The fix makes the bootstrap admin always hold its admin grant, so it can never be the ungranted account when the switch flips. It runs on every startup, so a deployment already in this state heals itself on the next restart — once you're on the fixed build, restart the API and the admin's access returns with no manual step. If you need to unblock before upgrading, granting the admin any role through the console's user-management screen (from another admin, or via the API) also restores it, because then it has a grant row.

2. The ClickHouse Missing columns: 'ingested_at'. Also real and also fixed. The funnel's "events of interest" count queried a column name that doesn't exist in the event lake (the table has event_time and ingest_time), so that query failed on every call and the console silently fell back to a Postgres-derived estimate. It now queries the correct column. You'd have seen the funnel's top number working but sourced from the fallback rather than the lake; after the fix it reads the lake directly.

Both fixes are verified against a real Postgres with the full migration chain (the lockout is reproduced in a test and then shown fixed). I'll follow up here once the PR merges and the images publish.

[beenuar]

Merged — #1569 is on main (013ecd38), all required checks green.

To pick it up: docker compose pull && docker compose up -d once the :latest images publish from this merge (within the hour), or wait for the next tagged release, which will carry both fixes in its changelog. On the fixed build, the admin's access is restored automatically on the next API start — no manual step.

[beenuar]

This is now released in v20.4.0 — both the SSO admin-lockout and the funnel ClickHouse error are fixed in it.

To upgrade: docker compose pull && docker compose up -d (the v20.4.0 images and the 7.13.0 Helm chart are published and verified live), or pin to v20.4.0 explicitly. On the fixed build your admin's access returns automatically on the next API start — no manual step, because the grant is re-applied on every startup.

Thanks again for the detailed report, including the API logs — the ClickHouse line is what let us catch the second bug alongside the one you hit first.