Streamlining the developer experience with a resource centric api

[hifabienne]

Why this is a priority

We're focused on empowering our customers and community to seamlessly integrate Zitadel with their existing tools and workflows. We recognize that the current developer experience, particularly when interacting with our APIs, presents unnecessary friction. By prioritizing this improvement, we aim to significantly reduce complexity, accelerate integration times, and foster a more intuitive and enjoyable development journey. This directly translates to faster adoption, increased customer satisfaction, and a more vibrant and engaged community.

Current Pain Points

Currently, Zitadel's API landscape is structured around use cases. This means that depending on the role of the requesting user (e.g., authenticated user, instance administrator, organization administrator), different APIs need to be utilized to manage the same underlying resources. This approach has led to several challenges:

• Confusion and Complexity: Developers often struggle to determine the correct API endpoint based on the user context, leading to wasted time and potential errors.
• Inconsistent Experience: The need to navigate different API structures for similar actions creates an inconsistent and less intuitive development experience. Customers often "don't care" about the requesting user's role but simply want to manage a specific resource.
• Naming Inconsistencies: Over time, naming conventions across our APIs, UI, and documentation have diverged, adding to the cognitive load for developers.
• Documentation Discrepancies: Ensuring our documentation accurately reflects the nuances of the use-case-oriented APIs has proven challenging, leading to frustration and integration roadblocks.

A Shift to a Resource-Based API Approach

To address these pain points, we are embarking on a significant refactoring of our APIs to adopt a resource-based approach. This means organizing our APIs around core resources like "Users," "Organizations," "Applications," etc. Regardless of the requesting user's role, developers will interact with consistent API endpoints to manage these resources. Permissions will then dictate the level of access and information available based on the authenticated user's context.

Key Improvements We Will Deliver:

• Intuitive and Consistent API Structure: Developers will benefit from a logical and predictable API design, making it easier to discover and utilize the functionalities they need. For example, managing users will be consistently done through a dedicated /users API, regardless of whether the action is performed by an end-user or an administrator.
• Simplified Integration: The resource-based approach will streamline the integration process, reducing the learning curve and accelerating development timelines. Developers can focus on the resources they need to interact with, rather than navigating complex role-based API structures.
• Enhanced Naming Conventions: We will meticulously review and standardize naming conventions across our APIs, UI, and documentation to ensure clarity and consistency. This will make it easier for everyone to understand and work with Zitadel.
• Up-to-Date and Accurate Documentation: Alongside the API refactoring, we are committed to ensuring our documentation is comprehensive, accurate, and directly aligned with the new resource-based API structure. This will include clear examples, use cases, and consistent terminology.

Acceptance Criteria

•  All resources are created in the resource api
• All API requests are tested with all available api protocols (e.g grpc, rest, etc)
• All APIs are implemented according to the naming defined in Inconsistent Naming #5888
• All API request are documented clearly and correct with examples
• All old APIs are marked as deprecated
• Errors are handled consistently across all requests and return same error code
• Error codes are documented
• SDKs are able to handle new resource API
• The implementation follows the API guidelines

Migration Path

• New APIs will be released with Zitadel V4 in a GA version
• Customers are able to switch over at their convinence
• Console will already be switched to new APIs
• In a future release the deprecated APIs will be removed (not defined yet, probably around Zitadel V6 & 7)

Technical Considerations

Resources

• Each resource will get its own service. e.g User Service, Project Service, Organization Service, etc.
• Based on the permission an administrator has, they will be able to perform the requests, and get the result they are allowed to see. Example: As an instance administrator, I will receive all users on the search users endpoint, as an organization administrator i will only recieve the users from the organization I am administrator.
• Endpoints will be clustered to their main resource, there will not be separat endpoints for each of the sub types. Example: ListUsers instead of ListHumanUsers/ListMachineUsers, AddUser instead of AddHumanUser/AddMachineUser

Settings and Policies

When designing our new resource-based API for Zitadel, we differentiate between policies and settings to ensure clear separation of concerns:

• Policies define the rules governing specific checks within Zitadel. These are the "how" of enforcement. They establish constraints and behaviors, such as password complexity requirements. We aim to support both predefined policies and user-defined policies, enabling flexible and adaptable security postures.
• Settings, on the other hand, are resource-specific configurations. They represent the "what" and "where" of resource behavior, providing the necessary parameters for their operation. These settings are used for configuration purposes, and are not rules for checks.

How current settings and policies will be structured

Policies

We need to be able to define policies in different contexts, e.g hirarchies (systems, instance, organization)

Type	Description	Tags	content
Force Local MFA	Enforce Multi-Factor for local accounts	auth, mfa
Force External MFA	Enforce Multi-Factor for external accounts	auth, mfa
Password Complexity	Set the minimum length of a password and Set characters that should be included in a password	auth, password	number
Password Expiration	Set when a password needs to be changed after a set period	auth, password	time
Max Password attempts	This policy enforces an account lockout after [number] consecutive failed password attempts.	auth, password, lockout	number
Max OTP attempts	This policy enforces an account lockout after [number] consecutive failed otp attempts.	auth, otp, lockout	number
Allowed Domains	Only users with the given domain are allowed to register and authenticate and invited.	auth, registration, login, domain	enabled/disable
Organization Domain Verification	This policy mandates domain verification via DNS or HTTP challenge upon the addition of any new domain to the organization.	domain, verification, organization	enabled/disable

Settings

We need to be able to define settings in different contexts, e.g hirarchies (systems, instance, organization, app, etc.)

• Features
• Notification Channels
  • Email (SMTP, HTTP)
  • SMS (Twilio, HTTP)
• Messages (enabled/disable, texts, templates?)
  • Password Changed (enable / disable)
  • Password Expiration Warning (enable / disable)
  • Domain Claimed
  • Initialization
    • Code Configuration (Secret Generator)
  • Invite User
    • Code Configuration (Secret Generator)
  • Password Change
    • Code Configuration (Secret Generator)
  • Password Reset
    • Code Configuration (Secret Generator)
  • Passwordless
    • Code Configuration (Secret Generator)
  • Verify Email
    • Email Code Configuration (Secret Generator)
  • Verify Email OTP
    • Code Configuration (Secret Generator)
  • Verify Phone
    • Code Configuration (Secret Generator)
  • Verify SMS OTP
    • Code Configuration (Secret Generator)
• AuthMethods
  • Password
  • Passkey (summerized with U2f)
  • TOTP
  • Email OTP
  • SMS OTP
  • External IDPs
    • Google
    • Microsoft
    • GitHub
    • GitHub Enterprise Server
    • Gitlab
    • Gitlab Self Hosted
    • Apple
    • Generic OIDC
    • Generic OAuth
    • Generic JWT
    • Generic SAML
• OIDC
  • Access Token Lifetime
  • ID Token Lifetime
  • Refresh Token Lifetime
  • IDLE Refresh Token Lifetime
  • App secret configuration (Secret Generator)
  • WebKeys
• Secret Generators --> Moved to Message Configurations
• Security
  • Allow iFrame
    • Allowed URLs
  • Allow Impersonation
• Branding
  • Logos
  • Icons
  • Colors
  • Font
  • Dark / Light Mode
• Languages
• Hosted Login
  • Self Register User (enabled/disable)
  • Self Register Organization (enable/disable) (only on instance level, not on org)
  • Forgot Password Flow (enable / disable)
  • Domain Discovery (enabled / disabled)
  • Username Discovery Protection (enabled / disable) --> opinion about this name? instead of ignore unknown usernames
  • Login with Email Address (enable/ disable) (default enabled)
  • Login with Phone Number (enable / disable) (default enabled)
  • Default Redirect URL
  • Terms and Services (enabled/disabled --> set link)
  • Privacy Policy (enabled/disabled --> set link)
  • Help Link (enabled/disabled --> set link)
  • Support Email (enabled/disabled --> set email)
  • Login Lifetimes (Note: Maybe in the future policy about aal)
    • Password Check
    • External Login Check
    • Multifactor Prompt
    • Multi-Factor Check
  • Hide Watermark
  • Hide Loginname suffix
  • Custom Texts
• Management Console
  • Custom Link (enabled/disabled --> set link, text)
  • Documentation Link (enabled/disabled --> set link)

Notes

• Remove domain suffix settings
• What do we need to implement to get the activity logs and pricing for Zitadel Cloud?

[eliobischof]

• Actions first, will be breaking
• How to version new services?
• How to deprecate old services, for example v2alpha?
• Estimate User Schema effects