[api] feat: migrate to Clerk authentication

[nednella]

Purpose

The Spring Boot API now acts as an OAuth2 Resource Server, with Clerk as the external Identity Provider (IdP).

Any HTTP request to a protected endpoint must include a valid JWT in the Authorization: Bearer <token> header.

The Onboarding Flow

Before API access is granted, users must call POST /api/v1/clerk/onboarding with a valid JWT issued by Clerk.

This action is required as Clerk does not provide a synchronous method of interacting directly with the user table. Sending this request guarantees that every Clerk user has a corresponding User and Profile in the local application, avoiding database sync issues.

If the onboarding process is not completed, the client router will block access to the application, and any authenticated request to a protected API endpoint will result in 403 Forbidden, instructing the user to first complete the onboarding process.

The flow itself is outlined below.

1. User signs up via Clerk

   • The client handles user registration using Clerk’s authentication components.
   • Clerk issues a session token with two custom claims:
     • onboarded: false
     • echo_id: null

2. Redirect to Onboarding

   • The client router forces users who do not have onboarded: true in their active session's claims to the /onboarding route.
   • All other application routes are inaccessible until onboarding completes.

3. Onboarding with the API

   • The client calls POST /api/v1/clerk/onboarding.
   • The API performs two critical steps:
     • Upsert a local user: Ensures a local User exists for the given Clerk user ID, found in the sub claim on the authenticated session token.
     • Updates Clerk user object: Updates the Clerk user for the given Clerk user ID by setting the user's external_id with the local UUID, and sets public_metadata.onboarded = true.

4. Session Refresh

   • The client forces a session refresh via Clerk to pull the latest user information into the relevant claims onboarded and echo_id.
   • The new session token contains the claims:
     • onboarded: true
     • echo_id: <UUID>

5. Access Granted

   • Once the onboarded claim is updated, the client router will redirect the user to /home, granting access.
   • Any following API requests will now contain claims that tie the authenticated Clerk user to a local User in db.

6. Webhooks

   • Post-registration, Clerk maintains db sync with the local application through webhooks for user-related events e.g. user.updated user.deleted. Any updates to the Clerk user, such as username and profile_image_url will be reflected in the local db too.

Changelog

GH Workflows

• Added API CI Pipeline job environment variables:
  • CLERK_ISSUER_URI
  • CLERK_JWK_SET_URI
  • CLERK_SECRET_KEY

Docker Compose

• Removed image redis

Environment

• Updated application*.yml files:
  • Migrated most static references to environment variables for easy environment configuration.
  • Each application-{profile}.yml file now refers to its own .env.{profile} config file for environment variables.
• Added .env.example file with templated instructions for configuring the required environment variables for each local environment.

Dependencies

• Removed spring-boot-starter-data-redis
• Added spring-boot-starter-oauth2-resource-server
• Added clerk:backend-api:3.1.0
• Added svix:1.9.0
• Added spring-boot-starter-webflux

DB Schema

• Renamed account table to user.
• Updated user table schema:
  • Removed username column - usernames are now handled within profiles.
  • Removed encrypted_password column - auth handled by the IdP.
  • Replaced enabled column with status column.
  • Added external_id column for linking external users provided by the IdP.
• Updated profile table schema:
  • Removed foreign key constraints fk_username fk_avatar_id fk_banner_id.
  • Replaced avatar_id and banner_id columns with image_url column.
  • username and image_url are now synchronised to Clerk user directly.
• Updated SQL function files to reflect changes made to user and profile tables.

Config

• Removed RegexConfig configuration class entirely.
• Removed SessionConfig configuration class entirely.
• Updated ConstantsConfig subclasses:
  • Removed Account subclass entirely.
• Updated ApiConfig subclasses:
  • Removed Auth subclass entirely.
  • Removed Account subclass entirely.
  • Added Clerk subclass with static endpoint constants:
    • ONBOARDING constant.
    • WEBHOOK constant.
  • Added Dev subclass with static endpoint constants.
    • PERSIST_ALL constant.
    • SYNC_ALL constant.
• Added ClerkConfig configuration class.
  • Provides application constants for referring to Clerk token claims & metadata.
  • Registers a Clerk bean for SDK interaction.
  • Registers a Svix Webhook bean for webhook validation.
• Added properties subpackage for environment variable validation on application startup:
  • Added AuthenticationProperties class to validate OAuth-related secrets are included.
  • Added ClerkProperties class to validate Clerk-related secrets are included.
  • Added CloudinaryProperties class to validate Cloudinary-related secrets are included.
  • Added DataSourceProperties class to validate data source secrets are included.

Security

• Removed AuthenticationConfig - the application now depends on Clerk as an external IdP.
• Added ClerkOnboardingFilter to analyse the token claims of an authenticated token, rejecting access with 403 Forbidden if the required claims are:
  1. Not present.
  2. Malformed.
  3. Indicative of a user not having completed the onboarding process.
• Added OAuth Resource Server AccessDeniedHandler and AuthenticationEntryPoint to ensure authentication-related exceptions are correctly handled by GlobalControllerAdvice.
• Refactored SecurityConfig SecurityFilterChain :
  • Replaced formLogin with oauth2ResourceServer.
  • Disabled sessionManagement (stateless session creation policy).
  • Added api/v1/dev/** and api/v1/clerk/webhook as public request endpoints.
  • Added ClerkOnboardingFilter to the filter chain using addFilterBefore AuthorizationFilter

Controllers

• Removed AuthController and all associated endpoints.
• Removed AccountController and all associated endpoints.
• Added ClerkController
  • POST /api/v1/clerk/onboarding: Completes onboarding for authenticated user.
  • POST /api/v1/clerk/webhook: Handles Clerk webhook events.
• Added DevController
  • POST /api/v1/dev/clerk/persist-all: Persists all Clerk users to the local database.
  • DELETE /api/v1/dev/clerk/sync-all: Updates the onboarding status of all Clerk users to false.

NOTE: DevController is only available under the dev profile.

Services

• Removed AuthService entirely.
• Refactored AccountService to UserService:
  • Removed all pre-existing methods.
  • Added methods upsertFromExternalSource deleteFromExternalSource for handling synchronisation between the external IdP and the local application.
• Added clerk package:
  • Added ClerkSyncService to handle user synchronisation between Clerk and the local application.
  • Added ClerkWebhookService to handle supported Clerk webhook events.
  • Added ClerkSdkService to handle direct interactions with Clerk's backend API.
• Added DevService:
  • Added persistAllUsers method for synchronising all development mode Clerk users to the local db.
  • Added unsyncAllUsers method for removing the onboarded status from all development mode Clerk users for cases when the local db is dropped.
• Refactored SessionService:
  • Removed authenticate and reauthenticate methods.
  • Added getAuthenticatedUserId getAuthenticatedUserClerkId isAuthenticatedUserOnboarded methods which fetch the relevant claims from the authenticated Jwt, acting as a wrapper for Spring's SecurityContextHolder.

NOTE: DevService is only available under the dev or test profile.

Repositories

• Renamed AccountRepository to UserRepository.
• Updated UserRepository methods:
  • Removed findByUsername
  • Removed existsByUsername
  • Added findByExternalId
  • Added existsByExternalId
  • Added deleteByExternalId

Entities

• Renamed Account entity to User entity, and updated its fields to reflect changes made to the user table.
• Added UserStatus enum, with single constant ACTIVE. This will be built upon in the future.
• Updated Profile entity fields to reflect changes made to the profile table.
• Added Profile static factory methods forUser and forTest.

Exceptions

• Added GlobalControllerAdvice exception handlers for:
  • InsufficientAuthenticationException - cases where no authentication is provided with the request.
  • InvalidBearerTokenException - cases when the supplied bearer token is invalid for whatever reason.
  • AccessDeniedException - cases when authorization has failed, such as when the user is rejected by ClerkOnboardingFilter after authentication has taken place.
• Updated custom Exception classes:
  • Removed UnauthorisedRequestException - unauthorised requests are instead rejected with more specific exceptions thrown by Spring Security's OAuth Resource Server defaults.
  • Removed IncorrectCurrentPasswordException - authentication now handled by an IdP.
  • Removed UsernameAlreadyExistsException - authentication now handled by an IdP.
  • Added DeserializationException for cases when there was an issue deserializing an object with a custom implementation of a deserializer.
  • Added ClerkException for cases when there was an issue interacting with the Clerk SDK.
  • Added WebhookVerificationException for cases when there was an issue verifying the legitimacy of a received webhook via its signature.

DTOs

• Removed account request-related DTOs entirely.
• Removed auth request-related DTOs entirely.
• Added ClerkWebhookEvent record:
  • Contains nested enum ClerkWebhookEventType that declares the supported event types.
  • Contains nested abstract data class ClerkWebhookEventData with implementations UserUpsert and UserDelete.
  • Webhook payloads are compared to the supported event types and deserialized using a custom implementation of the Jackson deserializer.
• Updated PageDTO response object to use native Java URI type for previous and next fields, and updated the PageMapper to handle native URI construction using builders.
• Updated ProfileDTO response object by removing banner_url field entirely, and replacing the avatar_url field with a simple image_url field, as reflected in the changes to the profile schema.
• Updated SimplifiedProfileDTO response object by replacing the avatar_url field with a simple image_url field, as reflected in the changes to the profile schema.
• Added a new DTO subclass adapter for internal-use transfer objects:
  • Added ClerkUserDTO adapter with a ClerkUserMapper for mapping Clerk SDK user responses to this smaller, easier to consume internal object.

Misc

• Removed dev mode AccountLoader and related user JSON data from resources/data/users.json.
• Added Utils class with utility methods checkNotNull convertHeaders.
• Added OffsetLimitRequest static factory method of for easier usage.
• Removed custom Username and Password annotation validators.

Testing

• Unit tests:

  • Refactored all unit test method names to more accurately describe each individual tests logic.
  • Refactored all @WebMvcTest controller tests to use MockMvcTester with assertJ assertThat.
  • Removed unit tests for deleted custom annotation validators.
  • Added unit testing for OffsetLimitRequest.
  • Added improved unit testing for PageMapper.
  • Added unit testing for ClerkWebhookEventType static method fromString.
  • Added unit testing for ClerkWebhookEventDeserializer.
  • Added unit testing for ClerkUserMapper.
  • Added unit testing for ClerkOnboardingFilter.
  • Added unit testing for ClerkSyncService.
  • Added unit testing for ClerkWebhoookService.
  • Added unit testing for SessionService.
  • Added unit testing for UserService.
  • Added unit testing for ClerkController.

• Integration tests:

  • Created a dedicated CI/CD Clerk instance and integrated with the test environment using GitHub Actions secrets.
  • Refactored base IntegrationTest class to connect to CI/CD Clerk instance via the backend SDK, creating and deleting Clerk users as part of the integration test lifecycle.
  • Added ClerkTestUtils utility class to interface interactions with the Clerk SDK during integration tests.
  • Refactored all integration test method names to more accurately describe each individual tests logic.
  • Refactored all @SpringBootTest controller tests to use WebTestClient with native API assertions, replacing the old TestRestTemplate.
  • Added DatabaseCleaner as simple method of wiping relevant tables between integration tests, preventing dirty databases from affecting chained tests within the same context.
  • Deleted redundant SQL files resources/sql/post-interaction-cleanup.sql resources/sql/profile-interaction-cleanup.sql.
  • Removed AccountRepositoryIT class entirely.
  • Added integration testing for UserService to ensure the IdP upsert logic.
  • Added integration testing for ClerkOnboardingFilter to ensure the filters logic with real tokens.
  • Added integration testing for ClerkController.