feat(backend): introduce handshake (#2300)

* feat(backend): Try the new Client Handshake mechanism

* feat(backend): Update authenticateRequest handler to support multi-domain handshake

* feat(repo): Introduce tests for client handshake (#2265)

* Test suite start

* feat(backend,nextjs,utils): Fix jest

* first test

* Fix bug in jwks cache for multiple runtime keys

* Add all the tests, including many failing

* Add all the tests, including many failing

* fix(shared): Correctly construct proxy URL

---------

Co-authored-by: Nikos Douvlis <nikosdouvlis@gmail.com>
Co-authored-by: Bryce Kalow <bryce@clerk.dev>

* chore(backend): Refactor authenticateRequest to clarify logic

* fix(backend): Fix options passing to authenticateRequest

* feat(backend): Add sec-fetch-dest check for satellite sync, adjust tests to support additional URLs

* feat(backend): Account for clock skew in dev, but still log error

* feat(backend): Refactor backend tests to account for recent refactoring to authenticateRequest

* feat(backend): Treat handshake payload as a signed jwt

* fix(backend): Add tests and adjust logic to ensure existing tests pass

* chore(backend): Refactor tests to conform to new method signature

* chore(repo): Add changeset

* feat(*): Drop interstitial (#2304)

* feat(backend): Remove interstitial endpoints

* feat(backend,types): Remove local interstitial script

* feat(types): Clean retheme types

* feat(backend): Remove interstitial and interstitial rules

* feat(clerk-js): Remove interstitial from clerk-js

* feat(nextjs): Remove interstitial from authMiddleware

* feat(fastify): Remove interstitial

* feat(gatsby-plugin-clerk): Remove interstitial

* feat(remix): Remove interstitial

* feat(clerk-sdk-node): Remove interstitial

* fix(nextjs): Always respect redirect header if found

As it's possible that we trigger a redirect from authenticateRequest that isn't a handshake status (dev multi-domain sync, for example)

* chore(repo): Fix sdk tests

* fix(clerk-js): Fix tests related to db-jwt

* fix(clerk-js): Keep hasJustSynced check

* chore(*): Fix linter

* chore(backend): Remove unused AuthErrorReason properties, destructure from authenticateContext

* chore(clerk-js): Remove unused @ts-expect-error directive

* fix: Address build issues in sdk-node

* fix(remix): Correct Remix build issues

* chore(repo): Apply linting fixes

---------

Co-authored-by: Sokratis Vidros <sokratis.vidros@gmail.com>
Co-authored-by: Colin Sidoti <51144033+colinclerk@users.noreply.github.com>
Co-authored-by: Nikos Douvlis <nikosdouvlis@gmail.com>
Co-authored-by: Colin Sidoti <colin@clerk.dev>

.changeset/nice-doors-fail.md

@@ -0,0 +1,9 @@
+---
+'@clerk/backend': major
+---
+
+- Refactor the `authenticateRequest()` flow to use the new client handshake endpoint. This replaces the previous "interstitial"-based flow. This should improve performance and overall reliability of Clerk's server-side request authentication functionality.
+- `authenticateRequest()` now accepts two arguments, a `Request` object to authenticate and options:
+  ```ts
+  authenticateRequest(new Request(...), { secretKey: '...' })
+  ```