Add support for expert access to mcp servers with auth

[Steve-Mcl]

Description

This pull request permits the FF Expert to access instance MCP servers by issuing short-lived tokens that are passed to the AI backend for the MCP client connections to authenticate.

These special-purpose tokens (with the ff-expert:mcp scope) are not exposed to the user in the UI nor are they modifiable like regular project HTTP token APIs

Expert MCP HTTP Token Restrictions:

• Prevents creation, modification, and listing of Expert MCP HTTP tokens (ff-expert:mcp scope) via the standard project HTTP token API routes, ensuring these tokens can only be managed internally. [1] [2] [3] [4]
• Adds explicit checks in the authentication flow to ensure only HTTP tokens are accepted for Expert MCP access.
• Adds unit tests to verify that Expert MCP tokens cannot be created, modified, or listed via the API.

Expert API MCP Server Feature and Token Management:

• Implements in-memory caching (with LRU eviction and TTL) for MCP access tokens to minimize unnecessary token creation and database access, and ensures tokens are only used if they have sufficient remaining lifetime.
• Refactors the logic for filtering and preparing MCP servers and their features for the user, including:
  • Caching applications and instances for efficiency.
  • Ensuring only accessible and running MCP servers are presented.
  • Generating or retrieving appropriate access tokens for each instance, based on instance security settings and team features.
  • Attaching the generated access token to each MCP server object returned to the client. [1] [2] [3] [4]

Code Quality and Documentation:

• Improves comments and documentation throughout the affected files, clarifying the rationale behind the filtering and token management logic.
• Minor type and parameter documentation corrections for clarity. [1] [2]

Unit tests added: test/unit/forge/routes/api/expert_spec.js

▼ Expert API
  ▼ service enabled
    ▼ Chat Endpoint
      ✔ should include only permitted mcp features when granular RBACs is enabled
      ✔ should not generate an access token for MCP server when feature teamHttpSecurity is disabled
      ✔ should not generate an access token for MCP server when instance setting httpNodeAuth is not set
      ✔ should generate an access token for MCP server access when feature teamHttpSecurity is enabled
    ▼ MCP features Endpoint
      ✔ should not generate an access token for MCP server when feature teamHttpSecurity is disabled
      ✔ should not generate an access token for MCP server when instance setting httpNodeAuth is not set
      ✔ should generate an access token for MCP server access when feature teamHttpSecurity is enabled
      ✔ should get MCP server access token from cache
      ✔ should use basic auth for MCP server access when httpNodeAuth is set to basic

Unit tests added: test/unit/forge/ee/routes/httpTokens/index_spec.js

▼ NR HTTP Bearer Tokens
  ✔ cannot create Expert MCP HTTP token via API
  ✔ cannot modify an existing Expert MCP HTTP token via API
  ✔ does not list Expert MCP tokens via API

UPDATE:

Handle instance auth change cache invalidation

Additional tests added to test/unit/forge/routes/api/expert_spec.js

▼ Expert API
  ▼ service enabled
    ▼ Chat Endpoint
      ✔ should clear cached MCP server access token when project setting httpNodeAuth is changed
    ▼ MCP features Endpoint
      ✔ should clear cached MCP server access token when project setting httpNodeAuth is changed

Related Issue(s)

closes #6546 - implementation
closes #6547 - tests

Checklist

• I have read the contribution guidelines
• Suitable unit/system level tests have been added and they pass
• Documentation has been updated
  • Upgrade instructions
  • Configuration details
  • Concepts
• Changes flowforge.yml?
  • Issue/PR raised on FlowFuse/helm to update ConfigMap Template
  • Issue/PR raised on FlowFuse/CloudProject to update values for Staging/Production
• Link to Changelog Entry PR, or note why one is not needed.

Labels

• Includes a DB migration? -> add the area:migration label

[codecov]

Codecov Report

❌ Patch coverage is 90.47619% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.62%. Comparing base (debae34) to head (fd4610d).
⚠️ Report is 29 commits behind head on main.

Files with missing lines	Patch %	Lines
forge/routes/api/expert.js	84.78%	7 Missing ⚠️
forge/routes/auth/index.js	25.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6549      +/-   ##
==========================================
+ Coverage   76.45%   76.62%   +0.17%     
==========================================
  Files         397      398       +1     
  Lines       20011    20093      +82     
  Branches     4812     4836      +24     
==========================================
+ Hits        15300    15397      +97     
+ Misses       4711     4696      -15     

Flag	Coverage Δ
backend	76.62% <90.47%> (+0.17%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Steve-Mcl]

2 issues unearthed while testing:

1. Basic auth - pass is encrypted in DB so not aviable to provide as token for MCP
2. Due to token caching, when instance http node auth is changed, its MCP servers are not accessible until the cache expires

No 1 requires some design and thought (and a chat with Nick)
No 2 is solved in an incoming commit. Essentially, clear the cache for an instance if instance settings are altered

[Steve-Mcl]

No 1 requires some design and thought (and a chat with Nick)

Meeting scheduled for Tomorrow to resolve this.

It is looking likely basic auth will be excluded at this time. As such, this PR will require a small change to filter out MCP servers that have basic auth enabled in the instance settings (both in the MCP Feature request endpoint and in the chat endpoint) (and tests will need adjusting too)

[Steve-Mcl]

No 1 requires some design and thought (and a chat with Nick)

Meeting scheduled for Tomorrow to resolve this.

Not supporting Basic auth at this time.

Instead, to help a user understand (at point of use) that an MCP server is "there" but not "available" we should display them disabled (or display them conditionally via a checkbox or similar)

See #6553 which is covers this