Skip to content

Swift SDK — Tier 3 Assessment #2309

New issue
New issue
Open
Open
Swift SDK — Tier 3 Assessment#2309
@movetz

Description

@movetz
movetz
opened on Feb 24, 2026

Swift SDK - Tier 3 Assessment

Date: 2026-02-23
Branch: main
SDK Version: v0.11.0
Auditor: mcp-sdk-tier-audit skill (automated + subagent evaluation)
Result: Tier 3

Tier Assessment: Tier 3

The Swift SDK achieves perfect server conformance (30/30) and has a comprehensive README with rich code examples, but is blocked from Tier 2 by incomplete OAuth/auth client conformance (20% date-versioned), missing label taxonomy, a triage compliance rate below the 80% threshold, pre-1.0 versioning, and the absence of roadmap and versioning documentation.

Currently, the main functionality is covered, and we are moving forward with the auth updates. We have also discovered several org gaps (such as roadmap, documentation, etc.). We plan to address the org updates soon (this week), while the auth updates require additional discussion before we can provide a final estimate.

Requirements Summary

# Requirement Tier 1 Standard Tier 2 Standard Current Value T1? T2? Gap
1a Server Conformance 100% pass rate >= 80% pass rate 100% (30/30) PASS PASS None
1b Client Conformance 100% pass rate >= 80% pass rate 20% (4/20 date-versioned) FAIL FAIL All 16 auth scenarios failing; core 4/4 pass
2 Issue Triage >= 90% within 2 biz days >= 80% within 1 month 66.7% (24/36) FAIL FAIL 12 issues exceeding SLA; median 663h, p95 6193h
2b Labels 12 required labels 12 required labels 5/12 FAIL FAIL Missing: P0–P3, needs-confirmation, needs-repro, ready-for-work
3 Critical Bug Resolution All P0s within 7 days All P0s within 2 weeks 0 open P0s PASS PASS None (P0 label absent — 0 may reflect labeling gap)
4 Stable Release Required + clear versioning At least one stable release 0.11.0 (pre-1.0) FAIL FAIL No stable release >= 1.0.0
4b Spec Tracking SDK release within 30d of spec Within 6 months 86d gap FAIL PASS 86d > 30d for T1; 86d < 180d for T2
5 Documentation Comprehensive w/ examples Basic docs for core features 29/48 PASS; 10/36 core features undoc. FAIL FAIL 14 features FAIL; 5 PARTIAL; 10 core features have no docs
6 Dependency Policy Published update policy Published update policy Not found FAIL FAIL No DEPENDENCY_POLICY.md, dependabot.yml, or renovate.json
7 Roadmap Published roadmap Plan toward Tier 1 Not found FAIL FAIL No ROADMAP.md or docs/roadmap.md
8 Versioning Policy Documented breaking change policy N/A Not found FAIL N/A No VERSIONING.md or BREAKING_CHANGES.md

Tier Determination

  • Tier 1: FAIL — 2/10 requirements met (failing: client conformance, triage, labels, stable release, spec tracking, documentation, dependency policy, roadmap, versioning policy)
  • Tier 2: FAIL — 3/9 requirements met (failing: client conformance, triage, labels, stable release, documentation, dependency policy, roadmap)
  • Final Tier: 3

Server Conformance Details

Pass rate: 100% (30/30)

Scenario Status Checks
server-tools-list PASS 1/1
server-tools-call-with-progress PASS 1/1
server-tools-call-with-logging PASS 1/1
server-tools-call-simple-text PASS 1/1
server-tools-call-sampling PASS 1/1
server-tools-call-mixed-content PASS 1/1
server-tools-call-image PASS 1/1
server-tools-call-error PASS 1/1
server-tools-call-embedded-resource PASS 1/1
server-tools-call-elicitation PASS 1/1
server-tools-call-audio PASS 1/1
server-server-sse-multiple-streams PASS 2/2
server-server-initialize PASS 1/1
server-resources-unsubscribe PASS 1/1
server-resources-templates-read PASS 1/1
server-resources-subscribe PASS 1/1
server-resources-read-text PASS 1/1
server-resources-read-binary PASS 1/1
server-resources-list PASS 1/1
server-prompts-list PASS 1/1
server-prompts-get-with-image PASS 1/1
server-prompts-get-with-args PASS 1/1
server-prompts-get-simple PASS 1/1
server-prompts-get-embedded-resource PASS 1/1
server-ping PASS 1/1
server-logging-set-level PASS 1/1
server-elicitation-sep1330-enums PASS 5/5
server-elicitation-sep1034-defaults PASS 5/5
server-dns-rebinding-protection PASS 2/2
server-completion-complete PASS 1/1

Client Conformance Details

Full suite pass rate: 20.8% (5/24)

Suite breakdown: Core: 4/4 (100%), Auth date-versioned: 0/16 (0%), Auth informational: 1/4 (25%)
Baseline: None found.

Conformance Matrix (date-versioned scenarios only — scored for tier)

2025-03-26 2025-06-18 2025-11-25 All*
Server — 26/26 30/30 30/30 (100%)
Client: Core — 2/2 4/4 4/4 (100%)
Client: Auth 0/2 0/3 0/11 0/16 (0%)
Client Total 0/2 2/5 4/15 4/20 (20%)

* unique scenarios — a scenario may apply to multiple spec versions

Informational (not scored for tier)

draft extension
Client: Auth 1/1 0/3

Interpretation: All client failures are in the Auth category. The 0/16 rate across all date-versioned auth scenarios indicates OAuth/authorization is not yet implemented in the client. Core client functionality is fully conformant (4/4). The auth/resource-mismatch draft scenario passes, suggesting partial auth plumbing exists.

Core Scenarios

Scenario Status Checks
tools_call PASS 1/1
sse-retry PASS 3/3
initialize PASS 1/1
elicitation-sep1034-client-defaults PASS 5/5

Auth Scenarios (date-versioned)

Scenario Spec Version Status Checks Notes
auth/2025-03-26-oauth-metadata-backcompat 2025-03-26 FAIL 0/4 Auth not implemented
auth/2025-03-26-oauth-endpoint-fallback 2025-03-26 FAIL 0/3 Auth not implemented
auth/token-endpoint-auth-post 2025-06-18, 2025-11-25 FAIL 0/3 Auth not implemented
auth/token-endpoint-auth-none 2025-06-18, 2025-11-25 FAIL 0/3 Auth not implemented
auth/token-endpoint-auth-basic 2025-06-18, 2025-11-25 FAIL 0/3 Auth not implemented
auth/scope-step-up 2025-11-25 FAIL 0/2 Auth not implemented
auth/scope-retry-limit 2025-11-25 FAIL 0/1 Auth not implemented
auth/scope-omitted-when-undefined 2025-11-25 FAIL 0/1 Auth not implemented
auth/scope-from-www-authenticate 2025-11-25 FAIL 0/1 Auth not implemented
auth/scope-from-scopes-supported 2025-11-25 FAIL 0/1 Auth not implemented
auth/pre-registration 2025-11-25 FAIL 0/1 Auth not implemented
auth/metadata-var3 2025-11-25 FAIL 0/4 Auth not implemented
auth/metadata-var2 2025-11-25 FAIL 0/4 Auth not implemented
auth/metadata-var1 2025-11-25 FAIL 0/5 Auth not implemented
auth/metadata-default 2025-11-25 FAIL 0/5 Auth not implemented
auth/basic-cimd 2025-11-25 FAIL 0/1 Auth not implemented

Auth Scenarios (informational — not scored for tier)

Scenario Spec Version Status Checks
auth/resource-mismatch draft PASS 1/1
auth/cross-app-access-complete-flow extension FAIL 0/2
auth/client-credentials-jwt extension FAIL 0/1
auth/client-credentials-basic extension FAIL 0/1

Issue Triage Details

Analysis period: Last 36 issues
Labels present: bug, enhancement, question, good first issue, help wanted (5/12)
Labels missing: needs-confirmation, needs-repro, ready-for-work, P0, P1, P2, P3 (7 missing)

Metric Value T1 Req T2 Req Verdict
Compliance rate 66.7% >= 90% >= 80% FAIL
Issues triaged 24/36 — — —
Exceeding SLA 12 — — —
Median hours 663h (~27.6d) — — —
P95 hours 6193h (~258d) — — —
Open P0s 0 0 0 PASS*

* P0 label does not exist in the repo — 0 open P0s trivially passes but does not reflect actual critical bug tracking.

Documentation Coverage

Documentation Coverage Assessment

Documentation locations found:

  • README.md: Comprehensive 1408-line README covering client and server usage with inline Swift code examples for most features
  • SECURITY.md: Security policy only (no feature docs)
  • No docs/ or documentation/ directory
  • No examples/ directory
  • No DocC catalog found in Sources/

Feature Documentation Table

# Feature Documented? Where Has Examples? Verdict
1 Tools - listing Yes README.md:134-140 Yes (2 examples) PASS
2 Tools - calling Yes README.md:140-173 Yes (2 examples) PASS
3 Tools - text results Yes README.md:155-159 Yes (switch .text) PASS
4 Tools - image results Yes README.md:159-164 Yes (switch .image) PASS
5 Tools - audio results Yes README.md:165-167 Yes (switch .audio) PASS
6 Tools - embedded resources Yes README.md:167-169 Yes (switch .resource) PASS
7 Tools - error handling Yes README.md:460-468, 733-737 Yes (isError, MCPError) PASS
8 Tools - change notifications Partial README.md:664-676 (capability) No notification handler example PARTIAL
9 Resources - listing Yes README.md:175-183 Yes (1 example) PASS
10 Resources - reading text Yes README.md:185-201 Yes (1 example) PASS
11 Resources - reading binary Partial README.md:764-787 (text only) No binary blob example PARTIAL
12 Resources - templates No — No FAIL
13 Resources - template reading No — No FAIL
14 Resources - subscribing Yes README.md:188-201 Yes (1 example) PASS
15 Resources - unsubscribing No — No FAIL
16 Resources - change notifications Yes README.md:193-201 Yes (ResourceUpdatedNotification) PASS
17 Prompts - listing Yes README.md:204-215 Yes (1 example) PASS
18 Prompts - getting simple Yes README.md:214-229 Yes (1 example) PASS
19 Prompts - getting with arguments Yes README.md:214-229 Yes (1 example) PASS
20 Prompts - embedded resources No — No FAIL
21 Prompts - image content No — No FAIL
22 Prompts - change notifications No — No FAIL
23 Sampling - creating messages Yes README.md:287-330, 938-970 Yes (client + server) PASS
24 Elicitation - form mode Yes README.md:332-390, 979-1018 Yes (client + server) PASS
25 Elicitation - URL mode Yes README.md:376-380, 1020-1028 Yes (client + server) PASS
26 Elicitation - schema validation Partial README.md:984-998 Partial (schema shown, no validation example) PARTIAL
27 Elicitation - default values No — No FAIL
28 Elicitation - enum values No — No FAIL
29 Elicitation - complete notification No — No FAIL
30 Roots - listing Yes README.md:391-417, 1030-1047 Yes (client + server) PASS
31 Roots - change notifications Yes README.md:415-416, 1042-1046 Yes (notifyRootsChanged + RootsListChangedNotification) PASS
32 Logging - sending log messages Yes README.md:1049-1098 Yes (3 examples) PASS
33 Logging - setting level Yes README.md:419-443, 1101-1113 Yes (client + server) PASS
34 Completions - resource argument Yes README.md:275-285, 894-906 Yes (client + server) PASS
35 Completions - prompt argument Yes README.md:233-274, 880-893 Yes (client + server) PASS
36 Ping No — No FAIL
37 Streamable HTTP transport (client) Yes README.md:109-131 Yes (HTTPClientTransport) PASS
38 Streamable HTTP transport (server) Partial README.md:1285-1298 (table) No code example PARTIAL
39 SSE transport - legacy (client) No — No FAIL
40 SSE transport - legacy (server) No — No FAIL
41 stdio transport (client) Yes README.md:109-118 Yes (StdioTransport) PASS
42 stdio transport (server) Yes README.md:659-682 Yes (StdioTransport) PASS
43 Progress notifications Yes README.md:521-547, 1116-1152 Yes (client + server) PASS
44 Cancellation Yes README.md:471-519 Yes (2 examples) PASS
45 Pagination Partial README.md:181-182 (cursor returned) No pagination loop example PARTIAL
46 Capability negotiation Yes README.md:92-107, 551-587 Yes (capabilities init, strict/default) PASS
47 Protocol version negotiation No — No FAIL
48 JSON Schema 2020-12 support No — No FAIL
— Tasks - get (experimental) No — No INFO
— Tasks - result (experimental) No — No INFO
— Tasks - cancel (experimental) No — No INFO
— Tasks - list (experimental) No — No INFO
— Tasks - status notifications (experimental) No — No INFO

Summary

Total non-experimental features: 48
PASS (documented with examples): 29/48
PARTIAL (documented, no examples or prose-only): 5/48
FAIL (not documented): 14/48

Core features with at least basic docs (PASS+PARTIAL): 26/36 (72%)
All features documented with examples: 29/48 (60%)

Tier Verdicts

Tier 1 (all non-experimental features documented with examples): FAIL

Features missing documentation or examples: #8, #11, #12, #13, #15, #20, #21, #22, #26, #27, #28, #29, #36, #38, #39, #40, #45, #47, #48

Tier 2 (basic docs covering core features): FAIL

Core features completely missing documentation: #12 Resources-templates, #13 Resources-template-reading, #15 Resources-unsubscribing, #20 Prompts-embedded-resources, #21 Prompts-image-content, #22 Prompts-change-notifications, #27 Elicitation-default-values, #28 Elicitation-enum-values, #29 Elicitation-complete-notification, #36 Ping

Policy Evaluation

Policy Evaluation Assessment

SDK path: /Users/yehorsobko/Documents/Macpaw/swift-sdk
Repository: modelcontextprotocol/swift-sdk

1. Dependency Update Policy: FAIL

File Exists (CLI) Content Verdict
DEPENDENCY_POLICY.md No N/A
docs/dependency-policy.md No N/A
.github/dependabot.yml No N/A
.github/renovate.json No N/A
renovate.json No N/A

Verdict: FAIL — No dependency update policy files exist in the repository.

2. Roadmap: FAIL

File Exists (CLI) Content Verdict
ROADMAP.md No N/A
docs/roadmap.md No N/A

Verdict:

  • Tier 1: FAIL — No roadmap file exists.
  • Tier 2: FAIL — No roadmap file exists (not even a plan toward Tier 1).

3. Versioning Policy: FAIL

File Exists (CLI) Content Verdict
VERSIONING.md No N/A
docs/versioning.md No N/A
BREAKING_CHANGES.md No N/A
CONTRIBUTING.md (versioning section) No N/A

Note: README.md:1394-1400 mentions Semantic Versioning and notes that pre-1.0 minor bumps may contain breaking changes — useful context, but not a standalone policy document.

Verdict:

  • Tier 1: FAIL — No dedicated versioning policy document found.
  • Tier 2: N/A — only requires stable release.

Overall Policy Summary

Policy Area Tier 1 Tier 2
Dependency Update Policy FAIL FAIL
Roadmap FAIL FAIL
Versioning Policy FAIL N/A

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions