feat: MCP server integration — full read+write surface, hybrid architecture, per-user credentials

[teslashibe]

MCP server integration — full read + write surface for all teslashibe packages

Summary

Expose every teslashibe scraper, scoring, and service package as a set of MCP (Model Context Protocol) tools available to the Claude Managed Agent that ships in the agent-setup template. The architecture is designed so that MCP tool definitions live in the package they wrap (single source of truth, no drift) and agent-setup owns transport, auth, credentials, and response shaping (one place for cross-cutting concerns). Adding a new package is a 4-step process; bumping an existing package is automatic.

Scope covers both read and write functionality on every in-scope package.

---

Architecture

User → Expo client → Fiber API (:8090) → Anthropic Managed Agent
                                                ↓ (MCP over HTTP/SSE)
                                  agent-setup/backend/internal/mcp
                                                ↓ resolves per-user creds, builds *Client
                            linkedin/mcp · x/mcp · facebook/mcp · …
                                                ↓ Provider.Tools() handler.Invoke
                                  linkedin-go · x-go · facebook-go · …

Hybrid layout (the only architecture decision worth making)

• Each scraper package owns its tool definitions in a thin mcp/ subpackage. When the package adds a method, the tool ships in the same PR. Tests live next to the code they wrap. Reusable beyond this template.
• agent-setup/backend/internal/mcp/ owns MCP transport, per-user authentication, credential resolution, response middleware (truncation, pagination caps, compact JSON), and the registry that mounts every provider.
• github.com/teslashibe/mcptool (new tiny repo, ~200 LOC) is the shared contract every package depends on. Stable. Stdlib + invopop/jsonschema only.

github.com/teslashibe/mcptool          ← shared Tool/Provider contract + helpers
github.com/teslashibe/linkedin-go/mcp  ← Provider, ~16 tools, coverage test
github.com/teslashibe/x-go/mcp         ← Provider, ~35 tools, coverage test
… (same pattern in every package)
github.com/teslashibe/agent-setup/backend/internal/mcp
                                       ← transport, auth, registry, middleware

---

Five mechanisms that prevent drift and let us scale

1. Shared contract package — github.com/teslashibe/mcptool

type Tool struct {
    Name, Description string
    InputSchema       map[string]any
    WrapsMethod       string                                    // for coverage tests
    Invoke            func(ctx, client any, raw json.RawMessage) (any, error)
}

func Define[C any, I any](
    name, desc, wrapsMethod string,
    handler func(ctx context.Context, c C, in I) (any, error),
) Tool                                                           // schema = jsonschema.Reflect(new(I))

type Provider interface {
    Platform() string                                            // "linkedin"
    Tools() []Tool
}

// Uniform helpers for every package
func PageOf[T any](items []T, cursor string) Page[T]
func TruncateString(s string, max int) string
func CompactJSON(v any) string

2. Schema derived from typed Go input structs

type SearchPeopleInput struct {
    Query string `json:"query" jsonschema:"description=keywords or name,required"`
    Limit int    `json:"limit,omitempty" jsonschema:"minimum=1,maximum=50,default=10"`
}

Method signature drift → struct field drift → schema regenerates on build. Zero hand-maintained JSON schema.

3. Coverage test in every package's mcp/ subpackage

Reflects on *Client, asserts every exported method is either wrapped by a tool or in a documented exclusion list (excluded.go with reason comments). Fails CI the moment a contributor adds a method without exposing it.

func TestEveryClientMethodIsWrappedOrExcluded(t *testing.T) {
    methods := exportedMethods(reflect.TypeOf(&linkedin.Client{}))
    wrapped := wrapsMethodSet(linkmcp.Provider{}.Tools())
    for _, m := range methods {
        if !wrapped[m] && !excluded[m] {
            t.Fatalf("linkedin.%s is not exposed via MCP and not in exclusion list", m)
        }
    }
}

4. Auto-bump pipeline in agent-setup

• Dependabot for backend/go.mod: weekly PRs for all 14 packages, auto-merge if CI passes
• Nightly workflow: go get -u && go test against latest minor of every package — breaking changes surface within 24h
• go.work file for local dev across packages without publishing

5. Cursor rule — agent-setup/.cursor/rules/mcp-tool-conventions.mdc

Loaded automatically when editing files matching *-go/mcp/** or backend/internal/mcp/**. Documents naming, descriptions, schema patterns, response shaping, exclusion-list etiquette. Same conventions applied by humans and AI.

---

Adding a new package — 4 steps

1. Build the package as normal (e.g. bluesky-go)
2. Add bluesky-go/mcp/ subpackage using mcptool.Define[*bluesky.Client, …] for each tool + excluded.go for any unexposed methods
3. Coverage test runs in CI — guarantees no method left behind
4. In agent-setup: one line in providers slice + add to Dependabot. Done.

---

Per-user credentials — encrypted DB + Settings UI

Storage

CREATE TABLE platform_credentials (
    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id         UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
    platform        TEXT NOT NULL,
    credential      BYTEA NOT NULL,                   -- AES-GCM encrypted JSON blob
    label           TEXT,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    last_used_at    TIMESTAMPTZ,
    UNIQUE (user_id, platform)
);

• Encryption: AES-GCM with a key from CREDENTIALS_ENCRYPTION_KEY env var (32 bytes, base64).
• Nonce per row, stored as prefix on the ciphertext.
• Plaintext blob: {"cookies": "raw cookie string OR JSON exported from extension", "format": "cookie-string|cookie-json|bearer", ...platform-specific fields}.

CRUD endpoints

Method	Path	Purpose
POST /api/platforms/:platform/credentials	upsert credential for current user
GET /api/platforms/:platform/credentials	metadata only (created_at, last_used_at, label) — never plaintext
GET /api/platforms	list all platforms, marking which the user has credentials for
DELETE /api/platforms/:platform/credentials	revoke

Mobile UI — Settings → Platform Connections

• Section per platform with status (Connected / Not Connected) + last-used timestamp
• "Connect" opens a modal with: paste-cookie textarea + a one-line link to the cookie-extractor Chrome extension
• Accepts both raw cookie strings and JSON exported from the extension; backend normalizes
• Disconnect button per platform

Per-request flow

1. Anthropic agent calls our MCP server with the user's signed JWT (issued at session create)
2. MCP middleware validates JWT → resolves user_id
3. Tool dispatch looks up platform_credentials for (user_id, platform), decrypts, builds the *Client
4. Tool invokes; if it returns a 401/403, mark the credential last_failed_at and return a structured "credential expired, please reconnect" error so the agent can surface it

---

Per-user agent provisioning

The Anthropic Managed Agent's MCP URL is configured at agent-create time and is static for the agent's lifetime, so per-user credential isolation requires per-user agents.

• On first session for a user, lazily create an Anthropic Agent + Environment named engagement-studio-user-<userID> with MCP URL https://api.example.com/mcp/v1?user=<userID>&token=<server-issued JWT>
• Cache the agent + environment IDs on the user row
• Subsequent sessions reuse the same agent
• Provision script (cmd/provision/main.go) becomes idempotent and is also exposed as a service method called from session creation

---

Token-efficient response shaping

Every tool response goes through middleware that:

• Caps array lengths — pagination defaults to limit=10, max 50 per tool. Beyond that, the response includes a next_cursor and the agent re-invokes
• Truncates long strings — post bodies, comments default to 800 chars with … and a truncated: true flag; the agent can pass expand=true to get full content for a specific item
• Drops noisy fields — internal IDs, raw HTML, redundant nested objects unless explicitly requested via fields=
• Compact JSON — no indentation, omit nulls/empty values
• Summary shapes preferred — list endpoints return {id, title, author, score, created_at}-style summaries; full object fetched via a separate get_* tool when needed

These rules are implemented once in agent-setup/backend/internal/mcp/middleware/ and applied uniformly. Tools opt in via mcptool.PageOf[T] / mcptool.Summary[T] helpers.

---

In-scope packages and tool surface

Counts below reflect the actual exported *Client methods as of audit. Each platform exposes both read and write where the package supports it.

Scrapers (10)

#	Package	Module	Read	Write	Total
1	linkedin-go	github.com/teslashibe/linkedin-go	12	4	16
2	x-go	github.com/teslashibe/x-go	19	17	36
3	facebook-go	github.com/teslashibe/facebook-go/groups	11	7	18
4	tiktok-go	github.com/teslashibe/tiktok-go	17	11	28
5	threads-go	github.com/teslashibe/threads-go	24	17	41
6	reddit-go	github.com/teslashibe/reddit-go	13	17	30
7	hn-go	github.com/teslashibe/hn-go	28	10	38
8	nextdoor-go	github.com/teslashibe/nextdoor-go	12	10	22
9	producthunt-go	github.com/teslashibe/producthunt-go	22	16	38
10	instagram-go	github.com/teslashibe/instagram-go	32	17	49

Scoring libraries (2)

#	Package	Module	Tools
11	x-viral-go	github.com/teslashibe/x-viral-go	x_viral_score, x_viral_optimize
12	redditviral-go	github.com/teslashibe/redditviral-go	reddit_viral_score, reddit_viral_optimize

Service packages (2)

#	Package	Module	Tools
13	elevenlabs-go	github.com/teslashibe/elevenlabs-go	12 (agent CRUD, calls, conversations, phone numbers)
14	codegen-go	github.com/teslashibe/codegen-go	codegen_run (run a coding-agent CLI against a working dir)

Total tool surface: ~330 tools. Full per-tool listing is generated automatically by go run ./cmd/mcp-inventory and committed to docs/mcp-inventory.md on every PR.

Out of scope

• zenoh-go — infrastructure only (Zenoh pub/sub bindings)
• magiclink-auth-go — already integrated as the template's auth layer

---

Detailed write surface (write-side audit, for the doubters)

The following exported write methods exist today and must be exposed:

• linkedin-go: JoinGroup, LeaveGroup, CreateGroupPost, SendMessage
• x-go: CreateTweet, Reply, QuoteTweet, DeleteTweet, Like, Unlike, Retweet, Unretweet, Bookmark, Unbookmark, Follow, Unfollow, Mute, Unmute, Block, Unblock, SendDM, SendNewDM
• facebook-go: CreatePost, CreateComment, ReactToPost, JoinGroup, JoinGroupWithAnswers, LeaveGroup, CreateGroup
• tiktok-go: LikeVideo, FollowUser, CollectVideo, RepostVideo, DeleteRepost, BlockUser, MuteUser, PostComment, ReplyToComment, DeleteComment, LikeComment
• threads-go: CreatePost, Reply, Quote, UploadImage, Like, Unlike, Repost, DeleteRepost, DeletePost, Follow, Unfollow, Block, Unblock, Mute, Unmute, Restrict, Unrestrict
• reddit-go: Subscribe, Unsubscribe, Submit, SubmitLink, Reply, Edit, Delete, Upvote, Downvote, Unvote, Save, Unsave, Hide, Unhide, Report (+ chat send, PM send)
• hn-go: Upvote, Unvote, Hide, Fave, Flag, Comment, Submit, SubmitShowHN, SubmitAskHN, UpdateUserSettings
• nextdoor-go: CreatePost, DeletePost, ReactToPost, RemoveReaction, CreateComment, DeleteComment, CreateChannel, SendMessage, DeleteMessage
• producthunt-go: CreateComment, ReplyToComment, UpdateComment, DeleteComment, Upvote, RemoveUpvote, CreateReview, FollowTopic, UnfollowTopic, CreateCollection, AddPostToCollection, RemovePostFromCollection, FollowCollection, UnfollowCollection, FollowUser, UnfollowUser
• instagram-go: Follow, Unfollow, Block, Unblock, MutePosts, UnmutePosts, FollowHashtag, UnfollowHashtag, MarkStorySeen, PostComment, LikeComment, UnlikeComment, DeleteComment, LikePost, UnlikePost, SavePost, UnsavePost
• elevenlabs-go: CreateAgent, UpdateAgent, DeleteAgent, OutboundCall, ImportTwilioNumber, AssignAgent
• codegen-go: Run / RunJSON (the only "method" — write-by-nature, generates code edits)

---

Deliverables

Repo	What we add
teslashibe/mcptool (new)	Shared Tool / Provider contract, Define[C, I], response helpers (PageOf, TruncateString, CompactJSON), schema reflection. ~200 LOC + tests.
Each of 14 *-go repos	mcp/ subpackage exposing Provider + Tools(), excluded.go with reasoned exclusions, coverage test, README section.
agent-setup/backend/internal/mcp/	Streamable-HTTP transport, per-user JWT auth middleware, registry (one-line per platform), response-shaping middleware, error mapping.
agent-setup/backend/internal/platforms/	Encrypted credentials store, AES-GCM helpers, CRUD handlers, validation/normalization for cookie input formats.
agent-setup/backend/internal/db/migrations/	platform_credentials table; users columns for cached anthropic_agent_id / anthropic_environment_id.
agent-setup/backend/cmd/provision/main.go	Refactored into a service used both by CLI provision and by lazy per-user agent creation in session flow.
agent-setup/backend/cmd/mcp-inventory/main.go	Generates docs/mcp-inventory.md from registered providers.
agent-setup/mobile/app/(app)/settings.tsx	"Platform Connections" section with paste-cookie modal per platform.
agent-setup/.github/workflows/	nightly-deps.yml, Dependabot config bumping all 14 packages.
agent-setup/.cursor/rules/mcp-tool-conventions.mdc	Naming, schema, description, response-shape conventions.
agent-setup/docs/	mcp-architecture.md, mcp-inventory.md (generated), credentials-setup.md.

---

Acceptance criteria

Architecture

• github.com/teslashibe/mcptool repo exists, tagged v0.1.0, deps = stdlib + invopop/jsonschema only
• All 14 in-scope packages have an mcp/ subpackage exposing mcptool.Provider
• Every mcp/ subpackage has a coverage test that fails if a *Client method is added without exposure or exclusion-list entry
• All public methods on every *Client are either wrapped or in excluded.go with a reason comment
• All scraper repos that needed it are public on GitHub (✅ done as part of pre-work)

Backend

• MCP server mounted at /mcp/v1 (streamable-HTTP transport) inside the existing Fiber app
• Per-user JWT auth on MCP requests; user_id resolved before tool dispatch
• platform_credentials migration applied; users table extended with cached agent IDs
• Credentials encrypted at rest via AES-GCM; CREDENTIALS_ENCRYPTION_KEY validated at boot
• CRUD endpoints POST/GET/DELETE /api/platforms/:platform/credentials work end-to-end
• GET /api/platforms returns list with per-platform connection status
• Per-user Anthropic agent + environment created lazily on first session, IDs cached on user row
• Response middleware enforces limit=10 default, max 50, string truncation at 800 chars, compact JSON
• Tool errors map to structured payloads (e.g. {"error": "credential_expired", "platform": "x", "reconnect_url": "/settings/platforms/x"})

Mobile

• Settings screen has "Platform Connections" section listing all 14 platforms
• Modal accepts both raw cookie strings and extension-exported JSON
• Disconnect button works; status updates without reload
• Errors from MCP tool calls (e.g. expired credential) surface a "Reconnect" CTA in the chat UI

Tooling & docs

• cmd/mcp-inventory generates docs/mcp-inventory.md listing every tool with its description, input schema, and source method
• docs/mcp-architecture.md explains the hybrid layout, dataflow, and credential model
• docs/credentials-setup.md explains how end-users connect each platform (with the cookie-extractor extension link)
• README updated with the MCP overview, link to inventory, and credential setup
• .cursor/rules/mcp-tool-conventions.mdc exists and is auto-loaded for *-go/mcp/** and backend/internal/mcp/**

CI/CD

• Dependabot config opens weekly PRs for all 14 scraper packages with auto-merge on green CI
• Nightly workflow runs go get -u ./... && go test ./... and posts an issue on breakage
• All package CI workflows include the coverage test in their default suite

Verification

• End-to-end: a fresh user signs up, connects LinkedIn via Settings, asks the agent "search LinkedIn for senior Rust engineers in Berlin", agent invokes linkedin_search_people, results render in chat
• End-to-end write: same user asks agent to "post a thread on X about our launch", agent invokes x_create_tweet, tweet appears (or returns a structured error if cookies expired)
• Negative path: revoked credential → tool returns credential_expired error → mobile UI surfaces "Reconnect X" prompt
• All 14 platforms have at least one happy-path integration smoke test (cassette-based or skipped without creds)

---

Rollout order

1. Foundation — mcptool repo + agent-setup/backend/internal/mcp skeleton + transport + auth + registry (no platforms wired yet)
2. Credentials — migration + store + encryption + CRUD + Settings UI
3. Per-user agents — lazy provisioning + caching
4. Platforms wave 1 — linkedin, x, facebook, tiktok, reddit (the most complete five)
5. Platforms wave 2 — threads, hn, nextdoor, producthunt, instagram
6. Platforms wave 3 — x-viral, redditviral, elevenlabs, codegen
7. Polish — inventory generator, docs, cursor rule, Dependabot, nightly workflow, end-to-end smoke tests

---

Out of scope for this issue

• A full credential-rotation / refresh-token flow (initial version is paste-cookie only; refresh-on-401 is a follow-up)
• A web-extension that pushes cookies directly into our API (manual paste only for v1)
• Tool-level usage analytics / billing dashboards
• A standalone MCP-server binary (everything lives in the Fiber app for v1)