Bing Webmaster Tools integration - #2
Merged
Merged
Conversation
Adds specs/0009 recording the Bing Webmaster Tools integration design, verified against the live API rather than the docs. Two unknowns blocked the design and were settled by scripts/bing-oauth-spike.ts: - Refresh tokens: public reports describe Bing rotating refresh tokens and then rejecting the rotated ones, which would be fatal here since Better Auth overwrites the stored token with whatever a provider returns. Not reproduced — Bing returns no refresh_token on refresh at all, so the original is preserved and survives reuse. genericOAuth is safe. - Account identity: Bing has no userinfo endpoint and issues no id_token, but the access token is base64url JSON carrying webmasteruid and webmasteremail. src/shared/bing.ts decodes it, so getUserInfo needs no network call. The spike script is retained: the refresh finding rests on a short observation window and specs/0009 calls for re-checking it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Round 1 of specs/0009: the three independent layers, built in parallel isolated worktrees and integrated here. - bing_connections across both dialects (SQLite/D1 + Postgres) with generated migrations, mirroring gsc_connections. - createBingClient: mints tokens via Better Auth, unwraps Bing's WCF `d` envelope, parses /Date(ms)/ values, and maps 401/403/404/429 to typed errors. Reads are free — no credit metering, same as GSC. - bingProviderConfig for genericOAuth with explicit endpoints (Bing has no discovery document) and a getUserInfo that decodes the access token instead of calling a userinfo endpoint that does not exist. - Encrypted-at-rest API key helpers for self-hosters, who cannot complete an OAuth flow locally because Bing rejects localhost redirect URIs. Two integration points were kept out of worker ownership and wired by hand: registering the provider in auth-config, and exporting the new table from the D1 barrel that drizzle-kit and the D1 client both read. GetRankAndTrafficStats rows are surfaced as-is: only GetUserSites was verified against the live API, so its field names are not yet pinned. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Round 2a of specs/0009. Mirrors the GSC layering so the two integrations read the same way. BingConnectionRepository upserts on project_id, so re-selecting a site replaces the mapping rather than adding a second, and coalesces connected_account_email so a null never clobbers a stored value. BingService gates site selection on Bing's isVerified boolean (Bing has no permissionLevel string), and keeps GSC's disconnect semantics: the OAuth grant is unlinked only when the disconnector is the connector and no other project still uses that account. A dead grant surfaces as requiresReconnect rather than throwing, so one revoked connection cannot break the whole site list. Deliberately out of scope, both flagged rather than stubbed: API-key connections (the client is OAuth-only, so those rows raise instead of silently failing) and any reshaping of GetRankAndTrafficStats rows, whose field names are still unverified against the live API. Reads stay free — no credit metering. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
BingService wrote connectedAccountEmail: null because the client had no way to reach it — Bing publishes no userinfo endpoint, which is what GSC calls for the equivalent. But the email is already a claim on the access token, so createBingClient.getConnectedEmail() decodes it with no extra network call. setSite and the grant listing now surface it, both best-effort: a missing or undecodable claim yields null rather than failing the connection or hiding the whole account list. Round 3's UI needs this for the "connected by" line. Splits BingService.test.ts, which crossed the 400-line lint cap once these cases were added; disconnect and grant-failure coverage now lives alongside it in BingService.disconnect.test.ts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Round 2b of specs/0009 — the layer that makes the client reachable. Server functions mirror gsc.ts: grant status, connection status, site listing, select, and disconnect, all project-scoped through requireProjectContext. Site selection gates on Bing's isVerified boolean rather than GSC's permissionLevel string. The MCP tool derives its table columns from the keys actually present on the returned rows instead of hard-coding them, because Bing's GetRankAndTrafficStats field names are still unverified — inventing Clicks/Impressions would produce a table that silently renders nothing. A missing connection or a revoked grant returns an actionable connect or reconnect message rather than surfacing as a fault, matching how the Search Console tools behave. Reads stay free — no credit metering. Registration in mcp/server.ts is wired here rather than by the worker, so no lane could collide on the shared tool registry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Live call against a verified site pins what GetRankAndTrafficStats actually
returns: one row per day carrying Date, Clicks, and Impressions, plus the
__type marker every WCF payload has. The date arrives as
/Date(1781852400000-0700)/ — the offset is informational, the milliseconds
are already UTC, which the existing parser handles.
Rows are now typed and mapped to { date, clicks, impressions } instead of
passed through as Record<string, unknown>. The MCP tool needs no change: it
derives columns from the keys present, which are now meaningful names rather
than Bing's PascalCase.
Adds a --step=call probe to the spike script, which is how the shape was
captured and how the next endpoint's shape should be.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The refresh behaviour now rests on two independent grants rather than one ten-minute window: no refresh_token returned on refresh, original reusable, observed before and after regenerating the OAuth client secret. The multi-day expiry complaint in the public reports remains untested and is called out as such. webmasteruid also survived the credential regeneration, confirming it keys the Bing account rather than the grant or the OAuth client. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Round 3 of specs/0009 — the user-facing surface.
Bing gets its own page rather than a source toggle on Search Performance.
GetRankAndTrafficStats accepts no date range, no device or country filter,
and no paging, so sharing that page's chrome would advertise controls Bing
cannot honour. The page shows clicks/impressions totals and the daily rows,
and falls back to the connection card when the project isn't connected.
Adds getBingPerformance, which round 2b deliberately left out: not-connected
and dead-grant both resolve to { connected: false } so the page renders the
connect card instead of an error boundary.
The connect card mirrors the Search Console one, with two differences that
follow from Bing's API: sites are gated on isVerified rather than a
permission-level string, and an unconfigured deployment gets an explanatory
setup notice instead of a button, because Bing rejects localhost redirect
URIs and allows one redirect URI per client.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both of these passed while proving less than they claimed. schema-parity.test.ts never imported the bing schemas, so the D1/Postgres mirror guard silently skipped the new table — the 120 green cases covered only pre-existing schemas. Verified the fix bites by deleting a column from the Postgres mirror and confirming a precise failure. The getConnectedEmail suite sat outside the describe that owns the fetch stub, so "makes no network call" could not detect a network call: an added fetch would hit the real global and leave the mock uncalled. Moved it into the stubbed scope and verified by sneaking a fetch into the client, which now fails the test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…secret Typing the rank/traffic rows against the live API left stale claims behind in four places. The MCP tool still derived table columns dynamically to absorb variability that no longer exists, its model-facing description said the row shape "varies by site", and its empty state told callers to try a different date range — for an endpoint that accepts no date range at all. Columns are now fixed like every other MCP tool, and two tests that asserted the dynamic behaviour are replaced by ones asserting the real shape, including that an unparseable date renders as unknown rather than invented. Also stops logging grant.accountId on an unexpected fault. That value is the webmasteruid, which doubles as Bing's site verification code, so the inherited GSC logging line put a secret into Worker logs and Sentry — a lower-trust store than the database. Logs the Better Auth row id instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…code Three reviewers flagged the API-key lane as scaffolding. Verification agreed in part: the two crypto helpers had no caller and no column to write to, and were two one-line wrappers over better-auth's symmetricEncrypt that will be rewritten when the real storage lands — so they go. The auth_mode column stays. Removing it means regenerating the head migration and snapshot in both dialects, then adding a third migration to restore it when the lane lands; a NOT NULL discriminator is the right shape for a two-mode connection. The getPerformance guard stays too — authMode is read back from the database as a union, and refusing an unhandled member at a trust boundary beats silently building an OAuth client for a non-OAuth row. The real defect was the spec claiming API-key mode "is also supported" with the key "stored encrypted at rest in a dedicated column" — no such column exists. It now says deferred, and three other overclaims are corrected: the surface described top queries, top pages and crawl issues that v1 does not build; the MCP section promised a crawl-issues tool that does not exist; and the Context still called the refresh finding ten minutes of observation after Consequences recorded the replication. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reviewers flagged two duplications between the Bing and Search Console cards. Verification rejected the larger one and confirmed the smaller. REJECTED — merging BingSitePicker into SitePicker. The claimed nine-line diff is really 19 hunks, and "(no access)" vs "(not verified)" mean opposite things to a user: one is a permission problem, the other an action to take in Bing Webmaster Tools. A merged component needs a glyph plus ten label strings, which reads worse than two clear files. SitePicker is also consumed by shipped GSC onboarding, so refactoring it to land a Bing feature risks an existing activation flow for no user-visible benefit. APPLIED — the card chrome. StatusPill was byte-identical, IntegrationCard differed by its title and ConnectedState by a glyph and one label: three props, no configuration bag. Both cards lose ~107 lines. The card bodies stay separate, since their connect flows, setup warnings, and invalidation sets genuinely differ — Bing has no dashboard cache to invalidate because the card only appears in project settings. Also removes getBingGrantStatus and the GRANT_STATUS_KEY invalidations. No query ever registered that key, so every invalidation of it was a no-op; the GSC equivalent exists because onboarding and the re-engagement modal read it, and Bing has neither. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Clicking Connect on a cloudflare_access deployment failed with "Could not
start Bing sign-in". Better Auth's oauth2.link needs a Better Auth session,
which cloudflare_access and local_noauth do not have — Search Console solves
this with a hand-rolled flow and Bing had no equivalent, so the card offered
a button that could not work in the mode this repo defaults to.
Adds the Bing counterpart of gsc/selfHostedOAuth.ts: HMAC-signed state, code
exchange, and an encrypted account row written exactly the way Better Auth's
setTokenUtil would, so getAccessToken reads it back unchanged. The one
Bing-specific difference is identity — no id_token and no userinfo endpoint,
so the account id comes from the access token's webmasteruid claim.
The redirect URI is {origin}/api/bing/oauth/callback. Bing permits one per
registered client and rejects localhost, so each deployment needs its own
client and a localhost dev server still cannot complete the flow.
Five review axes and two verification passes missed this because all of them
read code; it only surfaced when the button was actually clicked.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ationale Verified end to end on a cloudflare_access deployment: connect, site selection, and daily rows all work. The spec justified API-key mode with "a self-hoster cannot complete an OAuth flow" — that is now only true of localhost, since Bing refuses localhost redirect URIs but is perfectly happy with a self-hosted deployment on a real domain. The remaining use for the API-key lane is local development without a public tunnel. The setup notice now names BETTER_AUTH_SECRET, which gates the flow and whose absence shows up as "Setup required" with no explanation, and gives the exact callback path to register. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This handles token exchange, at-rest encryption, signed state, and an open-redirect guard, and had no coverage — the Search Console original it mirrors has none either, so this is new ground rather than parity. Eleven cases, built around driving the real authorize step to obtain a genuinely signed state so the callback tests exercise verification rather than a hand-made string. Covers: tampered state rejected, state issued for another user refused, an off-origin callbackURL collapsing to "/" instead of becoming an open redirect, denied consent returning without exchanging anything, the grant being keyed by webmasteruid, a stored refresh token surviving a re-link that returns none (Bing never returns one), and both failure paths — a rejected code exchange and an unreadable access token — storing nothing. The two security assertions were mutation-tested: removing the HMAC check and the same-origin guard each fails exactly the test that covers it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Live MCP output showed the offset moving between 08:00Z and 07:00Z across March — these are midnight in Bing's own reporting timezone (US Pacific, shifting with daylight saving), not instants. Formatting them in the viewer's timezone labels every row a day early for anyone at UTC-9 or further west; Honolulu and Anchorage render a 9 Feb bucket as 8 Feb. The helper moves to its own module so it can be tested without dragging the page's server-function imports into the test's module graph, and reads the day in UTC. Mutation-tested: dropping the UTC pin fails the tests under TZ=America/Anchorage. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Probed live so the next person does not have to. Both endpoints share one row shape carrying AvgImpressionPosition, so average position and a striking-distance view are reachable — but the rows are sampled at roughly 16 dates across five months, which makes query-level trends lumpy and a 28-day slice potentially empty. Also records that the traffic window varies by site and that its Date is a Pacific day bucket rather than an instant. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Issues are disabled on the fork, so the follow-up lands where specs/0008 already puts this kind of note. Captures what the probed query/page endpoints make possible — striking distance first — and what still cannot be built honestly: no device or country dimension exists, and sampled query rows make a date-range control promise precision the data lacks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a Bing Webmaster Tools connection alongside the existing Search Console one: OAuth connect, per-project site mapping, a daily clicks/impressions page, and a read-only MCP tool. Bing is free and first-party, so like GSC it consumes no credits.
Verified end to end on a
cloudflare_accessdeployment — connect, site selection, and daily rows all work against a real Bing account.The design and the reasoning behind it are in
specs/0009-bing-webmaster-integration.md.Why it isn't just "GSC with a different logo"
Bing's API is not a second Search Console. GSC has one endpoint taking arbitrary date ranges, dimensions, filters and paging. Bing has ~15 fixed-shape methods with no date range, no dimensions, and no paging. That single fact drove most of the design decisions — most visibly, Bing gets its own page rather than a source toggle on Search Performance, because sharing that page's chrome would advertise controls Bing cannot honour.
Two questions were settled by probing the live API rather than trusting the docs, because Microsoft's own documentation is wrong in places and public bug reports suggested the integration might not be viable at all:
refresh_tokenon refresh at all, replicated across two independent grants before and after regenerating the client secret. What remains untested is the multi-day complaint in those same reports.id_token, sogetUserInfodecodeswebmasteruidandwebmasteremailfrom the access token itself — no network call. That id survived a client-secret regeneration, so it keys the Bing account rather than the grant.scripts/bing-oauth-spike.tsis the probe, retained deliberately: the spec asks for the refresh behaviour to be re-checked periodically, and its--step=callis how any further Bing endpoint's shape should be pinned before coding against it.How to review
src/server/lib/bingClient.ts— the boundary where Bing's oddities are absorbed: a WCFdenvelope on every response,/Date(ms±HHMM)/timestamps, PascalCase fields, and a scope string that comes back as"Read"whenwebmaster.readwas requested.BingService— the judgment calls live here: disconnect only unlinks a grant when no other project still uses it, and a dead grant surfaces as "reconnect" rather than breaking the whole site list.bing/selfHostedOAuth.tsdeserves real scrutiny. It mirrors the Search Console equivalent (HMAC-signed state, code exchange, anaccountrow encrypted exactly assetTokenUtilwrites it) and it has been exercised end to end, but it has no automated test coverage and it handles tokens.SearchConsoleConnectionCard— shipped code this feature otherwise doesn't go near. Mechanical (three props) and typechecked, but it is the one change that could regress something already working.Deliberately out of scope, recorded in the spec: API-key connections (now only needed for local dev, since Bing refuses
localhostredirect URIs), URL submission, crawl issues, and Bing keyword data.Setup for another environment
Bing permits one redirect URI per registered OAuth client, so each environment needs its own client, registered against
{origin}/api/bing/oauth/callback. RequiresBING_CLIENT_ID,BING_CLIENT_SECRET, and aBETTER_AUTH_SECRETof 32+ characters — the card shows "Setup required" until all three are present.Review notes
Findings from a five-axis review that were rejected after verification, flagged in case you disagree:
BingSitePickerinto the GSCSitePicker. Rejected: the claimed nine-line diff is really 19 hunks,"(no access)"and"(not verified)"mean opposite things to a user, and a merged component needs ten label props.SitePickeris also consumed by shipped onboarding.auth_modecolumn as speculative. Rejected: removing it means regenerating head migrations in both dialects, then a third migration to restore it.Unfixed nitpicks left for your judgment:
listSites()andgetConnectedEmail()each mint a token, so one site-list render mints twice per grant. Token endpoint only, no Bing quota impact.listBingSitesreturnsaccountId(thewebmasteruid) to the browser. It's only ever the requester's own grant, but that value doubles as Bing's site verification code.BingConnectionRepositorywritesupdatedAtwith SQLite-shapedcurrent_timestamp, which is not ISO under Postgres. Copied verbatim fromGscConnectionRepository, so it's a pre-existing repo bug replicated rather than introduced — worth fixing in both places or neither.BingServicetest files duplicate a ~90-line mock harness. Sharing it is awkward becausevi.mockmust be called in the file it applies to; the split was forced by the 400-line lint cap.Three checks on this branch were found to prove less than they claimed, each fixed and each fix verified by deliberately breaking the thing it guards: the schema parity test never imported the Bing schemas; a "makes no network call" assertion sat outside the describe that stubs
fetch; and a test file was silently failing to collect after a new import reachedenvfromcloudflare:workers.Worth noting for calibration: the review found real defects, but the one bug that stopped the feature working — no self-hosted OAuth path, so Connect failed in the auth mode this repo defaults to — was found by clicking the button, not by any of the five axes or two verification passes, all of which only read code.
🤖 Generated with Claude Code