[Security][Critical] Spider endpoints completely unauthenticated — POST /spider/*

[bahdcoder]

Vulnerability

All Spider endpoints (/spider/crawl, /spider/search, /spider/links, /spider/screenshot, /spider/transform) have no authentication middleware applied. The route registration explicitly omits TokenAuth with a TODO comment acknowledging this is temporary.

Impact

• Any unauthenticated user can invoke Spider.cloud API calls through the server, using the platform's paid API quota without authorization.
• The server becomes an open proxy for web crawling, search, screenshots, and link extraction.
• Attackers can enumerate and crawl arbitrary URLs, take screenshots of internal services, and search for content at no cost to themselves.
• No rate limiting is applied to these routes, enabling high-volume abuse.

Endpoint

• POST /spider/crawl — parameter url is passed to Spider.cloud without auth
• POST /spider/search — parameter search is passed to Spider.cloud without auth
• POST /spider/links — parameter url is passed to Spider.cloud without auth
• POST /spider/screenshot — parameter url is passed to Spider.cloud without auth
• POST /spider/transform — parameter data is passed to Spider.cloud without auth

Recommendation

Apply TokenAuth middleware to the /spider route group. The TODO comment indicates this was planned but deferred. Add rate limiting to prevent abuse even for authenticated users.

[bahdcoder]

Audit note — verified against main at commit 12f7522

• Spider routes still registered without auth at cmd/server/serve_routes_admin.go:152-162:

  if deps.SpiderClient != nil {
      spiderHandler := handler.NewSpiderHandler(...)
      r.Route("/spider", func(r chi.Router) {
          r.Post("/crawl", spiderHandler.Crawl)
          r.Post("/search", spiderHandler.Search)
          r.Post("/links", spiderHandler.Links)
          r.Post("/screenshot", spiderHandler.Screenshot)
          r.Post("/transform", spiderHandler.Transform)
      })
      slog.Info("spider routes registered (NO AUTH - temporary)")
  }

• Still valid: no middleware.RequireAuth / TokenAuth applied; the log line explicitly acknowledges the unauthenticated state.