Repository navigation
Spider as a web search scraper provider #18018
j-mendez
started this conversation in
Feature Requests & Suggestions
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
What features would you like to see added?
A
spidervalue forwebSearch.scraperProvider, configured withSPIDER_API_KEY(and an optionalSPIDER_API_URL), alongside the current scraper providers.More details
The scraper would call
POST https://api.spider.cloud/scrapewith{"url": ..., "return_format": "markdown"}and readcontentfrom the first item of the JSON array it returns. The page's own HTTP status comes back instatus, so a 404 page can be dropped the same way the other scrapers drop failures. No new npm dependency is needed.Proposed changes, as two PRs:
SPIDER = 'spider'toScraperProviders, thespiderApiKeyandspiderApiUrlkeys inpackages/data-schemas, resolution inpackages/api/src/web/, and thelibrechat.example.yamlentry. Docs go to librechat.ai.I would like to take this. I can start once it is approved and assigned.
API reference: https://spider.cloud/docs/api?utm_source=librechat&utm_medium=directory&utm_campaign=librechat-listing
Which components are impacted? Endpoints / Tools (web search)
All reactions