Skip to content
/ LLMWebCrawler Public

A Web Crawler based on LLMs implemented with Ray and Huggingface. The embeddings are saved into a vector database for fast clustering and retrieval. Use it for your RAG.

95 stars 11 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

Aavache/LLMWebCrawler

Error
Looks like something went wrong!

About

A Web Crawler based on LLMs implemented with Ray and Huggingface. The embeddings are saved into a vector database for fast clustering and retrieval. Use it for your RAG.

Topics

python nlp api machine-learning raylib distributed-computing transformer ray webcrawler webcrawling rag pydantic fastapi huggingface milvus vector-database large-language-models llm

Resources

Readme
Activity

Stars

95 stars

Watchers

1 watching

Forks

11 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages