community: Batching added in embed_documents of HuggingFaceInferenceAPIEmbeddings

[abhishek9998]

• Description: Batching added in embed_documents of HuggingFaceInferenceAPIEmbeddings
• Issue: HuggingFaceInferenceAPIEmbeddings getting 413 request code because of not batching mechanism like SentenceTransformer #16443
• Dependencies: tqdm
• Twitter handle: @Abhishingadiya

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchain	⬜️ Ignored (Inspect)	Visit Preview		May 8, 2024 7:54pm

[hwchase17]

where is this used?

[abhishek9998]

Example Code

embeddings = HuggingFaceInferenceAPIEmbeddings(
    api_key=inference_api_key, 
    api_url=api_url,
    model_name="bge-large-en-v1.5"
)
pinecone.init(api_key=os.getenv("PINECONE_API_KEY"), environment=environment)

loader = PyPDFDirectoryLoader("data")
docs = loader.load()
text_splitter = RecursiveCharacterTextSplitter(chunk_size=2000, chunk_overlap=200)
chunks = text_splitter.split_documents(docs)

vectordb = Pinecone.from_documents(chunks, embeddings, index_name=index_name, namespace=namespace)

this code snippet is getting 314 request code from huggingface.py

response = requests.post(
            self._api_url,
            headers=self._headers,
            json={
                "inputs": texts,
                "options": {"wait_for_model": True, "use_cache": True},
            },
        )
        return response.json()

we should support batch size here. like local model embedding

Description

I am trying to use pinecone with hugging face inference as a embedding model. My total chunks are 420. and it is trying to process in one request.
Also embedding_chunk_size is not parsable from Pinecone.from_documents() method

[abhishek9998]

Any Update @baskaryan, @hwchase17 ?
PS: I am using custom branch of langchain for my prod due to HuggingFaceInferenceAPIEmbeddings is not supporting batching mechanism.
same issue: https://discuss.huggingface.co/t/batch-size-limit-32/82471