fix: PineconeIndex upsert request size limit

[siddicky]

Resolves #166

Overview

This pull request introduces a batch upsert functionality to the PineconeIndex class within the semantic-router project. Previously, attempting to upsert a large volume of data exceeding 3MB resulted in a PineconeApiException due to surpassing Pinecone's maximum request size limit.

Changes

• Added a _batch_upsert method to PineconeIndex for handling individual batches of upsert operations.
• Modified the add method to break down the data into manageable batches (default size of 100 records per batch) before upserting to avoid exceeding the 2MB request size limit.

Testing

• Ensured that existing unit tests pass.
• Added new tests to cover the batch upsert functionality.
• Manually tested with datasets exceeding the previous 3MB limit to confirm successful upsert without exceptions.

Usage

Users can now add large datasets without manual batching. The add method automatically handles data in batches, ensuring compatibility with Pinecone's size constraints.

Impact

This enhancement prevents PineconeApiException errors during large data upserts, improving the reliability and usability of the semantic-router's Pinecone integration for users dealing with substantial datasets.

[codecov]

Codecov Report

Attention: Patch coverage is 12.50000% with 7 lines in your changes are missing coverage. Please review.

Project coverage is 78.34%. Comparing base (01eb8d7) to head (cfa13de).
Report is 4 commits behind head on main.

Files	Patch %	Lines
semantic_router/index/pinecone.py	12.50%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #165      +/-   ##
==========================================
+ Coverage   78.33%   78.34%   +0.01%     
==========================================
  Files          41       41              
  Lines        1943     1944       +1     
==========================================
+ Hits         1522     1523       +1     
  Misses        421      421              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jamescalam]

lgtm, thanks!