Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ENH] Multimodal Embeddings #1293

Merged
merged 21 commits into from
Nov 7, 2023
Merged

[ENH] Multimodal Embeddings #1293

merged 21 commits into from
Nov 7, 2023

Conversation

atroyn
Copy link
Contributor

@atroyn atroyn commented Oct 26, 2023 •
edited
Loading

Description of changes

This PR introduces multi-modal embeddings into Chroma.

  • It adds the generic EmbeddingFunction which can take various data types. Existing functions take the Documents type.
  • Adds Images as a type (numpy NDArray taking ints or floats)
  • Add OpenCLIPEmbeddingFunction which is an EmbeddingFunction[Union[Documents, Images]]

Test

Integration tests pass.

A new test for multimodal embedding functions: chromadb/test/ef/test_multimodal_ef.py

Documentation

See #1294

TODOs

  • Tests
  • Wiring through FastAPI Nothing to wire through
  • Documentation
  • Telemetry
  • JavaScript

@github-actions
Copy link

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

  • Can you think of any use case in which the code does not behave as intended? Have they been tested?
  • Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
  • If appropriate, are there adequate property based tests?
  • If appropriate, are there adequate unit tests?
  • Should any logging, debugging, tracing information be added or removed?
  • Are error messages user-friendly?
  • Have all documentation changes needed been made?
  • Have all non-obvious changes been commented?

System Compatibility

  • Are there any potential impacts on other parts of the system or backward compatibility?
  • Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

  • Is this code of a unexpectedly high quality (Readbility, Modularity, Intuitiveness)

@atroyn atroyn mentioned this pull request Oct 26, 2023
Merged
4 tasks
atroyn
atroyn commented Oct 26, 2023
View reviewed changes
chromadb/api/types.py Show resolved Hide resolved
chromadb/api/types.py Show resolved Hide resolved
chromadb/api/types.py Outdated Show resolved Hide resolved
chromadb/api/types.py Outdated Show resolved Hide resolved
chromadb/api/types.py Show resolved Hide resolved
chromadb/api/models/Collection.py Outdated Show resolved Hide resolved
chromadb/api/models/Collection.py Show resolved Hide resolved
chromadb/api/models/Collection.py Outdated Show resolved Hide resolved
chromadb/api/models/Collection.py Show resolved Hide resolved
@atroyn atroyn changed the title [DRAFT] Multimodal Embeddings [ENH] Multimodal Embeddings Nov 6, 2023
@atroyn atroyn marked this pull request as ready for review November 6, 2023 03:14
@rlancemartin rlancemartin mentioned this pull request Nov 6, 2023
Merged
@atroyn atroyn mentioned this pull request Nov 7, 2023
Open
@atroyn atroyn merged commit 4db9955 into main Nov 7, 2023
95 checks passed
@atroyn atroyn deleted the anton/multimodal-ef branch November 7, 2023 17:57
atroyn added a commit that referenced this pull request Nov 7, 2023
@atroyn
Revert "[ENH] Multimodal Embeddings (#1293)"
2607adb 
This reverts commit 4db9955.
@atroyn atroyn mentioned this pull request Nov 7, 2023
Merged
atroyn added a commit that referenced this pull request Nov 7, 2023
@atroyn
Revert "[ENH] Multimodal Embeddings" (#1344)
69cd2ba 
Reverts #1293
@atroyn atroyn restored the anton/multimodal-ef branch November 7, 2023 18:00
atroyn added a commit that referenced this pull request Nov 7, 2023
@atroyn
[ENH] URI Data Loader (#1294)
8d0405e 
## Description of changes

This PR adds URIs and DataLoaders into Chroma. 
- `DataLoader` works like `EmbeddingFunction`, except it takes a `URIs`
and outputs the specified datatype.
- A `DataLoader` using `pillow` for image file loading. 
- Adds `uris` as a field on `add`, `query`, as well as an `include`
field
- Adds `data` as an `include` field 

URIs specify a place where data can be loaded from, and can be used to
load data for embedding, or as the result of retrieval.

This makes multimodal retrieval with data stored externally as files
seamless and extensible.

This PR is stacked on #1293

## Test

Integration tests pass. 

A new unit test for data loaders:
https://github.com/chroma-core/chroma/blob/c71c3efa15d2a9252db26470b9ff1cdb10a2b681/chromadb/test/data_loader/test_data_loader.py

Try the notebook:
https://github.com/chroma-core/chroma/blob/c71c3efa15d2a9252db26470b9ff1cdb10a2b681/examples/multimodal/multimodal_retrieval.ipynb

## Documentation

Documentation for this and
#1293
chroma-core/docs#157

## TODOs
- [x] Concurrent Loading
- [x] Tests
- [x] Wiring through FastAPI
- [x] Documentation
baskaryan added a commit to langchain-ai/langchain that referenced this pull request Nov 10, 2023
@rlancemartin @efriis @baskaryan
Add Chroma multimodal cookbook (#12952)
d2e50b3 
Pending:
* chroma-core/chroma#1294
* chroma-core/chroma#1293

---------

Co-authored-by: Erick Friis <erick@langchain.dev>
Co-authored-by: Bagatur <baskaryan@gmail.com>
pprados pushed a commit to pprados/langchain that referenced this pull request Nov 20, 2023
@rlancemartin @efriis
@baskaryan @pprados
Add Chroma multimodal cookbook (langchain-ai#12952)
8c0f7f5 
Pending:
* chroma-core/chroma#1294
* chroma-core/chroma#1293

---------

Co-authored-by: Erick Friis <erick@langchain.dev>
Co-authored-by: Bagatur <baskaryan@gmail.com>
xieqihui pushed a commit to xieqihui/langchain that referenced this pull request Nov 21, 2023
@rlancemartin @efriis
@baskaryan @xieqihui
Add Chroma multimodal cookbook (langchain-ai#12952)
254ded2 
Pending:
* chroma-core/chroma#1294
* chroma-core/chroma#1293

---------

Co-authored-by: Erick Friis <erick@langchain.dev>
Co-authored-by: Bagatur <baskaryan@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@HammadB HammadB HammadB approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@atroyn @HammadB