[BLD] Auto rebuild protos if they change in rust #1902
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This stack of pull requests is managed by Graphite. Learn more about stacking. |
Reviewer ChecklistPlease leverage this checklist to ensure your code review is thorough before approving Testing, Bugs, Errors, Logs, Documentation
System Compatibility
Quality
|
HammadB
commented
Mar 20, 2024
| &["../../idl/"], | ||
| )?; | ||
| tonic_build::configure() | ||
| .emit_rerun_if_changed(true) |
There was a problem hiding this comment.
this is the change - .emit_rerun_if_changed(true)
HammadB
force-pushed
the
hammad/proto_auto_build
branch
from
54d2113
to
4bb06ef
Compare
1 task
HammadB
force-pushed
the
hammad/query_server
branch
from
76ed958
to
5c8f1c3
Compare
HammadB
force-pushed
the
hammad/proto_auto_build
branch
from
4bb06ef
to
15aaebb
Compare
1 task
HammadB
force-pushed
the
hammad/proto_auto_build
branch
from
15aaebb
to
3869c4a
Compare
atroyn
added a commit
to csbasil/chroma
that referenced
this pull request
Apr 3, 2024
commit 193988d268f472f792fe27bfa3c0fb92b333791a
Author: Mihir1003 <32996071+Mihir1003@users.noreply.github.com>
Date: Wed Apr 3 04:43:12 2024 +0530
[ENH] Support langchain embedding functions with chroma (#1880)
## Description of changes
*Summarize the changes made by this PR.*
- New functionality
- Adding a function to create a chroma langchain embedding interface.
This interface acts as a bridge between the langchain embedding function
and the chroma custom embedding function.
- Native Langchain multimodal support: The PR adds a Passthrough data
loader that lets langchain users use OpenClip and other multi-modal
embedding functions from langchain with chroma without having to handle
storing images themselves.
## Test plan
*How are these changes tested?*
- installing chroma as an editable package locally and passing langchain
integration tests
- pytest test_api.py test_client.py succeeds
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
Co-authored-by: Anton Troynikov <atroyn@users.noreply.github.com>
commit fdfda56d2e83fea5df1343ef3df0d43528bd27a8
Author: Liquan Pei <liquanpei@gmail.com>
Date: Tue Apr 2 10:02:41 2024 -0700
[CLN] Clean up unused imports (#1959)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- ...
- New functionality
- This PR cleans up unused import and addresses some warnings by cargo
build and cargo test.
- Unused variables are not touched.
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit d3f61b145eeb0f4108f1b1a1b5919a36c0e3b8b6
Author: James <37276661+capjamesg@users.noreply.github.com>
Date: Tue Apr 2 04:08:43 2024 +0100
[ENH] Create `RoboflowEmbeddingFunction` (#1434)
## Description of changes
This PR adds a new `RoboflowEmbeddingFunction` with which a user can
calculate CLIP text and image embeddings using [Roboflow
Inference](https://inference.roboflow.com).
## Test plan
You can test the embedding function using the following code:
```python
import chromadb
import os
from chromadb.utils.embedding_functions import RoboflowEmbeddingFunction
import uuid
from PIL import Image
client = chromadb.PersistentClient(path="database")
collection = client.create_collection(name="images", metadata={"hnsw:space": "cosine"})
# collection = client.get_collection(name="images")
IMAGE_DIR = "images/train/images/"
SERVER_URL = "https://infer.roboflow.com"
API_KEY = ""
results = []
ef = RoboflowEmbeddingFunction(API_KEY)
documents = [os.path.join(IMAGE_DIR, img) for img in os.listdir(IMAGE_DIR)]
embeddings = ef(images = [img for img in documents])
ids = [str(uuid.uuid4()) for _ in range(len(documents))]
print(len(embeddings))
collection.add(
embeddings=embeddings,
documents=documents,
ids=ids,
)
query = ef(prompt = "baseball")
results = collection.query(
query_embeddings=query,
n_results=3
)
top_result = results["documents"]
for i in top_result:
print(i)
```
You will need a [Roboflow API
key](https://docs.roboflow.com/api-reference/authentication#retrieve-an-api-key)
- [X] Tests pass locally with `pytest` for python, `yarn test` for js
## Documentation Changes
I will file a PR to the `chroma-core/docs` repository with
documentation.
---------
Co-authored-by: Anton Troynikov <atroyn@users.noreply.github.com>
commit 1cd2ced9577369171c6a19ffa9e1a5de5b23ace1
Author: Liquan Pei <liquanpei@gmail.com>
Date: Mon Apr 1 16:47:51 2024 -0700
[ENH] Add memberlist for compaction service (#1949)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- ...
- New functionality
- This PR adds memberlist consumption for the compaction service and
filter collections based on memberlist using Rendezvous assignment
policy.
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
- [ ] Manual local integration testing with Tilt
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit 978547c32d52da81b6991827a4808144fc1166ea
Author: Liquan Pei <liquanpei@gmail.com>
Date: Mon Apr 1 12:35:04 2024 -0700
[BUG] Fix memberlist manager to only list memberlist with the provide… (#1956)
…d memberlist name
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- This PR fixes memberlist manager to only list memberlist with the
provided memberlist name.
- New functionality
- ...
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit ab05830948d6d459ac4b39c943e5ff81a9b5aced
Author: Liquan Pei <liquanpei@gmail.com>
Date: Mon Apr 1 11:37:49 2024 -0700
[CLN] Change Task to CompactionJob for higher level compaction work d… (#1955)
…escription
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- This PR changes Task to CompactionJob for higher level compaction work
description for better code readability.
- New functionality
- ...
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit d8c1055d6f6709bc7fa2e100c553b50d162cecf9
Author: Liquan Pei <liquanpei@gmail.com>
Date: Mon Apr 1 10:25:15 2024 -0700
[CLN] Remove unused worker_entrypoint (#1954)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- This PR cleans up unused worker_entrypoint
- New functionality
- ...
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit b6e86a60a3484314c71c7612768830ffd58423e1
Author: Liquan Pei <liquanpei@gmail.com>
Date: Fri Mar 29 11:16:47 2024 -0700
[ENH] Compaction service setup (#1945)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- ...
- New functionality
- This PR adds the necessary artifacts to setup compaction service.
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit 6af795b059b9d606c6d96d2de20d1f1af1b774fe
Author: Liquan Pei <liquanpei@gmail.com>
Date: Fri Mar 29 07:45:18 2024 -0700
[ENH] Compaction manager (#1944)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- ...
- New functionality
- This PR adds a compaction manager that gets a list of collections to
compact and orchestrate the execution of compaction tasks.
- It uses FuturesUnordered to await the compaction tasks and allow them
to run in parallel.
- We will add retry and timeout in follow up PRs.
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit 2c416aa655f4049ce41eea6bc1a12b77ed676f99
Author: Liquan Pei <liquanpei@gmail.com>
Date: Thu Mar 28 23:09:22 2024 -0700
[ENH] Add compaction service memberlist propagation (#1946)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- ...
- New functionality
- This PR adds memberlist manager for compaction service.
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit 99f05494f2c261028e1cf1f6f8741a7670a1b04c
Author: Ben Eggers <64657842+beggers@users.noreply.github.com>
Date: Thu Mar 28 17:04:08 2024 -0700
[CLN] Create public if not exists (#1943)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Usually local postgres has the schema already, but staging currently
doesn't. We should just create if not exists.
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit a78aedee895dd813ed7f8a372788346bd92f2f57
Author: Ben Eggers <64657842+beggers@users.noreply.github.com>
Date: Thu Mar 28 14:43:15 2024 -0700
[CLN] Stop clearing the sysdb on each deploy (#1941)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- ...
- New functionality
- ...
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit 4da1a6ccdbc6f249b3d1429af1320cf22b240892
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Thu Mar 28 14:22:20 2024 -0700
[ENH] Remove seq_id from protos, record and result types. Refactor rust to use LogRecord / OperationRecord (#1935)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Remove the seq_id concept from our record and result types. This is
unused and not needed. The python segments still internally use this
though.
- Refactor rust to use LogRecord/OperationRecord
- Remove Segment Manager and Ingestor from rust, as well as other pulsar
based paraphenelia such as message_id. This resulted in the /ingest
module being removed altogether.
- New functionality
- None
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit 1d77f99edc43ade4d3490a4638b5e5d802050303
Author: Trayan Azarov <tazarov@users.noreply.github.com>
Date: Thu Mar 28 19:59:01 2024 +0200
[ENH]: Ollama embedding function (#1813)
## Description of changes
*Summarize the changes made by this PR.*
- New functionality
- New Ollama embedding function (Python and JS)
- Example of how to run Ollama with the embedding function
## Test plan
*How are these changes tested?*
- [x] Tests pass locally with `pytest` for python, `yarn test` for js
## Documentation Changes
chroma-core/docs#222
commit 5f3f141c55103ae949f726ccd88d1568e592dc69
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Wed Mar 27 17:51:57 2024 -0700
[CLN] Restructure EmbeddingRecord -> LogRecord. Use 'log_offset' as term instead of 'id' (#1934)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- This PR follows #1933 and restructures EmbeddingRecord to be a nested
type around OperationRecord. We rename EmbeddingRecord to be LogRecord.
A LogRecord is a model of an OperationRecord stored on a log, and just
keeps a log_offset The log is assumed to be per-collection.
- Rename the Logservices use of "ID" to instead be "log_offset" to
standardize on the "log_offset" terminology.
- Rename RecordLog -> LogRecord for conceptual clarity and to align with
other types.
- Fixes a proto style guide violation where we used camelcase for
fieldnames -
https://protobuf.dev/programming-guides/style/#message_and_field_names.
- In GetColllectionCompactionInfo rename first_log_id to
first_log_offset - which is much clearer. Also rename first_log_id_ts to
first_log_offset. What is a id_ts? Confusing name.
- New functionality
- None
## Test plan
*How are these changes tested?*
- [x] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
None
commit 1ce93c7e41344446dc18af7b27474b5d5bcd4a68
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Wed Mar 27 10:48:48 2024 -0700
[CLN] Rename SubmitEmbeddingRecord to OperationRecord. Remove Topic concept from Segment/Collection. (#1933)
## Description of changes
This PR has two high-level aims
- Clean up artifacts of our design that were implicitly coupled to
pulsar. Namely topics and multiplexing.
- Begin a rename of SubmitEmbeddingRecord, EmbeddingRecord to names that
more accurately reflect their intents.
I apologize for how large this PR is, but in practice, breaking
something up like this is not really feasible AFAICT, unless we allow
test-breaking stacked PRs...
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Renames SubmitEmbeddingRecord to OperationRecord in order to more
correctly identify what it is - a record of an Operation (future PRs
will rename EmbeddingRecord as well to LogRecord to make its intent
clearer).
- An OperationRecord does not need to store collection_id. This was an
artifact of the pulsar log when we needed to demux data. We now improve
the Ingest interface by presenting producers/consumers over logical log
streams by collection.
- Remove the concept of topic from the Producer/Consumer interfaces - it
is no longer needed in a post pulsar-world. This also means
Collection/Segment don't need to store Topic.
- Removed the AssignmentPolicy concept. This only existed for
multiplexing - which is not a concept without pulsar.
- Update the Rust code with the topic field removed and with the
OperationRecord naming.
- Update Go code with the SysDB changes (No assignment policy + no log)
no as well as the OperationRecord naming.
- New functionality
- None
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit 739e942b165dd7fae7888ea34656d79e584e045f
Author: Liquan Pei <liquanpei@gmail.com>
Date: Tue Mar 26 17:01:24 2024 -0700
[ENH] DataChunk abstraction with PullLog, Group and BF (#1929)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- ...
- New functionality
- This PR adds DataChunk abstraction
- PullLog, Group and BF are reimplemented with DataChunk
- Compaction orchestrator is implemented with DataChunk
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
---------
Co-authored-by: Hammad Bashir <HammadB@users.noreply.github.com>
commit b8dfc5b0f35a20f5049b1981d2885da2bae58242
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Tue Mar 26 15:44:39 2024 -0700
[CLN] Remove pulsar from python codebase (#1932)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Removes pulsar from the python codebase
- New functionality
- None
## Test plan
*How are these changes tested?*
- [x] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
None
commit 6b68fd11950d0ab43cd5470851f31f26018b745e
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Sun Mar 24 17:24:59 2024 -0700
[BUG] Fix as_nanos bad merge (#1928)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- #1927 accidentally reverted #1918
- New functionality
- None
## Test plan
*How are these changes tested?*
- [x] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
none
commit 53696308d918a21d3fd0796daac6d6e1256479a4
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Sun Mar 24 16:33:53 2024 -0700
[ENH] make hnsw query orchestrator use BF operator (#1927)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- This is a low-quality commit of wiring up the BF operator just to get
things working e2e. We will rewrite a lot of this marginally when the
chunk abstraction lands.
- New functionality
- None
## Test plan
*How are these changes tested?*
- [x] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
None
commit a7cf00d37fe7a25a769775a24095ac67cf1f45e0
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Sun Mar 24 16:25:31 2024 -0700
[BUG] BF operator should min(data, k) to prevent panic (#1926)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- The brute force operator should return min(data_size, k)
- New functionality
- None
## Test plan
*How are these changes tested?*
- [x] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
None
commit 71865259f3fe8a335ad0bc987a7cf94c1724984a
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Sun Mar 24 14:41:37 2024 -0700
[CLN] Clean up log error message (#1925)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- The error message here seems to have been copied from grpc sysdb and
was mentioning segment which was confusing.
- New functionality
- None
## Test plan
*How are these changes tested?*
- [x] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
None
commit 917fa0d14f07d75829d354ab8f6e7b9e6cb6af2c
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Sun Mar 24 14:40:50 2024 -0700
[BUG] Rust server should wire up system (#1924)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Rust server boot code should wire up the system.
- New functionality
- None
## Test plan
*How are these changes tested?*
- [x] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
None
commit 93b1f2a03de97f1f0c1c2313c0ab52e0c561940a
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Sun Mar 24 14:39:05 2024 -0700
[BUG] Log service N+1 query (#1923)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- The rust log service makes an additional query when it doesn't need to
if the timestamp is provided and we read less than a batch size. This
means that we are done reading the log and don't need to fetch again.
- New functionality
- None
## Test plan
*How are these changes tested?*
- [x] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
None
commit ba7b52e00b164e0cdeb43bc42affbe5d10fd629f
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Sun Mar 24 14:24:45 2024 -0700
[BUG] Log service incorrectly doesn't hydrate collection id (#1922)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Log service incorrectly did not return the collection ID of the
record. I suspect that we meant to strip the collection id and rehydrate
it into the proto for space reasons. I honored this intent and rehydrate
the colleciton id.
- Add some readme for running tests locally with PG
- Fixed a bug in the log_service test where the input is mutated, which
makes the source of truth have no collection id. This was passing when
we incorrectly returned no collection id but was correctly failing now.
I patched the test by cloning the records for a SOT.
- For the test I fixed, the expected vs actual order was incorrect.
- New functionality
- None
## Test plan
*How are these changes tested?*
- [x] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
None
commit 3a484552b2e1c69dd692c8f05cdc8b3a043fb68d
Author: Liquan Pei <liquanpei@gmail.com>
Date: Sun Mar 24 13:18:35 2024 -0700
[BUG] Use nano seconds as end_timestamp (#1918)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Match the timestamp with the LogService, which uses nanoseconds.
- New functionality
- ...
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit 1c140bc536c4fe3438647adcff1d8dd9c6c58c69
Author: Ben Eggers <ben@beneggers.com>
Date: Sat Mar 23 18:37:19 2024 -0700
build correctly in release mode
commit b602cfc2998b4a01b885b8d9ef0a8b0a405e4813
Author: Ben Eggers <64657842+beggers@users.noreply.github.com>
Date: Sat Mar 23 17:15:48 2024 -0700
[BUG] Point query service to the right logservice port (#1921)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Point the query service to port 50051 on the log service, not 50052.
We only use 50052 for the external-facing service during testing since
it conflicts with the sysdb on 50051.
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit 8c7cc1b537e380d3132982bfeeefb037e2c64897
Author: Ben Eggers <64657842+beggers@users.noreply.github.com>
Date: Sat Mar 23 17:12:25 2024 -0700
Make tilt and docker faster (#1920)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Tighten up dockerfiles and tiltfile: Get rid of Alpine linux, make
dockerfiles a little friendlier to cache and make the final images
smaller, and tell tilt only to reload some of the dockerfiles on certain
path reloads.
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit 650c09cd985dfe650cef81c7867174f3dabefb26
Author: Ben Eggers <ben@beneggers.com>
Date: Fri Mar 22 17:30:57 2024 -0700
k8s commands override docker commands
commit d66ff6b23a9134613b4c30c472550202f086e041
Author: Ben Eggers <ben@beneggers.com>
Date: Fri Mar 22 15:55:40 2024 -0700
Rust worker now rebuilds in ~53s on my machine
commit 08f23d209f600236e0c132ec4adf837753677760
Author: Ben Eggers <ben@beneggers.com>
Date: Fri Mar 22 11:30:35 2024 -0700
run release query service
commit 8f189d5de5197ba43f9bdab5d0da80aa9274e995
Author: Sumaiyah <sumaiyah.kola@outlook.com>
Date: Fri Mar 22 16:28:00 2024 +0000
[ENH] Add optional kwargs when initialising SentenceTransformerEmbeddingFunction class (#1891)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Add optional kwargs for `SetenceTransformer` when initialising
`SentenceTransformerEmbeddingFunction` class (Issue
[#1857](https://github.com/chroma-core/chroma/issues/1857))
## Test plan
*How are these changes tested?*
- [x] Tests pass locally with `pytest` for python
- installing chroma as an editable package locally and testing with the
code
```python
import chromadb
from chromadb.utils import embedding_functions
sentence_transformer_ef =
embedding_functions.SentenceTransformerEmbeddingFunction(prompts={"query":
"query: ", "passage": "passage: "})
print(sentence_transformer_ef.models['all-MiniLM-L6-v2'].prompts)
```
returned
```bash
{'query': 'query: ', 'passage': 'passage: '}
```
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
I have added the documentation for
`SentenceTransformerEmbeddingFunction` initialisation.
Co-authored-by: sumaiyah <o##wJdTOvNkIC!1R@bQO>
commit 468cb4cd04c3a5011716bedaae5a95519ef5a229
Author: Ben Eggers <64657842+beggers@users.noreply.github.com>
Date: Fri Mar 22 09:22:36 2024 -0700
[BUG] remove query service build cache - breaks staging (#1917)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Remove the build cache for the query service. Right now it makes the
stagnig query node recompile the binary.
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit dd7707c62c7cc18e67d922fbcfb423f7cb513dda
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Fri Mar 22 08:44:28 2024 -0700
[CLN] Add trace for memberlist update (#1912)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Add trace
- New functionality
- None
## Test plan
*How are these changes tested?*
- [x] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
None
commit e2ee34e63f574ca8052f8225be611accada54014
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Thu Mar 21 16:51:12 2024 -0700
[CHORE] Go Memberlist debug (#1913)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Add some debugs for memberlist in go
- New functionality
- None
## Test plan
*How are these changes tested?*
- [x] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit 234d1902d83b1338d91b9447fdc5a7d5a1631918
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Thu Mar 21 16:03:18 2024 -0700
[BUG] Make memberlist name configurable (#1911)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Make memberlist label configurable
- New functionality
- None
## Test plan
*How are these changes tested?*
- [x] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
None
commit 81375e82992b67d02d579d8bda1637975f0f0503
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Thu Mar 21 15:33:00 2024 -0700
[BUG] Rename memberlist to match new memberlist name (#1910)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- We renamed the memberlist to query-service-memberlist given the
query/compactor split.
- This aligns all the defaults with that name
- New functionality
- None
## Test plan
*How are these changes tested?*
- [x] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
None
commit f5e173d09bc0ec34bd9f1e90803c4fb57869643f
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Thu Mar 21 14:49:37 2024 -0700
[ENH] Add brute force operator (#1907)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- /
- New functionality
- Adds a brute force operator type
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
None
commit 7417f3de4841be16c3c4b878fee5700e4bc5d465
Author: Ben Eggers <64657842+beggers@users.noreply.github.com>
Date: Thu Mar 21 13:49:38 2024 -0700
[ENH] Seed distributed sysdb (#1909)
## Description of changes
*Summarize the changes made by this PR.*
- New functionality
- Seed distributed sysdb with default tenant and default db
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit e6f3aec9d37cc34f9b820e23737b0817a96a2029
Author: Liquan Pei <liquanpei@gmail.com>
Date: Thu Mar 21 01:21:18 2024 -0700
[ENH] Implement the PullLog operator (#1906)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- ...
- New functionality
- This PR implements the PullLog operator
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit 3985032d6c7cb0b691aeff56f1367bb5e7bbfa1b
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Wed Mar 20 15:43:33 2024 -0700
[ENH] Operator/Executor error handling (#1903)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Allows errors to be propagated from operators back to the original
caller of the task.
- New functionality
- None
## Test plan
*How are these changes tested?*
Compile checks show that the error propagates as expected.
- [x] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
None
commit 976aee3875993b6573f5f06eb9cd7c61b38f6f4a
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Wed Mar 20 15:36:01 2024 -0700
[BLD] Auto rebuild protos if they change in rust (#1902)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Auto rebuild protos if they change in rust
- New functionality
- None
## Test plan
*How are these changes tested?*
- [x] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
None
commit 51883f37a447ecfe9c1aa181ab3c3135aebb3a16
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Wed Mar 20 15:34:30 2024 -0700
[ENH] Add query-service server (#1899)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Add a new entrypoint for query-service. Delete the old worker
entrypoint that did not have read/write decoupling.
- Make dispatcher configurable
- Wrapped hnsw orchestrator logic into run() so server is unaware of it
- Make server struct {} have the resources it needs
- New functionality
- Add dynamic creation for log
- Add dynamic creation for sysdb
- Make server respond to query by using orchestrator
## Test plan
*How are these changes tested?*
- [x] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
None
commit 82e3db68896d8fe42091e3d1529cd93a4e3b6c10
Author: Liquan Pei <liquanpei@gmail.com>
Date: Wed Mar 20 12:52:14 2024 -0700
[ENH] Add end_timestamp to PullLog API (#1897)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- ...
- New functionality
- This PR adds end_timestamp to the PullLog API
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit af40bb11b4986052dfe551dc6dd8ed05a40eea7c
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Tue Mar 19 22:10:34 2024 -0700
[TST] Add test that the default config loads and parses (#1901)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Adds a basic test to ensure that the default config loads and parses
- New functionality
- None
## Test plan
*How are these changes tested?*
This is a test
- [x] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
None
commit c04a31ad241e8e2641b0fdaf37852f33cac65d79
Author: Ben Eggers <64657842+beggers@users.noreply.github.com>
Date: Tue Mar 19 20:39:19 2024 -0700
[BUG] Add logservice config to query service (#1900)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Add log service config to query service.
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit 9f165338cbe6ab5e416e38a529a350412f11adc9
Author: Ben Eggers <64657842+beggers@users.noreply.github.com>
Date: Tue Mar 19 17:18:35 2024 -0700
[BUG] Wipe sysdb and reapply schema each time (#1898)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Correctly wipe the sysdb and re-apply schema each push.
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit c07281fdaa8d98e6cd9ee97dfe6fc2db12dcafa3
Author: Ben Eggers <64657842+beggers@users.noreply.github.com>
Date: Tue Mar 19 15:58:20 2024 -0700
[ENH] Clean DB before applying (for now) (#1896)
## Description of changes
*Summarize the changes made by this PR.*
- New functionality
- Call `atlas schema clean` before applying migrations.
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit 93a659dff11f2b6f839596afc5efe3314dd92856
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Tue Mar 19 15:41:02 2024 -0700
[ENH] Add push based operators, centralized dispatch, hardcode query plan as state machine (#1888)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Change on_start to async
- Move on_start into the executor.
- New functionality
- This PR adds operators to the query/compaction workers
- Implements a PullLog operator
- Adds a HnswQueryOrchestrator, which can be thought of as a passive
state machine of a hardcoded query plan.
- Adds a dispatcher with worker threads for scheduling tasks. Worker
threads pull for tasks.
Pending work I will address in following prs:
- [ ] Make the callers of dispatch unaware of wrap()
- [ ] PullLogs should poll until completion @Ishiihara is taking this
- [ ] Error handling
- [ ] Wrap the orchestrator
- [ ] Add a server struct and have it create the aforementioned
orchestrator wrapper and then push to it.
- [ ] Impl configurable
## Test plan
*How are these changes tested?*
- [x] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
None
commit db4cabaa0bf6c3cf5f62cdb18f323eabb92caf32
Author: Ben Eggers <64657842+beggers@users.noreply.github.com>
Date: Tue Mar 19 13:38:39 2024 -0700
[BUG] install jq in migration, not sysdb (#1894)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- We use jq for some env var stuff. We need it in the migration, not the
sysdb service (which knows how to correctly URL-escape args)
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit ab4f039e10ff94988d3ef3bafb9805b169cb83df
Author: Ben Eggers <64657842+beggers@users.noreply.github.com>
Date: Mon Mar 18 17:40:32 2024 -0700
[ENH] Run the sysdb migration in sh so we can do env var substitition (#1887)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Run the SysDB migration in `/bin/sh` so we can sub in env vars as
needed. This requires installing atlas into a debian image instead of
using the atlas image directly.
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit 2829c57d6c5a58c14e42ca10ad22537b5c2a20df
Author: Ben Eggers <64657842+beggers@users.noreply.github.com>
Date: Mon Mar 18 14:56:07 2024 -0700
[ENH] Let us specify sysdbMigration target (#1886)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Template out values in sysdb-migration.yaml so we can point it at
databases besides localhost postgres.
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit ea954bffaa435d0abdc2bbc0aaa88e12f2964225
Author: Ben Eggers <64657842+beggers@users.noreply.github.com>
Date: Fri Mar 15 18:42:37 2024 -0700
[CLN] Clean up k8s, rename services and values (#1881)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Standardize k8s on the naming scheme we agreed on.
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit 41ad7ba1debc0db86f7885b17aa733504cdab396
Author: Weili Gu <3451471+weiligu@users.noreply.github.com>
Date: Fri Mar 15 16:26:06 2024 -0700
[ENH] Compactor flush (#1879)
### FlushCollectionCompaction
https://github.com/chroma-core/chroma/pull/1869
https://linear.app/trychroma/issue/CHR-298/segment-compaction-flush
- grpc in collection service FlushCollectionCompaction
- table_catalog: FlushCollectionCompaction
- in one transaction:
- segment: RegisterFilePaths
- collection: UpdateLogPositionAndVersion
- tenant: UpdateTenantLastCompactionTime // optional, should add a
config to disable
### Tests
- test_utils to create tenant, database, collection, segments
- change existing tests to use test suite: begin suite set up tenant, db
(unless testing tenant and db in the test itself), end suite clean up,
begin test set up sample objects (eg, collection and segments)
- each tests should create objects that with different IDs and Names
commit 47fdc612ee38042cf01d4f92a001b827021c5d7b
Author: Ben Eggers <64657842+beggers@users.noreply.github.com>
Date: Thu Mar 14 21:25:51 2024 -0700
[BUG] Fix Golang codebase test flakiness
commit fa3a6f13530f43fc7651004e7053667bd5ed5dae
Author: Ben Eggers <64657842+beggers@users.noreply.github.com>
Date: Thu Mar 14 19:06:57 2024 -0700
Revert "[ENH] fush API for compactor" (#1874)
Reverts chroma-core/chroma#1869
breaks tests
commit c1835806183ec631760af3ae5ee9571c09952619
Author: Weili Gu <3451471+weiligu@users.noreply.github.com>
Date: Thu Mar 14 18:13:00 2024 -0700
[ENH] fush API for compactor (#1869)
### FlushCollectionCompaction
https://linear.app/trychroma/issue/CHR-298/segment-compaction-flush
- grpc in collection service FlushCollectionCompaction
- table_catalog: FlushCollectionCompaction
- in one transaction:
- segment: RegisterFilePaths
- collection: UpdateLogPositionAndVersion
- tenant: UpdateTenantLastCompactionTime // optional, should add a
config to disable
### Tests
- test_utils to create tenant, database, collection, segments
- change existing tests to use test suite: begin suite set up tenant, db
(unless testing tenant and db in the test itself), end suite clean up,
begin test set up sample objects (eg, collection and segments)
- each tests should create objects that with different IDs and Names
commit 21e5fcceefb6a9101e474f4d91a485734a550a80
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Wed Mar 13 17:30:11 2024 -0700
[ENH] Add uint blockfile key/val (#1854)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Lets the blockfile use uint vals and keys.
- Some name cleanup / unification
- New functionality
- /
## Test plan
*How are these changes tested?*
Basic sanity tests.
- [x] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
None
commit b31a07cf112af78df0d9c2a7b92320dff5923c03
Author: Ben Eggers <64657842+beggers@users.noreply.github.com>
Date: Wed Mar 13 17:15:54 2024 -0700
[ENH] Property testing for metadata indices (#1794)
## Description of changes
*Summarize the changes made by this PR.*
- New functionality
- Make MetadataIndex and BlockfileMetadataIndex properly generic. Let
the type system do the work!
- Property testing for metadata indices. I learned a lot about the
borrow checker :0)
## Test plan
*How are these changes tested?*
- [x] Tests pass locally with `pytest` for python, `yarn test` for js
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit 79f0cd17cc6424a2d78e718ec5a255211855d935
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Wed Mar 13 17:11:05 2024 -0700
[ENH] Add Segment Interfaces (#1853)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Adds an interface type for segments. We can add flush etc here in the
future. I will move hnsw to use this in a stacked PR.
- New functionality
- None.
## Test plan
*How are these changes tested?*
No logical changes. These are just interfaces.
- [x] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
None
commit 97c7539bf1420f1293ab906eaaa8babe276ddd5b
Author: Matthew Keller <matthew.ed.keller@gmail.com>
Date: Wed Mar 13 19:58:05 2024 -0400
[CLN] Apply Prettier Config to JS Client Repo (#1831)
## Description of changes
*Summarize the changes made by this PR.*
Applies Prettier formatting to the JS Client Repo and add a presubmit
rule to enforce that behavior.
## Test plan
N/A just formatting changes.
commit d425111fbf98d901bb4a416fd0a414bbeeddd324
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Wed Mar 13 16:56:03 2024 -0700
[ENH] Add arrow blockfile provider (#1847)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- /
- New functionality
- Adds a provider for ArrowBlockfiles
## Test plan
*How are these changes tested?*
Not tested for now. A future PR will contains tests.
- [x] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
None
commit 69660b8f638ad40b3131e6e9cfbd11eb426ecbd6
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Wed Mar 13 16:43:15 2024 -0700
[ENH] Add Arrow-backed blockfile (#1846)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Adds roaring bitmap value types, builders etc
- New functionality
- Adds the base of the blockfile implementation for the arrow-backed
blockfile.
## Test plan
*How are these changes tested?*
Basic
- [x] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
None
commit dd7331148bb0cc822069a85bc810e147d5e2aa9d
Author: Ben Eggers <ben@beneggers.com>
Date: Wed Mar 13 16:25:24 2024 -0700
[BUG] grace=None not 0
commit 3c4881740e2a1ba1ecafec8f2e7e1cf996cb29f2
Author: Ben Eggers <ben@beneggers.com>
Date: Wed Mar 13 10:21:56 2024 -0700
[CLN] undo unecessary change in 0e2776c98a6637e1cbab5c1582214bfdbb861499
commit 0e2776c98a6637e1cbab5c1582214bfdbb861499
Author: Ben Eggers <ben@beneggers.com>
Date: Wed Mar 13 10:20:05 2024 -0700
[BUG] Shut down test system to hopefully fix test flakiness
commit 17d427bd15d2eca30561e1f048ceb0194c16e32c
Author: Matthew Keller <matthew.ed.keller@gmail.com>
Date: Tue Mar 12 18:52:05 2024 -0400
[CHORE] add .chrome_env file to the gitignore (#1834)
## Description of changes
*Summarize the changes made by this PR.*
The js tests automatically generate this file so it shouldn't be checked
in.
## Test plan
*How are these changes tested?*
- [x] confirmed that this file is no longer included.
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit 19512e6ca4ca599f93911aa28b704d3508cb13e6
Author: nicolasgere <nicolas.gerelm@gmail.com>
Date: Tue Mar 12 13:22:08 2024 -0700
[BUG]: fix auth (#1864)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Hosted auth is not working due to a missing import
commit e715cd2ac85281a415aa54750f67fc3240f6e419
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Tue Mar 12 11:19:41 2024 -0700
[ENH] Sparse index (#1825)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- /
- New functionality
- Adds a sparse index for blockfiles
## Test plan
*How are these changes tested?*
Basic unit tests for now
- [x] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
None
commit 73766c315b54a192f2ef241d82520443173fefdf
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Mon Mar 11 22:30:35 2024 -0700
[ENH] Block delta (#1823)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- None
- New functionality
- Adds a BlockDelta which is used to mutate a block. It reads the
contents of a block and then can allows changes to be applied directly
to it.
- Adds the ability to apply() a block delta
- Add a BlockProvider, which is the interface for loading and managing
blocks.
## Test plan
*How are these changes tested?*
- [x] Basic unit tests `cargo test`
## Documentation Changes
None
commit 9b607c492120e4a99019710c9758ae810396cbd9
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Mon Mar 11 11:45:11 2024 -0700
[BLD] Add docker arg to build in release mode (#1859)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Adds the ability to build dockerfile for rust worker in release mdoe
- New functionality
- /
## Test plan
*How are these changes tested?*
I manually modified the arg and made sure the expected control flow was
followed.
- [x] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
None required.
commit 017138ae49b370a0cd460737e30662abb855851d
Author: Weili Gu <3451471+weiligu@users.noreply.github.com>
Date: Mon Mar 11 09:11:46 2024 -0700
[ENH] last compaction time (#1856)
## Description of changes
https://linear.app/trychroma/issue/CHR-366/add-api-to-get-tenant-last-compaction-time
Add functionalities to set and get tenants' last compaction time.
## Test plan
- [ ] tenant_database_service_test.go
- [ ] tenant_test.go
commit 1e5e024fa2984e06d1ab17ee3d9c4960c56400b4
Author: Weili Gu <3451471+weiligu@users.noreply.github.com>
Date: Sat Mar 9 13:38:30 2024 -0800
[TST] fix flacky coordinator tests (#1852)
Fix flack coordinator tests, two root causes:
since we replaced in memory map with real DB, tests become flacky
because
1. we didn't clean up the state in apis_test.go
2. tests were using default names and running asynchronously
fix:
1. making apis_test.go setup and teardown the db
2. disable async flag in go test
in longer term: we can run tests asyn when we have individual tests
create it's own objects in DB instead of sharing same ones. Unless in
some special cases where we want to test very specific namespace, these
tests need to be configured to run synchronously. We also need to
complete delete function (including cascading deletions) for some
objects before we can make tests async so that we don't need to clean
the entire DB.
one more fix to atlas set up file due to renaming the go packages.
commit 186165dfbfd6eeb3db92c07d8ea638751b3f861d
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Sat Mar 9 09:44:50 2024 -0800
[TST] Scheduler test fix (#1851)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Fixes an issue with the scheduler test where tiebreaks on 0 last
compaction time are random. This uses contains instead of order. When
@Ishiihara adds a the api for last compaction time to the sysdb this
will be fixed and reverted!
- New functionality
- None
## Test plan
*How are these changes tested?*
- [x] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
None
commit cf46f7789fd4a6a66aa1ca762fac5d7216b44c04
Author: Liquan Pei <liquanpei@gmail.com>
Date: Fri Mar 8 11:55:55 2024 -0800
[ENH] Compactor scheduler with system (#1844)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- ...
- New functionality
- This PR adds compactor scheduler with system components and scheduler.
## Test plan
*How are these changes tested?*
- [ ] cargo test
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit 34e2795216d610389e05cba52fcb781ba1b433bc
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Fri Mar 8 09:10:48 2024 -0800
[DOC] Update and rename README to README.md (#1848)
Doc cleanup
commit 722a307c46c36d312747cfe99ec7fb48a0dde9f8
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Fri Mar 8 08:29:19 2024 -0800
[ENH] Block iterator (#1822)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- None
- New functionality
- Adds a basic block iterator to allow for easily cloning a block into a
BlockDelta
## Test plan
*How are these changes tested?*
- [x] Tests pass locally with `cargo test`
## Documentation Changes
None
commit cb8f2bb76c0671b77349603b3ffe7977fe7f0ad7
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Fri Mar 8 07:54:47 2024 -0800
[ENH] Block builder (#1821)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- /
- New functionality
- Adds a block builder for creating BlockData. Blocks hold BlockData in
order to serve queries.
## Test plan
*How are these changes tested?*
- [x] Tests pass locally with `cargo test`
Basic unit tests were added
## Documentation Changes
None
commit 64b39b155c3efaea3e6d908371fa2e285a2b7a1f
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Thu Mar 7 17:18:58 2024 -0800
[ENH] Arrow-backed block (#1820)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- /
- New functionality
- Adds an arrow-backed block, the code documentation covers what this is
more in detail.
## Test plan
*How are these changes tested?*
- [x] Tests pass locally with `cargo test`
Tests will be in stacked PR
## Documentation Changes
None
commit 369eddde01d94ad2f4af4ad5c6fd2733ce0a7ecd
Author: Liquan Pei <liquanpei@gmail.com>
Date: Thu Mar 7 16:45:26 2024 -0800
[ENH] Add system scheduler to allow tasks to run with a schedule (#1839)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- ...
- New functionality
- This PR adds the compatibility to allow components to schedule tasks
## Test plan
*How are these changes tested?*
- [ ] cargo test
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit 2cbbb2cf3cabbcdcfc887c7dd09145eb95941541
Author: Weili Gu <3451471+weiligu@users.noreply.github.com>
Date: Thu Mar 7 15:58:15 2024 -0800
[ENH] log service staging (#1843)
## Description of changes
connect log service in staging
commit 123826f8661db2198ce77dc7e61f6daf600d9185
Author: Ben Eggers <64657842+beggers@users.noreply.github.com>
Date: Thu Mar 7 14:56:33 2024 -0800
[CLN] Run coordinator in `/bin/sh` (#1816)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Run coordinator in a shell instead of `exec` environment
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit 8087c22c512f6220112dda5288f48745ab915d0f
Author: Hammad Bashir <HammadB@users.noreply.github.com>
Date: Thu Mar 7 11:54:04 2024 -0800
[ENH] Blockfile provider (#1819)
## Description of changes
This PR intermingles one substantial change with two smaller ones,
apologies.
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Adds sizing methods for Keys and Values and a Clone method that is
aware of the idiosyncrasies of Arrow
- New functionality
- Adds a blockfile provider, which presents an open/create API for
managing blockfiles. All callers of blockfile:new() were migrated to
this way of managing blockfiles.
## Test plan
*How are these changes tested?*
- [x] Tests pass locally with `cargo test`
## Documentation Changes
None required.
commit 5f1263ba4c8c2749054f83065c583437d6408daf
Author: Ben Eggers <64657842+beggers@users.noreply.github.com>
Date: Thu Mar 7 11:00:51 2024 -0800
[ENH] Helm CD to hosted (#1840)
## Description of changes
*Summarize the changes made by this PR.*
- Changes to OSS k8s will trigger hosted-chroma to update its k8s to
stay in sync.
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit f7fff9a0f70edee70542ad59f8ee758037a2ef20
Author: Ben Eggers <64657842+beggers@users.noreply.github.com>
Date: Thu Mar 7 10:53:39 2024 -0800
[ENH] helm chart tweaks (#1836)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Move jaeger to `test/` where it belongs (it shouldn't be deployed
alongside prod distributed chroma)
- indent extra env configs and strings correctly.
## Test plan
*How are these changes tested?*
- [ ] Tests pass locally with `pytest` for python, `yarn test` for js,
`cargo test` for rust
## Documentation Changes
*Are all docstrings for user-facing APIs updated if required? Do we need
to make documentation changes in the [docs
repository](https://github.com/chroma-core/docs)?*
commit 0c44deb66637cf2e4dcf4fe502fa44d0c6b462ce
Author: Weili Gu <3451471+weiligu@users.noreply.github.com>
Date: Wed Mar 6 15:57:18 2024 -0800
[ENH] remove sysdb cache (#1835)
## Description of changes
https://linear.app/trychroma/issue/CHR-299/remove-cache-in-sysdb
- remove sysdb cache
- remove created field from collection object
## Test plan
- [ ] passing existing changes
commit b7e8b62e6444a779193f0afb06bcb47fb4f25a26
Author: nicolasgere <nicolas.gerelm@gmail.com>
Date: Wed Mar 6 13:15:22 2024 -0800
[ENH]: add grpc client interceptor (#1818)
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Add grpc client interceptor
## Test plan
*How are these changes tested?*
- Tested locally with distributed tracing
commit 401a7f0543ad259a16648c68d670661456a9818c
Author: Matthew Keller <matthew.ed.keller@gmail.com>
Date: Wed Mar 6 14:23:32 2024 -0500
[DOC] add a bit more detail to the js client DEVELOP.md (#1829)
commit 362d3506a0f4de1764c566a22255ece334107a65
Author: Trayan Azarov <tazarov@users.noreply.github.com>
Date: Wed Mar 6 20:57:27 2024 +0200
[ENH]: Updates to validation errors (#1703)
Refs: #1694
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Shortened validation error printouts to improve log readability
## Test plan
*How are these changes tested?*
- [x] Tests pass locally with `pytest` for python, `yarn test` for js
## Documentation Changes
N/A
commit 73a163067cb33d1eff75e193bd08ca4ec8202920
Author: Trayan Azarov <tazarov@users.noreply.github.com>
Date: Wed Mar 6 20:55:44 2024 +0200
[BUG]: Deleting FTS entries upon deletion of individual records (#1689)
Refs: #1664
## Description of changes
*Summarize the changes made by this PR.*
- Improvements & Bug fixes
- Follow up on #1664
## Test plan
*How are these changes tested?*
- [x] Tests pass locally with `pytest` for python
## Documentation Changes
N/A
commit 276226b82fe567f12c62ba7432f5b04cfb426ae0
Author: Trayan Azarov <tazarov@users.noreply.github…
atroyn
pushed a commit
to csbasil/chroma
that referenced
this pull request
Apr 3, 2024
## Description of changes *Summarize the changes made by this PR.* - Improvements & Bug fixes - Auto rebuild protos if they change in rust - New functionality - None ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Documentation Changes None
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of changes
Summarize the changes made by this PR.
Test plan
How are these changes tested?
pytestfor python,yarn testfor js,cargo testfor rustDocumentation Changes
None