Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support query vector field #6299

Closed
cydrain opened this issue Jul 5, 2021 · 1 comment · Fixed by #6570, #6598 or #7102
Closed

Support query vector field #6299

cydrain opened this issue Jul 5, 2021 · 1 comment · Fixed by #6570, #6598 or #7102
Assignees
Labels
kind/feature Issues related to feature request from users
Milestone

Comments

@cydrain
Copy link
Contributor

cydrain commented Jul 5, 2021

Please state your issue using the following template and, most importantly, in English.

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. E.g. I wish I could use Milvus to do [...]

Describe the solution you'd like
Support query vector field

@cydrain cydrain added the kind/feature Issues related to feature request from users label Jul 5, 2021
@cydrain cydrain self-assigned this Jul 5, 2021
@xiaofan-luan
Copy link
Contributor

Anyway we can hidden IO by an interface?
for instance, we need a block cache, manage a chunk of of data, and do auto load when data is not in memory.
User can limit memory consumption by specify max memory watermark. And we run eviction based on query frequency.

Before we discuss start to implement read on Minio, few things may want to be documented and discussed.

  1. How is current solution to manage memory in Segcore, need a document to explain.@FluorineDog @cydrain @neza2017
  2. How we can change data into chunk or block style? what's granularity of a block or chunk, any meta necessary to accelerate chunk query? (How we prune and index each block)
  3. what API is necessary to query from Minio? We should not read the entire file from Minio/S3, but use the range tag to specify a range in this file, examples in Minio:
    try (InputStream stream = minioClient.getObject(
    GetObjectArgs.builder()
    .bucket("my-bucketname")
    .object("my-objectname")
    .offset(1024L)
    .length(4096L)
    .build())) {
    // Read data from stream
    }

Further more, we need more discussion on how we boost the performance once we implement IO on object storage?
SSD Caching is one choice already in the plan.

S3 Select is probably a good way to decrease network ios, but I'm not sure if Minio has similar implementation.
https://docs.aws.amazon.com/zh_cn/AmazonS3/latest/API/API_SelectObjectContent.html

Third thing is importing another another kv store, be sure our currently solution is compatible with the kv index. Kv store would be a optional part in milvus to boost delete/get performance and do primary key dedup

@FluorineDog FluorineDog self-assigned this Jul 6, 2021
@yhmo yhmo added this to the 2.0-Backlog milestone Jul 6, 2021
@cydrain cydrain changed the title Add new C++ interface to read Minio file Support query vector field Jul 12, 2021
@wxyucs wxyucs added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Jul 13, 2021
@wxyucs wxyucs removed the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Jul 29, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature Issues related to feature request from users
Projects
None yet
5 participants