Deleting points and vectors from qdrant collection

[manosec]

Hi there, I've implemented class qdrant class around a custom class and using methods from an API call. So now When I call the delete API method, My points do not get deleted, I've gone through the documentation of qdrant where I found it is not deleted right after and I've also seen that passing parameter wait=true will delete but no use, is there anything I'm doing wrong. Please help me in this. And below is my delete method. I'm deleting the points by metadata fields which I used.

 def delete(self, filename):
        try:
            response = self.client.delete(
                    collection_name=COLLECTION_NAME,
                    points_selector=rest.FilterSelector(
                        filter=rest.Filter(
                            must=[
                                rest.FieldCondition(
                                    key="id",
                                    match=rest.MatchValue(value=filename),
                                ),
                            ],
                        )
                    ),
                )
            
            return response
        except Exception as e:
            return e
        

[manosec]

My response is successful but the point is still there and available for querying, I tried the same in the Qdrant cloud console but no success there. Getting successful response but the points are still available for querying.

[manosec]

The above is the sample screenshot of success response upon using delete request filter by points metadata in qdrant console. but the Ajai.pdf is still available for querying and I get the content response from the DB.

[timvisee]

At quick glance I don't see an immediate problem here.

Could you retrieve the point you expect to be deleted and dump it here, so that we know what your payload looks like?

For example:

POST /collections/testing_delete/points
{
    "ids": [0],
    "with_vectors": true,
    "with_payload": true
}

Also, could you show how you query this point afterwards?

[manosec]

@timvisee , I have tried retrieving the points by modifying the above example you provided an actual point, but I'm getting the empty response only even though the point is not deleted. Meaning I'm getting only empty responses to the query regardless of deletion. What is I'm doing wrong here? Please correct me. I have attached the sample screenshots below.

1. The below screenshot is a sample record in the DB.

2. The image below shows the query of the same record with ids(points) and its response.

Note: I'm using qdrant cloud console for testing the deletion of points(to verify). I'm working on qdrant python API to delete the points.

Thanks for your understanding.

[timvisee]

I wonder if this has to do with the limited precision of numbers in JavaScript.

Could you check if you see the same behavior when retrieving a point through Python, rather than through the Web UI?

It is a known issue: qdrant/qdrant-web-ui#87
Sadly it's extremely hard to fix, since it's so deeply integrated into JavaScript.

[315930399]

I met the same situation. Points is still there after I call delete api.

[timvisee]

Are you doing this through the JavaScript API or web interface? If so, this would be the reason for it.

Yesterday we've released Qdrant 1.8.3 which fixes this issue.

[315930399]

No, I try to delete by python client and http post.

[CHIRU98]

@timvisee still vector are the there even after upgrading the version to Qdrant 1.8.3 and deleting

[315930399]

I have solved this problem. I used the wrong key to delete. Are you using Langchain? If so, you need to delete using nested key like metadata.souce_document

[315930399]

You should check your payload structure first. Here is the nested key documentation:https://qdrant.tech/documentation/concepts/filtering/#nested-key

[CHIRU98]

thank for your reply 315930399.Yes i am using the Langchain and i am not able to access this https://qdrant.tech/documentation/concepts/filtering/#nested-key

[alexisCode50]

Verify your payload structure, you may need to use Nested keys, more info in this link: https://qdrant.tech/documentation/concepts/filtering/

[
  {
    "id": 1,
    "country": {
      "name": "Germany",
      "cities": [
        {
          "name": "Berlin",
          "population": 3.7,
          "sightseeing": ["Brandenburg Gate", "Reichstag"]
        },
        {
          "name": "Munich",
          "population": 1.5,
          "sightseeing": ["Marienplatz", "Olympiapark"]
        }
      ]
    }
  },
  {
    "id": 2,
    "country": {
      "name": "Japan",
      "cities": [
        {
          "name": "Tokyo",
          "population": 9.3,
          "sightseeing": ["Tokyo Tower", "Tokyo Skytree"]
        },
        {
          "name": "Osaka",
          "population": 2.7,
          "sightseeing": ["Osaka Castle", "Universal Studios Japan"]
        }
      ]
    }
  }
]

Filter:

POST /collections/{collection_name}/points/scroll
{
    "filter": {
        "should": [
            {
                "key": "country.name",
                "match": {
                    "value": "Germany"
                }
            }
        ]
    }
}

[d-deepika-kumari]

I am using below code to delete points using its point default id, which uniquely identifies points in collection in qdrant vector db.

#Collecting point ids in a list:
for doc in doc_to_delete[0]:
ids_to_delete.append(doc.id)

#deleting points from collection using point ids
vector_client.delete(
collection_name=collection_name,
points_selector=models.PointIdsList(points=ids_to_delete)

Can someone confirms me will it delete both points and its associated vectors from collection?

After running this code, I verified in qdrant UI and I can see points are getting deleted, but the vectors count remain the same.

isn't both vectors_count and points_count should be same?

Thanks,
Deepika

[timvisee]

These counts are approximate and do not guarantee correctness. So what you see in the above screenshot can be expected.

We document that here: https://qdrant.tech/documentation/concepts/collections/#approximate-point-and-vector-counts

[d-deepika-kumari]

Thanks for the clarification @timvisee . So, if I am deleting the points from qdrant collection, I can say the vectors associated with the point will also be deleted, is this understanding correct?

[timvisee]

Correct.

To put it in a different way: if you delete a point, their vectors are considered to be deleted as well. That means you will not be able to retrieve or search them anymore.

[d-deepika-kumari]

Thanks a lot!