Skip to content

Fix PgVectorStore doDelete function as batch update #2434

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

Fix PgVectorStore doDelete function as batch update #2434

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 15 additions & 6 deletions ...vector-store/src/main/java/org/springframework/ai/vectorstore/pgvector/PgVectorStore.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -319,12 +319,21 @@ private Object convertIdToPgType(String id) {

@Override
public void doDelete(List<String> idList) {
int updateCount = 0;
for (String id : idList) {
int count = this.jdbcTemplate.update("DELETE FROM " + getFullyQualifiedTableName() + " WHERE id = ?",
UUID.fromString(id));
updateCount = updateCount + count;
}
String sql = "DELETE FROM " + getFullyQualifiedTableName() + " WHERE id = ?";

this.jdbcTemplate.batchUpdate(sql, new BatchPreparedStatementSetter() {

@Override
public void setValues(PreparedStatement ps, int i) throws SQLException {
var id = idList.get(i);
StatementCreatorUtils.setParameterValue(ps, 1, SqlTypeValue.TYPE_UNKNOWN, convertIdToPgType(id));
}

@Override
public int getBatchSize() {
return idList.size();
}
});
}

@Override
Expand Down
44 changes: 44 additions & 0 deletions ...ctor-store/src/test/java/org/springframework/ai/vectorstore/pgvector/PgVectorStoreIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@
* @author Christian Tzolov
* @author Thomas Vitale
* @author Jihoon Kim
* @author CChuYong
*/
@Testcontainers
@EnabledIfEnvironmentVariable(named = "OPENAI_API_KEY", matches = ".+")
Expand Down Expand Up @@ -232,6 +233,47 @@ public void testToPgTypeWithNonUuidIdType() {
});
}

@Test
public void testBulkOperationWithUuidIdType() {
Copy link
Member

@ilayaperumalg ilayaperumalg Mar 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding the test. This test runs fine when running as a standalone but fails when all the tests within this class run. Looks like some race condition there. Could you check?

Copy link
Contributor Author

@CChuYong CChuYong Mar 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your review!
i figured out that getNativeClientTest also initializes vector table, so i also added code for dropping table in getNativeClientTest. fb829af

Copy link
Member

@ilayaperumalg ilayaperumalg Mar 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the fix!

this.contextRunner.withPropertyValues("test.spring.ai.vectorstore.pgvector.distanceType=" + "COSINE_DISTANCE")
.run(context -> {

VectorStore vectorStore = context.getBean(VectorStore.class);

List<Document> documents = List.of(
new Document(new RandomIdGenerator().generateId(), "TEXT", new HashMap<>()),
new Document(new RandomIdGenerator().generateId(), "TEXT", new HashMap<>()),
new Document(new RandomIdGenerator().generateId(), "TEXT", new HashMap<>()));
vectorStore.add(documents);

List<String> idList = documents.stream().map(Document::getId).toList();
vectorStore.delete(idList);

dropTable(context);
});
}

@Test
public void testBulkOperationWithNonUuidIdType() {
this.contextRunner.withPropertyValues("test.spring.ai.vectorstore.pgvector.distanceType=" + "COSINE_DISTANCE")
.withPropertyValues("test.spring.ai.vectorstore.pgvector.initializeSchema=" + false)
.withPropertyValues("test.spring.ai.vectorstore.pgvector.idType=" + "TEXT")
.run(context -> {
VectorStore vectorStore = context.getBean(VectorStore.class);
initSchema(context);

List<Document> documents = List.of(new Document("NON_UUID_1", "TEXT", new HashMap<>()),
new Document("NON_UUID_2", "TEXT", new HashMap<>()),
new Document("NON_UUID_3", "TEXT", new HashMap<>()));
vectorStore.add(documents);

List<String> idList = documents.stream().map(Document::getId).toList();
vectorStore.delete(idList);

dropTable(context);
});
}

@ParameterizedTest(name = "Filter expression {0} should return {1} records ")
@MethodSource("provideFilters")
public void searchWithInFilter(String expression, Integer expectedRecords) {
Expand Down Expand Up @@ -436,6 +478,8 @@ void getNativeClientTest() {
PgVectorStore vectorStore = context.getBean(PgVectorStore.class);
Optional<JdbcTemplate> nativeClient = vectorStore.getNativeClient();
assertThat(nativeClient).isPresent();

dropTable(context);
});
}

Expand Down