refactor: createVectorStoreNode refactoring and embeddings batching support (no-changelog)

[OlegIvaniv]

Summary

• Refactor vector store node architecture into modular operation-based components
• Implement batched document embedding to reduce embeddings API calls and speed-up insertion
• UI rendering performance optimizations with debounced node updates

Changes

Vector Store Architecture Improvements

• Restructured createVectorStoreNode into separate modules organized by operation type
• Created dedicated handlers for each operation (load, insert, retrieve, update, retrieve-as-tool)
• Added test suite for each operation handler

Embedding Performance Optimization

• Implemented batched document processing for vector store insertions
• Added configurable batch size parameter to control embedding operations
• For node version 1.1+, documents are processed in batches instead of individually
• Single embedding operation performed per batch to significantly reduce API calls

UI Rendering Performance

• Added debounced rendering of workflow nodes and connections in canvas
• Implemented dynamic debounce timing based on workflow execution state:
  • Fast updates during editing (0ms debounce)
  • Throttled updates during execution (200ms debounce, 50ms max wait)
• Reduces CPU/memory usage during execution with large workflows

In my tests populating 500 items to QDrant vector store went from ~5mins to 10s.

Before

CleanShot.2025-03-04.at.12.50.00121221.mp4

After

CleanShot.2025-03-04.at.12.46.28.mp4

Related Linear tickets, Github issues, and Community forum posts

https://linear.app/n8n/issue/AI-749

Review / Merge checklist

• PR title and summary are descriptive. (conventions)
• Docs updated or follow-up ticket created.
• Tests included.
• PR Labeled with release/backport (if the PR is an urgent fix that needs to be backported)

[codecov]

Codecov Report

Attention: Patch coverage is 82.65306% with 34 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...red/createVectorStoreNode/createVectorStoreNode.ts	45.45%	18 Missing ⚠️
...re/VectorStorePinecone/VectorStorePinecone.node.ts	0.00%	2 Missing ⚠️
..._store/VectorStoreQdrant/VectorStoreQdrant.node.ts	0.00%	2 Missing ⚠️
...re/VectorStoreSupabase/VectorStoreSupabase.node.ts	0.00%	2 Missing ⚠️
...reateVectorStoreNode/operations/updateOperation.ts	91.66%	2 Missing ⚠️
...editor-ui/src/components/canvas/WorkflowCanvas.vue	94.59%	2 Missing ⚠️
...re/VectorStoreInMemory/VectorStoreInMemory.node.ts	0.00%	1 Missing ⚠️
...re/VectorStorePGVector/VectorStorePGVector.node.ts	0.00%	1 Missing ⚠️
...rePineconeInsert/VectorStorePineconeInsert.node.ts	0.00%	1 Missing ⚠️
...rStorePineconeLoad/VectorStorePineconeLoad.node.ts	0.00%	1 Missing ⚠️
... and 2 more

📢 Thoughts on this report? Let us know!

[jeanpaul]

I tested it locally, and seems to work nicely. I like how this splits up the operations and adds tests for each operation individually. Nice work.

Request some changes, which are mostly docs-changes in the readme and some comments that don't really add a lot of value at the moment.

[OlegIvaniv]

Thanks for the review, @jeanpaul! I tweaked the docs a bit and included also a very small fix for filtering in retrieve-as-tool mode. Can you re-review, please?

[jeanpaul]

Thanks for addressing my comments and fixing AI-740 in the process too! Looks good to me, and works as expected, so let's ship it!

[cypress]

n8n    Run #9753

Run Properties:   Passed #9753  •  382766ce6a: 🌳 🖥️ browsers:node18.12.0-chrome107 🤖 OlegIvaniv 🗃️ e2e/*

Project	n8n
Branch Review	ai-749-speed-up-vector-store-ingestion
Run status	Passed #9753
Run duration	04m 39s
Commit	382766ce6a: 🌳 🖥️ browsers:node18.12.0-chrome107 🤖 OlegIvaniv 🗃️ e2e/*
Committer	Oleg Ivaniv
View all properties for this run ↗︎

---

Test results
Failures	0
Flaky	0
Pending	5
Skipped	0
Passing	440
View all changes introduced in this branch ↗︎

[github-actions]

✅ All Cypress E2E specs passed

[github-actions]

✅ All Cypress E2E specs passed

[janober]

Got released with n8n@1.84.0