Skip to content

MediaWiki ETL limit loading batch into 1! #60

@amindadgar

Description

@amindadgar

In llama-index pipeline when trying to load documents into vectorstore, it first loads into docstore and then into vectorstore.
In any case problems raised while loading into docstore the data would be missed to be loaded into vectorstore. So we need to limit the batch size to 1 meaning the data will be 1 by 1 loaded into docstore + vectorstore.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions