Skip to content

VectorStore.from_filespace() raises IndexBuildError when loading a store built with v1.0.0 (missing batch_size in metadata) #203

Description

@jamie-ons

Summary

VectorStore.from_filespace() fails when loading a vector store whose metadata.json has "batch_size": null (as produced by v1.0.0, which did not persist batch_size). This happens when the batch_size argument is None which is the default argument.

Steps to Reproduce

Using the DEMO/general_workflow_demo.ipynb as a start we can add the following after creating my_vector_store:

# 1. Create a VectorStore and manually simulate a v1.0.0 metadata file (or any file missing the batch_size)
my_vector_store = VectorStore(
    file_name="data/testdata.csv",
    data_type="csv",
    vectoriser=vectoriser,
    output_dir="testdata",
    overwrite=True,
)

import json
with open("testdata/metadata.json", "r") as f:
    metadata = json.load(f)

metadata["batch_size"] = None  # simulates v1.0.0 metadata

with open("testdata/metadata.json", "w") as f:
    json.dump(metadata, f, indent=2)

# 2. Attempt to reload - raises IndexBuildError
VectorStore.from_filespace(folder_path="testdata", vectoriser=vectoriser)

Expected Behaviour

from_filespace() loads successfully, falling back to a sensible default batch_size when the metadata value is null.

Actual Behaviour

IndexBuildError: Failed to initialise VectorStore instance from filespace.
context={"cause_message": "type object 'VectorStore' has no attribute 'batch_size'", "cause_type": "AttributeError", ...}

Root Cause

In src/classifai/indexers/main.py, the from_filespace(), batch_size is assigned as:

vector_store.batch_size = batch_size if batch_size is not None else metadata["batch_size"]

When both batch_size (the argument) and metadata["batch_size"] are None, vector_store.batch_size is set to None. Any subsequent call to .search() then fails because batch_size is used as a range step, which does not accept None.

Fix

from_filespace() should fall back to the same default as __init__ (128) when both values are None. The default should be defined in a single place to avoid duplication.

Affected Versions

  • Introduced in: v1.1.0
  • Affects: any store created with v1.0.0 loaded by v1.1.0+

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions