[Bug]: Vector Storage Not Persisting to RocksDB - Embedding Succeeds but Vector Count Remains 0

[ztonlly]

Bug Description

Issue: Vector Storage Not Persisting to RocksDB - Embedding Succeeds but Vector Count Remains 0

Summary

Priority: Critical
Type: Bug

Embedding generation works perfectly (4096-dimensional vectors, 0 errors), but vectors are never persisted to the RocksDB vector database. Vector Count remains at 0 even after successful embedding processing.

Steps to Reproduce

Environment

• OpenViking Version: v0.2.7
• Docker Image: ghcr.io/volcengine/openviking:latest
• Container OS: Debian-based
• Runtime: Python 3.13
• Backend: local (RocksDB)
• Embedding Model: NVIDIA nv-embed-v1 (4096-dim)
• API Key Provider: NVIDIA

Configuration

ov.conf

{
  "embedding": {
    "dense": {
      "api_base": "https://integrate.api.nvidia.com/v1",
      "api_key": "***",
      "provider": "openai",
      "model": "nvidia/nv-embed-v1",
      "dimension": 4096
    },
    "max_concurrent": 10
  },
  "storage": {
    "workspace": "/app/data",
    "vectordb": {
      "backend": "local",
      "name": "context",
      "dimension": 4096
    }
  }
}

collection_meta.json

{
  "CollectionName": "context",
  "Dimension": 4096,
  "VectorKey": "vector",
  "Fields": [...]
}

Reproduction Steps

1. Deploy OpenViking with v0.2.7:

   docker run -d \
     --name openviking \
     -p 1933:1933 \
     -v /mnt/m2_1/data/openviking/data:/app/data \
     ghcr.io/volcengine/openviking:latest

2. Configure NVIDIA embedding model (nv-embed-v1, 4096-dim)

3. Create a session with test messages:

   import requests
   
   BASE = 'http://localhost:1933/api/v1'
   API_KEY = 'ROOT_API_KEY'
   
   # Create session
   r = requests.post(f'{BASE}/sessions', headers={'X-API-Key': API_KEY})
   sid = r.json()['result']['session_id']
   
   # Add message
   msg = {'role': 'user', 'parts': [{'type': 'text', 'text': 'Test message'}]}
   requests.post(f'{BASE}/sessions/{sid}/messages',
                headers={'X-API-Key': API_KEY, 'Content-Type': 'application/json'},
                json=msg)
   
   # Commit
   requests.post(f'{BASE}/sessions/{sid}/commit',
                headers={'X-API-Key': API_KEY, 'Content-Type': 'application/json'},
                json={'archived': True})

4. Wait 20-30 seconds for embedding processing

5. Check observer status:

   curl http://localhost:1933/api/v1/observer/queue -H "X-API-Key: ROOT_API_KEY"
   curl http://localhost:1933/api/v1/observer/vikingdb -H "X-API-Key: ROOT_API_KEY"

Expected Behavior

Actual Behavior

Queue Status:
+----------------+---------+-------------+-----------+--------+-------+
| Queue          | Pending | In Progress | Processed | Errors | Total |
+----------------+---------+-------------+-----------+--------+-------+
| Embedding      | 0       | 0           | 12        | 0      | 12    |
+----------------+---------+-------------+-----------+--------+-------+

Vector DB Status:
+------------+-------------+--------------+--------+
| Collection | Index Count | Vector Count | Status |
+------------+-------------+--------------+--------+
| context    | 1           | 0            | OK      |
+------------+-------------+--------------+--------+

Observations:

• ✅ Embedding succeeds: 12 tasks processed, 0 errors
• ❌ Vector Count: 0 (no vectors stored)
• 📂 RocksDB files: ~12KB total (no actual vector data)
• 📝 QMD files: Generated successfully (~75 MD files)

Expected Behavior

Vector DB Status (Expected):
+------------+-------------+--------------+--------+
| Collection | Index Count | Vector Count | Status |
+------------+-------------+--------------+--------+
| context    | 1           | 12           | OK      |
+------------+-------------+--------------+--------+

Actual Behavior

Root Cause Analysis

Code Modifications Tested

We modified collection_schemas.py to handle dense-only embeddings (nv-embed-v1 doesn't produce sparse vectors):

# File: openviking/storage/collection_schemas.py
# Lines 211-216

# Add sparse vector if present
if result.sparse_vector:
    inserted_data["sparse_vector"] = result.sparse_vector
    logger.debug(f"Generated sparse vector with {len(result.sparse_vector)} terms")
else:
    logger.debug("No sparse vector (dense-only embedding)")

Verified Configurations

• ✅ storage.vectordb.dimension: 4096
• ✅ embedding.dense.dimension: 4096
• ✅ collection_meta.json: Dimension = 4096
• ✅ Embedding returns 4096-dimensional vectors
• ✅ NVIDIA API calls succeed with 0 errors

Attempted Solutions

1. Code Patches:

   • Disabled dimensions parameter in openai_embedders.py (NVIDIA API doesn't support it)
   • Made sparse_vector optional in collection_schemas.py
   • Set 4096 as default dimension

2. Database Reconstruction:

   • Completely cleared and rebuilt /app/data/vectordb/
   • Replaced collection_meta.json with correct 4096-dim config
   • Removed RocksDB lock files

3. Container Restart:

   • Multiple restarts with configuration changes
   • Fresh deployments with clean state

All attempts resulted in the same behavior: Embedding succeeds, Vector Count = 0.

Hypothesis

The issue appears to be in the vector persistence layer:

1. _vikingdb.upsert() in collection_schemas.py (line 230) is called
2. The upsert() method may fail silently without throwing exceptions
3. The RocksDB write operation may be blocked or fail due to internal locking/transaction issues
4. No error logs are generated despite Vector Count remaining 0

Logs

OpenViking Logs (last 20 minutes)

time="2026-03-19T14:13:16Z" level=info msg="mounted serverinfofs at /serverinfo"
time="2026-03-19T14:13:16Z" level=info msg="[queuefs] Initialized with backend: memory"
time="2026-03-19T14:13:16Z" level=info msg="mounted queuefs at /queue"
time="2026-03-19T14:13:16Z" level=info msg="[localfs] Initialized with base path: /app/data/viking"
time="2026-03-19T14:13:16Z" level=info msg="mounted localfs at /local"

Note: No errors, warnings, or debug logs related to vector storage failures.

Impact

Critical: Vector similarity search is completely non-functional despite:

• Successfully configured 4096-dimensional embeddings
• Zero embedding processing errors
• Proper QMD file generation

Workaround: Full-text search only (fts-only mode), losing semantic vector search capabilities.

Additional Context

• Workaround: Using QMD file-based retrieval with full-text search
• Related Issues: None found in existing issues
• Alternative Tested: Direct calls to upsert() fail with "IO error: lock /app/data/vectordb/context/store/LOCK: Resource temporarily unavailable"

Suggested Fix

Investigate the vector persistence flow in:

1. collection_schemas.py -> on_dequeue() -> upsert()
2. viking_vector_index_backend.py -> upsert() -> _adapter.upsert()
3. local_adapter.py -> get_collection() -> RocksDB write operations

Ensure that:

• Exceptions are properly caught and logged
• Lock acquisition failures are surfaced
• Vector data validation is checked before write
• Transaction commit status is verified

System Info

$ docker exec openviking python3 -c "import openviking; print(openviking.__version__)"
v0.2.7

$ docker exec openviking python3 -c "import sys; print(sys.version)"
3.13.0

References

• NVIDIA nv-embed-v1 API: https://integrate.api.nvidia.com/v1
• Expected vector dimension: 4096 (fixed,不接受dimensions参数)
• Dense-only embedding (no sparse vectors produced)

---

Minimal Reproducible Example

Error Logs

OpenViking Version

0.2.7

Python Version

3.13

Operating System

Linux

Model Backend

None

Additional Context

No response

[zhoujh01]

Thanks for the feedback. Version 0.2.7 has a bug. We recommend switching to the latest version, v0.2.9.