Upsert API fails with graph config while performing after /delete

[akset2X]

Getting the following weird error when I do upsert operation from the /upsert API. I am doing it with graph configuration.

#config.yml
# Index file path
path: ./tmp/index

# Allow indexing of documents
writable: True

# Enbeddings index
embeddings:
  path: sentence-transformers/all-MiniLM-L6-v2
  content: True
  functions:
  - name: graph
    function: graph.attribute
  expressions:
  - name: category
    expression: graph(indexid, 'category')
  - name: topic
    expression: graph(indexid, 'topic')
  - name: topicrank
    expression: graph(indexid, 'topicrank')
  graph:
    limit: 15
    minscore: 0.1
    topics:
      categories:
      - Society & Culture
      - Science & Mathematics
      - Health
      - Education & Reference
      - Computers & Internet
      - Sports
      - Business & Finance
      - Entertainment & Music
      - Family & Relationships
      - Politics & Government

"/index" API is working fine, but I want to upsert index, so that I can keep my old emeddings. Is there any solution?

INFO: Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO: 127.0.0.1:57206 - "POST /batchsearch HTTP/1.1" 200 OK
INFO: 127.0.0.1:57206 - "POST /delete HTTP/1.1" 200 OK
INFO: 127.0.0.1:57206 - "POST /add HTTP/1.1" 200 OK
INFO: 127.0.0.1:57206 - "GET /upsert HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "c:\users\ak\appdata\local\programs\python\python39\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 407, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "c:\users\ak\appdata\local\programs\python\python39\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 78, in call
return await self.app(scope, receive, send)
File "c:\users\ak\appdata\local\programs\python\python39\lib\site-packages\fastapi\applications.py", line 270, in call
await super().call(scope, receive, send)
File "c:\users\ak\appdata\local\programs\python\python39\lib\site-packages\starlette\applications.py", line 124, in call
await self.middleware_stack(scope, receive, send)
File "c:\users\ak\appdata\local\programs\python\python39\lib\site-packages\starlette\middleware\errors.py", line 184, in call
raise exc
File "c:\users\ak\appdata\local\programs\python\python39\lib\site-packages\starlette\middleware\errors.py", line 162, in call
await self.app(scope, receive, _send)
File "c:\users\ak\appdata\local\programs\python\python39\lib\site-packages\starlette\middleware\exceptions.py", line 79, in call
raise exc
File "c:\users\ak\appdata\local\programs\python\python39\lib\site-packages\starlette\middleware\exceptions.py", line 68, in call
await self.app(scope, receive, sender)
File "c:\users\ak\appdata\local\programs\python\python39\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 21, in call
raise e
File "c:\users\ak\appdata\local\programs\python\python39\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 18, in call
await self.app(scope, receive, send)
File "c:\users\ak\appdata\local\programs\python\python39\lib\site-packages\starlette\routing.py", line 706, in call
await route.handle(scope, receive, send)
File "c:\users\ak\appdata\local\programs\python\python39\lib\site-packages\starlette\routing.py", line 276, in handle
await self.app(scope, receive, send)
File "c:\users\ak\appdata\local\programs\python\python39\lib\site-packages\starlette\routing.py", line 66, in app
response = await func(request)
File "c:\users\ak\appdata\local\programs\python\python39\lib\site-packages\fastapi\routing.py", line 237, in app
raw_response = await run_endpoint_function(
File "c:\users\ak\appdata\local\programs\python\python39\lib\site-packages\fastapi\routing.py", line 165, in run_endpoint_function
return await run_in_threadpool(dependant.call, **values)
File "c:\users\ak\appdata\local\programs\python\python39\lib\site-packages\starlette\concurrency.py", line 41, in run_in_threadpool
return await anyio.to_thread.run_sync(func, *args)
File "c:\users\ak\appdata\local\programs\python\python39\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "c:\users\ak\appdata\local\programs\python\python39\lib\site-packages\anyio_backends_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "c:\users\ak\appdata\local\programs\python\python39\lib\site-packages\anyio_backends_asyncio.py", line 867, in run
result = context.run(func, *args)
File "c:\users\ak\appdata\local\programs\python\python39\lib\site-packages\txtai\api\routers\embeddings.py", line 85, in upsert
application.get().upsert()
File "c:\users\ak\appdata\local\programs\python\python39\lib\site-packages\txtai\api\base.py", line 80, in upsert
super().upsert()
File "c:\users\ak\appdata\local\programs\python\python39\lib\site-packages\txtai\app\base.py", line 400, in upsert
self.embeddings.upsert(self.documents)
File "c:\users\ak\appdata\local\programs\python\python39\lib\site-packages\txtai\embeddings\base.py", line 180, in upsert
self.graph.upsert(Search(self, True))
File "c:\users\ak\appdata\local\programs\python\python39\lib\site-packages\txtai\graph\base.py", line 415, in upsert
self.infertopics()
File "c:\users\ak\appdata\local\programs\python\python39\lib\site-packages\txtai\graph\base.py", line 541, in infertopics
topic = Counter(self.attribute(x, "topic") for x in ids).most_common(1)[0][0]
TypeError: 'NoneType' object is not iterable

What I think is.. it is related to #421
I think this issue may be caused by /delete API, because when I freshly did /add and /upsert a batch of text content, it worked fine. But when I /delete some of it, then /add and /upsert. I faced the mentioned issue.

[davidmezzetti]

Ok, thank you for the additional context on this and linking to #421.

[akset2X]

Any update here ?

Is there any option to configure topic generation to avoid stopwords such as "has", "you", "yourself" etc or some tagging options to get only proper nouns NN, JJ ? Please let me know, if there is no such option for now, it would be great if we could get those in future.

[davidmezzetti]

Sorry, it's on my list to look at this after the 5.4 release goes out.

[davidmezzetti]

I'll be checking in a fix for this shortly. There was a bug when deleting from the index and topics went to 0.

Regarding stopwords, see the stopwords configuration option: https://neuml.github.io/txtai/embeddings/configuration/#topics