Create index failed (terminated by signal 9: Killed)

[balazik]

Hi, this extension is exactly what I need for our nearest neighbor searches.
But it always fails when I try to create the index.

To reproduce:

• insert 5848000 float vectors with length 768
• create index with: CREATE INDEX ON <schema.table_name> USING ivfflat (<vector_column_name> vector_cosine_ops)
  exact query: CREATE INDEX ON nlp.vector_test USING ivfflat (embedding vector_cosine_ops);
• Get exception: [08006] An I/O error occurred while sending to the backend.

Logs:

LOG:  server process (PID 97127) was terminated by signal 9: Killed
DETAIL:  Failed process was running: CREATE INDEX ON nlp.vector_test USING ivfflat (embedding vector_cosine_ops)
LOG:  terminating any other active server processes
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.
LOG:  all server processes terminated; reinitializing
LOG:  database system was interrupted; last known up at 2021-05-24 11:24:42 CEST
LOG:  database system was not properly shut down; automatic recovery in progress
LOG:  redo starts at 5DD/9D2C8818
FATAL:  the database system is in recovery mode
redo done at 5DD/BE66DE20
database system is ready to accept connections

Details:
It seems that the index creation is eating all server memory (64GB). Then the system steps in (Ubuntu 20.04 LTS) and kills the process. I also tried to reduce the number of inverted lists to 10 but it doesn't help.

Do i need some partitioning or is this a bug ?

[ankane]

Hey @balazik, thanks for reporting! It looks like memory wasn't being freed correctly while the index was building. Can you see if the commit above fixes it?

Edit to add more info: Memory for the process building the index should stay roughly below maintenance_work_mem + shared_buffers.

SELECT pg_backend_pid();
SHOW maintenance_work_mem;
SHOW shared_buffers;

[balazik]

I deleted the VM machine where all tests were performed. Will recreate it and make new test, but it will take some time.

[ankane]

No worries, I just pushed 0.1.5 with the fix, but let me know if you still run into issues.

[balazik]

The new version works flawlessly!

Test:

• Made a new test with 5 848 000 float vectors (768 dims).
• ivfflat/vector_cosine_ops index creation query time: 23 min 23 sec (it run on slow VM with only 4 cores).
• Table total size (rows+index on disk): 45 GB
• 1 vector query time (limit to 10 results): 144ms

Mem:
Postgres process memory stays constant, Physical: ~217 MB, Shared: ~148MB
Where maintenance_work_mem = 64MB, shared_buffers = 128MB

Thank you for the quick fix and also for the great extension.

You can close the issue.

[ankane]

Awesome, thanks for checking and sharing the results!