Pydantic v2 Elasticsearch

[prrao87]

Updates

Updated Elasticsearch API to use Pydantic v2. In a similar way to Meilisearch, we are able to attach a running event loop to a multi-process pool of CPUs that can perform Pydantic validation and bulk indexing concurrently on multiple batches. The Elasticsearch async client allows us to efficiently do this in a non-blocking manner. The timing numbers on an M2 macbook pro are shown below.

$ cd dbs/elasticsearch/scripts
$ time python bulk_index.py
Found index wines in db, skipping index creation...

Processing chunks
Processed ids in range 1-10000
Processed ids in range 10001-20000
Processed ids in range 50001-60000
Processed ids in range 60001-70000
Processed ids in range 30001-40000
Processed ids in range 20001-30000
Processed ids in range 40001-50000
Processed ids in range 90001-100000
Processed ids in range 70001-80000
Processed ids in range 80001-90000
Processed ids in range 120001-129971
Processed ids in range 100001-110000
Processed ids in range 110001-120000
Finished execution!
python bulk_index.py  7.03s user 0.82s system 115% cpu 6.787 total