Tech Spike - Database evaluation

[mihaisc]

In the current format, elastic search does the job but we have the following problem, it is resource-intensive.
We need to find an alternative that can run with 1 or 2 cores and a minimum of 2-4 GB of RAM. It needs to be a NoSQL document database.

Potential candidates:
https://github.com/zincsearch/zincsearch
https://manticoresearch.com/blog/manticore-alternative-to-elasticsearch/

suggested steps:

• create a script to generate 1M DDoS: copy one and randomly generate the name and tags
• inject ddos in databases and review results
• create scripts to query the DBS: search, aggregate, filter by tags ( common things that we do in the market)

[an7e]

Solution: Elasticsearch
Uploaded data: 1m DDO's
Requests made: 20k
cpu_mem report: dowload
requests report: dowload

---

Solution: zincsearch
Uploaded data: 1m DDO's
Requests made: 20k
cpu_mem report: dowload
requests report: dowload

---

Solution: typesense
Uploaded data: 1m DDO's
Requests made: 20k
cpu_mem report: dowload
requests report: dowload
Note: pre-defined schema is required, but can be skiped over to auto-schema detection

---

Solution: manticoresearch
Note: table schema is required, adding documents to a real-time table will fail because of MVA
error: 'MVA elements should be integers'

---

Solution: meilisearch
Note: uses a lot of RAM when indexing documents, error when trying to upload a large number of documents

---

Solution: solr
Note: Java-based, scheme required

[mihaisc]

The conclusion is we are going to give typesense a shot, it looks the most promissing.

[an7e]

https://docs.google.com/spreadsheets/d/1pMCx0dB70NbwHqw2-yqriW_g69YBqqKo_LUkiqqZoSU/edit?usp=sharing