Add Elasticsearch `articles` table

[FrancescoCasalegno]

Context

• In the past, we had a SQL database with the two tables sentences and articles.
  • sentences contained an article sentence per row, plus a field (FOREIGN KEY) allowing to identify in articles from which article the sentence was coming from.

    Search/src/bluesearch/entrypoint/database/schemas.py

    Lines 55 to 56 in f77d8c1

    	Table(
    	"sentences",

  • articles contained metadata info on a article per row (e.g. year of publication, journal, ...)

    Search/src/bluesearch/entrypoint/database/schemas.py

    Lines 36 to 37 in f77d8c1

    	Table(
    	"articles",

• After Semantic search + Exact matches in ElasticSearch #610 we decided to use Elasticsearch instead of a SQL db to store our texts (i.e. sentences table).
• Moreover, Question-Answering systems need to work with contexts longer than a single sentence, so rather than sentences we want now a paragraph table (but still with a FOREIGN KEY linking each row to the article it comes from).
• After moving the paragraph table to Elasticsearch, it becomes natural to have also the article table on Elasticsearch, rather than on a separate SQL database.

Actions

• Create an articles table on Elasticsearch, with the same schema (here) of the SQL table we had until now.
• Perform some testing to make sure that it is possible to perform joins in Elasticsearch, i.e. retrieve the article rows from articles table given the article_ids of some rows in paragraphs, and vice versa retrieve all paragraphs given some article_ids.

[drsantos89]

tables created as per:

client.indices.create(
    index="articles",
    body={
        "settings": {"number_of_shards": 2,
                     "number_of_replicas": 1},
        "mappings": {
            "dynamic": "strict",
            "properties": {
                "article_id": {"type": "keyword"},
                "doi": {"type": "keyword"},
                "pmc_id": {"type": "keyword"},
                "pubmed_id": {"type": "keyword"},
                "title": {"type": "text"},
                "authors": {"type": "text"},
                "abstract": {"type": "text"},
                "journal": {"type": "keyword"},
                "publish_time": {"type": "date", "format": "yyyy-MM-dd"},
                "license": {"type": "keyword"},
                "is_english": {"type": "boolean"},
            }
        },
    }
)

and

client.indices.create(
    index="paragraphs",
    body={
        "settings": {"number_of_shards": 2,
                     "number_of_replicas": 1},
        "mappings": {
            "dynamic": "strict",
            "properties": {
                "article_id": {"type": "keyword"},
                "section_name": {"type": "keyword"},
                "paragraph_id": {"type": "short"},
                "text": {"type": "text"},
                "is_bad": {"type": "boolean"},
                "embedding": {
                    "type": "dense_vector",
                    "dims": 384,
                    "index": True,
                    "similarity": "dot_product"
                }
            }
        },
    }
)

sentence fields were removed from the paragraph index

[drsantos89]

SQL-style join queries are not supported in ES.
There are a few alternatives:

• use an article ID that is then introduced as a field in the paragraphs table.
• use nested query
• use has_child and has_parent queries

https://www.elastic.co/guide/en/elasticsearch/reference/current/joining-queries.html

It is not possible to run an approximate kNN search on a dense_vector field within a nested mapping. So the nested query is not a good solution for our use case.

I have tested the article ID solution and performs well. The notebook (test_join) is available on the experimental search GitLab repo.

[FrancescoCasalegno]

Update 2022-09-13

Actions

• Populate tables with ~10 articles.
• Ask for reviews to test a couple of queries against such tables (e.g. retrieve paragraphs from articles published after year XXX, ...)

[drsantos89]

With the notebook below one can test the articles and paragraphs index:
https://github.com/BlueBrain/Search/blob/feature/add-notebook-to-test-es-tables/src/bluesearch/k8s/test_es_tables.ipynb

• @FrancescoCasalegno
• @EmilieDel
• @jankrepl