Simple search

Now all our articles have been indexed, we can search for them in our index! To search for an article, you simply submit a query which gives us a response containing all articles that were deemed relevant.

To do this we use the submitted query to create a dis_max query, which chooses the single best matching field, and ignores the other. However, we use the tie_breaker parameter to take the _score from the other matching clauses into account as well. This gives us a reponse containing the articles mentioned above. We then iterate over these articles, summarising each articles text to relevant sentences using an nltk tokeniser, while also highlighting relevant terms. This is then shown on the SERP page below the title of each article.

dis_max query

def simple_query(query):
    dis_max = {
        "query": {
            "dis_max": {
                "queries": [
                    { "match": { "title": query }},
                    { "match": { "body": query }}
                ],
                "tie_breaker": 0.3
            }
        }
    }
    return dis_max

Simple search

def simple_search(query):
    es = Elasticsearch()
    es.indices.refresh(index="telegraaf")
    res = es.search(index="telegraaf", body=simple_query(query))

    for hit in res["hits"]["hits"]:
        hit["_source"]["text"] = summarise(query, hit["_source"]["text"])

    return res

Summarising the article

def summarise(query, text):
    tokenizer = nltk.data.load("nltk:tokenizers/punkt/dutch.pickle")
    summarisation = ""
    for sentence in tokenizer.tokenize(text):
        if any([word in sentence for word in query.split()]):
            print sentence.encode("utf-8"), word
            sentence = sentence.replace(word, "<b>{}</b>".format(word).encode("utf-8"))
            summarisation += sentence + " "
        if len(summarisation.split()) > 50:
            break

    if (len(summarisation) < 1):
        return " ".join(text.strip().split()[:50])
    else:
        return summarisation.strip()