How to "Raw" Index using Anserini?

[Mistobaan]

To make sure I understand and master the full pipeline I want to create the tokens from the documents myself (i.e. skipping the built-in stemming and tokenization) and index the generated tokens using the Lucene index.
My questions:

• What is the best example to follow?
• Is there any support in python?
• Any gotchas to be aware in using the Lucene index? (i.e. some inner optimization that prevents arbitrary sequence of characters to be valid terms).
• Can I directly specify the weight of each term or has to be the TF-IDF?

[lintool]

Hi @Mistobaan see https://github.com/castorini/pyserini/#how-do-i-search-my-own-documents

You'll need to disable stopword removal and stemming, and tokenize only on spaces. This is doable in Lucene but I'm not sure the right hooks are exposed in Python.

No, you cannot specify the weight of each term.

Hope this helps!

[Mistobaan]

Thank @lintool. Yes definitely helped in finding this document:

• https://github.com/castorini/pyserini/blob/master/docs/usage-analyzer.md
  The core of the logic in breaking up the text is inside the Lucene Analyzer.
  I found these analyzers:
• DefaultEnglishAnalyzer
• LSHAnalyzer
• Freebase
• FakeWordsEncoderAnalyzer
• TweetAnalyzer

With the new keyword/term (Lucene Analyzer) I was able to find out the Lucene feature to make the custom term frequency: https://issues.apache.org/jira/browse/LUCENE-7854

• https://lucene.apache.org/core/7_4_0/core/org/apache/lucene/analysis/tokenattributes/TermFrequencyAttribute.html
  and a good document that explains more of the internal objects:
  https://lucene.apache.org/core/4_0_0/core/org/apache/lucene/search/package-summary.html#scoring

Now the follow-up question is: If I create the custom/index in anserini/lucene how does it translate to ElasticSearch (to have distributed computing)?
I see that there is some sort of support for elastic search but does it just send the raw JSON document through ElasticSearch' API and let ES re-create the index? or does it send chunks of the serialized index?
I see:

• the various options for elastic search
• then seems it create a simple rest client
• connects to the collection
• creates a bulk index request
• IndexRequest is an ES Type
• Create and populate the ES XContentBuilder
  So if I can read the code well, to answer my own question, the ES indexer sends only some of the fields and only string/int or arrays. Now I am still not sure if ES will re-tokenize those and how I will have to read the code on that side to find out how those objects are stored/indexed. It seems that the XContentBuilder has a lot of capabilities and switches to populate the fields.

[lintool]

@Mistobaan

I see that there is some sort of support for elastic search but does it just send the raw JSON document through ElasticSearch' API and let ES re-create the index?

This ^^^^ There's no way for ES to "import" a Lucene index directly. It needs to re-index from scratch.

So, you'll need to set the tokenizer appropriately from the ES end, by twiddling with the ES configs, like here: https://github.com/castorini/anserini/blob/2d8359c917c0a54d2b239cd02e289b3b4790a6bb/src/main/resources/elasticsearch/index-config.cord19.json

[Mistobaan]

Thanks!