(Team 4) Design: Keyword match operator

[chenlica]

Team 4:

Please do the following:

• Put the design to https://github.com/TextDB/textdb/wiki/CS290-2016S-Task:-Keyword-Match-Operator
• Use this ticket to keep track of the progress.

Add @prakul to this issue.

[chenlica]

Per our discussion in the lecture today, please do the following:

• Design your feature(s) as operators.
• Come up with a few good test cases.
• Update this issue with your progress.

Please contact me to schedule a F2F meeting to discuss the details. Also I wonder whether you are still interested to solve the problem of "finding documents similar to a document."

[chenlica]

@akshaybetala and @prakul : please finish the documentation and performance test. Thanks.

[chenlica]

The initial performance numbers (time) for index-based search operator were very high (https://github.com/TextDB/textdb/wiki/CS290-2016S-Task:-Keyword-Match-Operator).

Here's the answer of @prakul :

The 'lucene query time' includes the time for preparing lucene query 
(in our custom function) from string, initializing the scan operator and 
obtaining tokens for the query based on our choosen analyzer. These 
are one off time expenses which our system is incurring for first query. 
For example : for 1M dataset, 

The time take for query : "medicine history" ->

Lucene Query time: 6.8490 seconds

is getting down to 

lucene Query time: 3.1050 seconds

on second run. 

My reasoning is that our implementation across operators is incurring this 
cost because of various initializations which lucene probably does only 
once and caches.

I still wonder why the overhead is so high. @sandeepreddy602 @rajesh9625 @inkudo @zuozhi for their thoughts.

[rajesh9625]

@chenlica

These are the results I got when I ran DictionaryMatcher: index-based search operator(PHRASEOPERATOR) on my machine:

Machine configuration : MacBook Pro, 2.7 GHz Intel Core i5, 8 GB 1867 MHz DDR3

Performance results for DictionaryMatcher with PHRASEOPERATOR:

Dictionary : {"medical","medication","medicare","medicaid"}
Lucene Query time: 0.5950 seconds
Match time: 5.6970 seconds
Total: 36528 results

Lucene Query time for me has always been under a second.

[zuozhiw]

I just ran the query "medicine" with keyword operator on 1 Million records. The numbers I get is pretty normal:
Lucene Query time: 0.4100 seconds
match time: 2.0910 seconds
total: 9114 results

So is "7192.9650 seconds" the real number or a typo?

And are we talking about "Lucene Query time" here or the total time in general?

[chenlica]

@rajesh9625 : please include more interesting entities in the dictionary, with more varieties and multiple-keyword entities.

@zuozhi : I think we are talking about total query time, since that's what a user experiences.

[chenlica]

This task is done.