Term occurs in document vector, but has collection frequency 0

[PepijnBoers]

I've found a term that occurs once in a document vector, but doesn't occur in the collection. Am I using the wrong analyzer or is this a bug? I've used the following Pyserini functions:

index_utils = pyutils.IndexReaderUtils('/Index/lucene-index.core18.pos+docvectors+rawdocs_all')
tf = index_utils.get_document_vector(docid)
analyzer = pyanalysis.get_lucene_analyzer(stemming=False, stopwords=False)
df = {term: (index_utils.get_term_counts(term, analyzer=analyzer))[1] for term in tf.keys()}

output:

tf = {.. 'hobbies:photographi': 1, ..}
df = {.. 'hobbies:photographi': 0, ..}

I assume the term is derived from this part in the raw text: "..<b>HOBBIES:</b>Photography..."

[lintool]

Per this: https://github.com/castorini/pyserini/#usage-of-the-index-reader-api

# Fetch the document vector:
doc_vector = index_utils.get_document_vector('FBIS4-67701')
# Result is a dictionary where the keys are analyzed terms (i.e., the stemmed form that 
# was actually indexed) and the values are the term frequencies.
print(doc_vector)

Yes, "hobbies:photographi" is a (poorly) stemmed (i.e., analyzed) form.

[lintool]

A bit more detail:

analyzer = pyanalysis.get_lucene_analyzer(stemming=False, stopwords=False)
df = {term: (index_utils.get_term_counts(term, analyzer=analyzer))[1] for term in tf.keys()}

The above code snippet isn't going to work because the index wasn't (at least by default) built using the analyzer config... so it's not going to find the term... and if it does, it's a coincidence of stemmed/non-stemmed forms matching.

[PepijnBoers]

Thank you for your response! I'm still not getting it completely though. As far as I understand; the document vector contains the stemmed form of each term, so by iterating over the keys in the document vector I'm looping over the stemmed versions.

You mention that here as well:

doc_vector = index_utils.get_document_vector('FBIS4-67701')
# Result is a dictionary where the keys are analyzed terms (i.e., the stemmed form that 
# was actually indexed) and the values are the term frequencies.

Then when I'm calling the index_utils.get_term_counts(term, analyzer), I'm searching the index, which to my understanding also contains stemmed terms, therefore I do not want to stem the term again and use a dummy analyzer.

Maybe the misunderstanding originates from whether the index contains stemmed terms or not. I've built my index using the following flags:

./target/appassembler/bin/IndexCollection -collection WashingtonPostCollection \
 -input /Volumes/Samsung_T5/WashingtonPost.v2/data -generator WashingtonPostGenerator \
 -index lucene-index.core18.pos+docvectors+rawdocs_all \
 -threads 1 -storePositions -storeDocvectors -storeRaw -optimize

[lintool]

Hrm, I think you're right! What's the docid in core18? Let me take a look at it.

[PepijnBoers]

docid: 1c7fe012-df9d-11e3-9743-bb9b59cde7b9

[lintool]

Yup, you're right, there's a bug here.

from pyserini.analysis.pyanalysis import get_lucene_analyzer, Analyzer
analyzer = get_lucene_analyzer(stemming=False, stopwords=False)

index_utils.get_term_counts('hobbies:photographi', analyzer)
# fails: (0, 0)

index_utils.get_term_counts('hobbies\:photographi', analyzer)
# works: (1, 1)

index_utils.get_term_counts('hobbies:photography')
# fails: (0, 0)

index_utils.get_term_counts('hobbies\:photography')
# works: (1, 1)

What's happening is that a:b is getting interpreted by Lucene as a field query, i.e., where "a" is the field name.

This is because we run the query through a query parser:
https://github.com/castorini/anserini/blob/master/src/main/java/io/anserini/index/IndexReaderUtils.java#L210

We shouldn't.

Although this does the right thing:

postings_list = index_utils.get_postings_list('hobbies:photography')
for posting in postings_list:
    print(f'docid={posting.docid}, tf={posting.tf}, pos={posting.positions}')

This requires a batch to Anserini and then a new maven artifact deploy. I'll get on it.

Thanks for catching the bug!

[lintool]

@PepijnBoers please take a look: castorini/anserini#1135

+1 with it if you're happy.

[lintool]

Closed with #86 and in v0.9.1.0 release.