Update protobuf definition

[lintool]

Per discussion in #4

CIFF to play with:

• Robust04, query terms only
• Robust04, all postings

[andrewtrotman]

I'm happy with the one file solution.

[andrewtrotman]

all the names are plurals except "total_postings_list". Can we make that "total_postings_lists"

[lintool]

@andrewtrotman good catch - will do.

[andrewtrotman]

I understand how num_postings_lists differs from total_postings_lists, the fist being the number in the file, the second being the number in a full index. But what is total_terms_in_collection, and how does it differ from total_postings_lists? Especially since total_postings_lists is defined as being "the vocabulary size".

[lintool]

Updated exports:

• Robust04, query terms only
• Robust04, all postings

[lintool]

I understand how num_postings_lists differs from total_postings_lists, the fist being the number in the file, the second being the number in a full index. But what is total_terms_in_collection, and how does it differ from total_postings_lists? Especially since total_postings_lists is defined as being "the vocabulary size".

total_terms_in_collection is, literally, how many tokens there are in the entire collection, i.e., the sum of all document lengths of all documents. How would you suggest I clarify the wording?

[lintool]

Alternatively, would total_postings_lists be more clear as vocab_size?

[andrewtrotman]

Now I understand what it is - thanks. You might call it sum_of_document_lengths

[lintool]

Suggested rename by @andrewtrotman : sum_of_document_lengths

[JMMackenzie]

I think Andrew wanted primary_key instead since it may be more clear? @andrewtrotman

[lintool]

I prefer collection_docid, which has semantic meaning. primary_key is logical, not semantic.

[lintool]

@andrewtrotman @JMMackenzie suggestions adopted.

[lintool]

Once everyone is happy I'll generate fresh exports.

[cmacdonald]

Two comments:

// The average document length. We store this value explicitly in case the exporting application wants a particular
// level of precision.
double average_doclength = 7;

This is obtainable from parsing the postings lists, no? Given it is provided, should an implementation use it? I'm not sure I follow the motivation for its presence.

string collection_docid = 2;  

a long time ago Terrier standardised nomenclature on docno for the external String for the primary key of the document, as this corresponded to the DOCNO tag in a TREC collection. docid is the integer internal to the system (or CIFF) in this case. I would suggest using the same nomenclature to avoid needing to clarify docid vs docno.

[lintool]

Two comments:

// The average document length. We store this value explicitly in case the exporting application wants a particular
// level of precision.
double average_doclength = 7;

This is obtainable from parsing the postings lists, no? Given it is provided, should an implementation use it? I'm not sure I follow the motivation for its presence.

I think the rationale is this: "We store this value explicitly in case the exporting application wants a particular level of precision" as noted in comments. Suppose the exporting application rounds avgdl to the nearest integer... it could use this field to tell the importer.

Don't feel too strongly though...

string collection_docid = 2;  

a long time ago Terrier standardised nomenclature on docno for the external String for the primary key of the document, as this corresponded to the DOCNO tag in a TREC collection. docid is the integer internal to the system (or CIFF) in this case. I would suggest using the same nomenclature to avoid needing to clarify docid vs docno.

I'm okay with docno, although it is a bit counter-intuitive, since "no" connotes an number, which is definitely isn't.

[chriskamphuis]

looks good to me

[andrewtrotman]

Make sure to make total_terms_in_collection clearer with a better comment