IamNikolayTverdokhleb / inverted_index Public

Notifications You must be signed in to change notification settings
Fork 0
Star 1

Vector Space Model and Language Model for Information Retrieval system based on collections of text documents.

1 star 0 forks Branches Tags Activity

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 16 Commits
.gitignore		.gitignore
CranReader.py		CranReader.py
LanguageModel.py		LanguageModel.py
Preprocessing.py		Preprocessing.py
README.md		README.md
VectorSpaceModel.py		VectorSpaceModel.py
main.py		main.py

Repository files navigation

inverted_index

That is document search engine.

For now we created reader for CRANFIELD http://ir.dcs.gla.ac.uk/resources/test_collections/cran/, which provides a set of documents and corresponding queries, as well as ranking of queries.

Preprocessing class provides methods for tokenizing and stemming of both queries and documents, as well as removing stop-words from them.

There's implementation of Vector Space Model with TF-IDF space and Language Model with 1-gram query likelyhood.

Note: VectorSpaceModel provides methods write_inverted_file read_inverted_file to write and read inverted file from memory, but our search machine appears to be really fast and efficient on chosen dataset so we are not using it.

About

Vector Space Model and Language Model for Information Retrieval system based on collections of text documents.

iar langauge-model query-likelihood query-doc-similarity space-vector-model

Report repository

Releases

No releases published

Packages

No packages published

Contributors 2

Languages

Python 100.0%