An internet search engine written mostly in python. Currently TF-IDF based.
It should be noted that the file naughty-words.txt is sourced from an external project and is licensed under the terms of the Creative Commons Attribution 4.0 International License. This is unlike the rest of this project, which is licensed under the terms of GNU General Public License V3.0.
Preforms initial tasks for setting up an instance. Use the --user option to specify a privileged MariaDB/MySQL user.
Purges old data from database after it reaches a specific age. Should be run as a cron job periodically.
Webcrawler. Crawls the internet for new URLs and outputs its findings to a file called 'raw-urls.txt'. The --sites option can be used to pass a list of URLs separated by commas.
Meta data extractor / web scraper. Extracts meta data from html and stores it in a MariaDB database. Uses the 'raw-urls.txt' file as input. The --key option must be used to provide the path to the key that decrypts db_creds.gpg
Preforms a TF-IDF based search against the database. Run search.py --help for more information.
Directory containing source files for website user interface / front end.
Outputs a results page.