Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
boot
config
docs
model_tests
technews_nlp_aggregator
tests
.gitignore
README.md
SETUP.md
boot_web.py
config.yml
config_dev.yml
create_test_data.py
create_train_data.py
db_coords_template.yml
deploy_new_model.sh
do_boost.py
do_predict.py
doc2vec_model_wf.py
exec_test_run.sh
gram_export_wf.py
gram_model_wf.py
only_update_model_dev.sh
pack_new_model.sh
pack_new_model_dev.sh
package-lock.json
restart_app.sh
restart_app_dev.sh
scrape_site.py
scrape_urls.py
similar_articles_eff.py
tfidf_dictionary_wf.py
tfidf_matrix_wf.py
tfidf_model_wf.py
token_model_wf.py
update_boost.sh
update_boost_dev.sh
update_model.sh
update_model_dev.sh
v_11.yml
v_13.yml

README.md

TECH NEWS AGGREGATOR

General purpose

The idea is to build a technology news portal aggregating tech news from several sources and draw conclusions from them

  • find articles discussing the same news item as a provided text, but from a different source
  • get a different angle about a specific news item or topic from a different source
  • find controversial news in a period of time, that have been treated seperately from several sources.

Dataset

Currently the dataset includes technical articles from the year 2017 and 2018, mainly from these source

The database contains around 80.000 articles and is growing by the day. It is updated almost every day with the latest news.

Currently deployed

Prototype

The web application is currently accessible here.

Similar articles and conteroversial topics

You can find "controversial" topics in the database. You can browse through the articles that have been paired because the system considers it likely that they are talking about the same topic, possibly under a different slant.

Articles about the same stories can be found on the first page aggregated here.

Finding Related articles

You can copy the text of a technical article into this text page and find similar articles in the dataset, possibly limiting the search in a date interval.

Alternatively, you can see how articles in the database related to each other entering the url of an article in this page.

Video introduction

A video introduction to this project can be found on my Youtube channel.