No description, website, or topics provided.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
LDA_parameter_setting
data_files
visualisation
0_generate_AK_IDs.ipynb
1_generate_publications_list.ipynb
2a_article_scraper.ipynb
2b_pdf_to_txt.md
3_data_preprocessing.ipynb
4_LDA.ipynb
README.md
licence.md
topic_model_blog.md

README.md

Topic modeling (Latent Dirichlet Allocation) of Turing Institute publications

Aim:

Identify topics researched at the Alan Turing Institute

Overview:

For more details see the blog post.

In summary, for each Turing fellow we downloaded up to 50 articles they published since 2012 (inclusive) using open access routes. The downloaded articles were converted to text files and cleaned. We analysed the text data using Latent Dirichlet Allocation (LDA) which identified 25 separate research topics in the dataset. The result is a visualisation of each Turing fellow and the extent to which their work falls under any of the identified research topics.

Procedure:

1: Identifying Turing fellows' publications

We used Microsoft's Academic Knowledge (AK) database to extract publications records for each Turing fellow. We first manually extracted a list of unique AK IDs for all fellows to get around entity matching issues to ensure we are targeting the right researchers in the database. We then used these IDs to retrieve a list of article titles, DOIs and source URLs.

Method: 1_generate_publications_list.ipynb
Results: publications_eng.csv

2: Publications download

(i) We built a web scraper to download each of the articles identified in step 1 (relying on open access routes)
Method: 2a_article_scraper.ipynb

(ii) The downloaded PDFs were converted to .txt files (using Contentmine's NORMA tool)
see 2b_pdf_to_txt.md for description

3: Data pre-processing

All data was cleaned and combined into final analysis-ready csv file
The final csv consists of 1 article per row and contains: fellow name, ID and affiliation, article title and ID, article full text and length, URLs to versions of the article available online and AK provided keywords associated with the article.
Method: 3_data_preprocessing.ipynb
Data in 2 parts: final_dataset_1.csv and final_dataset_2.csv

4: Analysis

The converted texts were analysed using LDA
Method: 4_LDA.ipynb
Results (topic proportions by researcher, used for visualisation purposes): data_original.csv

5: Visualisation

A visualisation of the final topics was built using an adaptation of the aster plot (see visualisation folder)