Introduction & Motivation •
Limitations •
Build •
Credit and License •
This project was last updated on 31/10/2023
This project is a web application that collates the "Areas of Expertise" that live on individual staff member profiles into a single page and allows them to be filtered with a free text box and department dropdown.
It is made up of a collection of web scraping python scripts which are run daily by a GitHub Action, this generates a JSON file. The updated file is picked up by Netlify which hosts the website.
The front end is a React web application that pulls the data out of the JSON into an HTML table.
There are quite a few limitations to this work. As it is a web scraper it is dependent on the Swansea University website for its information, if the website structure changes it will stop working.
This application covers the Faculty of Humanities and Social Sciences at Swansea University.
To build this project you will need to have Node.js and npm installed (should come bundled with node). You will also need to have Python 3 installed to run the data scraping scripts along with a number of libraries that can be installed with pip (should come bundled with python).
pip install flake8 pytest requests beautifulsoup4 datetime tqdm
You can then run the data scraping scripts from the scripts folder using python scrape_data.py.
With the JSON file generated you can generate the web application using npm install, once all the requisite
dependencies are installed you can start the application using npm start. This will launch the application at
http://localhost:3000.