Application that converts the tweets from the trending topics into news
Warning 08/01/2019: This application is not working since the update of the Twitter UI. It needs to be adapted to support it.
This application is divided in 2 parts:
- Client: the responsable of showing up the created news in a pretty way.
- Server: the responsable of gathering the necessary information to generate the news and then create them.
The server uses Selenium to go through the trending topics of Twitter. Then, using web scrapping, it obtains all the information of each tweet of each trending topic. This information is processed, the hashtags, links and other relevant things are extracted to have a pure text that can be used to generate the news. After the text analysis is done, the creation of the news starts. The text of the tweets of each trending topic are joined to create a big text that will be used as the base from where the program will create the news. Using the Markovify's implementation of Markov Chains and spaCy a Natural Language Processing software, sentences are created and putted together to form the news. Finally, a JSON file is created with the news of each trending topic.
The client gets the JSON and using React with Semantic UI it shows the news in a web page in a pretty way.
After download the application you have to install the necessaries things to run it.
For the Client:
- Install Node JS
- Inside the client folder run:
npm install
For the Server:
- Download the chrome driver from here and set an environment variable for it
- Install Python 3
- Install pip (probably it's already installed because of the instalation of Python)
- Install virtualenv:
pip install virtualenv
- Go into the server folder of the project and run:
virtualenv venv
source venv/bin/activate
- Install the necessary dependencies running:
pip install -r requirements.txt
Requirement: A Twitter account
Server: To run the server go into the src folder inside the server folder and run
python3 start.py username password
Note: The username and the password are the username (or mail) and password of the Twitter account.
Client: To run the client go to the client folder and run
npm start
Note: Should not be necessary to manually run it. The client is automatically called after the server finishes its work.