Twitter API

A framework to collect Twitter data using the official Twitter API.

• Stream of public tweets
• User’s followers

Dependencies

Node.JS and NPM

Make sure you have the latest version of Node.js installed on your machine. You can download Node.js here: NodeJS. Note: Node.js comes with NPM

Install

Clone or Download this project

Install project dependencies

• Open Terminal
• Navigate to the folder you saved this project.
• execute: npm install
• create a .env file to save your credentials
• Rename the folder config-sample to config

Twitter Credentials

To use this framework you need first to obtain a Twitter Developer Account: https://developer.twitter.com/

On your .env file add the following:

twitter_consumer_key=your_consumer_key
twitter_consumer_secret=your_consumer_secret
twitter_access_token=your_access_token
twitter_access_token_secret=your_access_token_secret

MongoDB

Tweets collected with the Stream connector are directly saved into a MongoDB database. Followers information collected through the Followers conenction can also be saved into a MongoDB database.

Install and run locally: https://www.mongodb.com/download-center/community Or set up a remote server, like https://www.mongodb.com/cloud/atlas

As a way to manage MongoDB, I suggest using MongoDB Compass: https://www.mongodb.com/download-center

MongoDB config

On your .env file add the following:

MONGODB_URI=mongodb://127.0.0.1:27017/your_database

Note: Change useLocalDB to "false" to use your remote server

Stream

Stream Dependencies

• MongoDB (See above)

Define hashtags to follow

• Edit config.stream.json:
• Add a list of hashtags or keywords to track.
• Separate the hashtags into different channels (collections).

e.g.:

{
    "channel1": [
        "#hastag1",
        "#hastag2",
        "any_keyworkd"
    ],
    "channel2": [
        "#hastag3"
    ]
}

Note: Tweets from different channels are saved on separated collection in the database.

Run Stream

In the folder's project, run: env-cmd node stream.js

Tweets will be saved to your MongoDB.

Followers

Define the users you want to get data from

Edit config.followers.json with a list of usernames.

e.g.:

    {
        "useJSON": true, [true|false]
        "useDB": false, [true|false]
        "users": [
            "username1",
            "username2"
        ]
    }

Run Followers

In the folder's project, run: node env-cmd node followers.js

Note: You can override the config file passing usernames directly.

e.g.: env-cmd node followers.js —-users=username1,username2

Results

--useJSON=true|false Save results to JSON file (default: true) Results are saved in the folder ./results/followers.

--useDB=true|false Save result to a MongoDB. Must setup first. Check isntruction above.