Social Media Analysis Dashboard

!R !Python !Dash

This project demonstrates an end-to-end data analysis pipeline that processes social media data (tweets) and presents the findings in an interactive web dashboard. The core data processing is performed in R, simulating a MapReduce workflow for efficient, parallelized analysis. The resulting insights are then visualized using a Python Dash application.

The primary goal is to showcase a full data lifecycle: from raw data generation and cleaning to distributed processing and final visualization.

---

🖼️ Dashboard Preview

(Suggestion: Take a screenshot of your running dashboard, save it in the project directory as dashboard_preview.png, and replace the line below with !Dashboard Preview)

A live preview of the dashboard will appear here.

---

✨ Key Features

• Interactive Visualizations: The dashboard provides dynamic charts for exploring trends and patterns in the data.
• MapReduce Simulation: Leverages R's parallel and foreach libraries to simulate a distributed MapReduce architecture, breaking down large analysis tasks into parallel map and reduce steps.
• Comprehensive Analysis:
  • Text Analysis: Identifies the most frequent words in the dataset.
  • User Engagement: Ranks users by their total engagement (likes + retweets).
  • Sentiment Analysis: Tracks the daily average sentiment score over time.
• Key Performance Indicators (KPIs): A summary panel displays high-level metrics like total tweets, unique users, and average engagement.

---

⚙️ Project Workflow

The project follows a clear, multi-stage workflow:

1. Data Generation (Optional): A sample dataset of tweets is generated using an R script (01_generate_sample_data.R).
2. Data Preprocessing: The raw text data is cleaned and standardized in R to prepare it for analysis. This includes removing punctuation, converting to lowercase, and handling special characters.
3. MapReduce Processing (R): The core analysis is performed using three simulated MapReduce jobs:
   • Job 1: Word Count: Counts the frequency of each word across all tweets.
   • Job 2: Daily Sentiment: Calculates the average sentiment score for each day.
   • Job 3: User Engagement: Aggregates the total likes and retweets for each user.
4. Data Persistence: The results from the MapReduce jobs are saved as .csv files.
5. Dashboard Visualization (Python): A Python script (full_dashboard.py) reads the processed .csv files and uses the Dash framework to build and serve an interactive web dashboard.

---

🚀 Getting Started

Follow these instructions to get the project running on your local machine.

Prerequisites

Make sure you have the following software installed:

• Git
• R (version 4.0.0 or newer) and an IDE like RStudio
• Python (version 3.8 or newer) and pip

Step 1: Clone the Repository

git clone https://github.com/your-username/social-media-dashboard.git
cd social-media-dashboard

(Replace your-username with your actual GitHub username.)

Step 2: Run the R Analysis Pipeline

The Python dashboard relies on data generated by the R scripts. You must run them to create the necessary .csv files.

1. Generate Sample Data (First time only):

   • Open scripts/01_generate_sample_data.R in RStudio.
   • Run the script. This will create a raw_tweets.csv file in your project's root directory.

2. Install R Packages:

   • Open any of the R scripts and install the required packages listed at the top (e.g., dplyr, parallel, foreach, syuzhet). You can do this by running install.packages("package_name") in the R console.

3. Run the Processing Scripts in Order: Execute the following scripts sequentially. Each script performs a specific task and prepares data for the next stage.

   Script	Purpose	Output Files
   01_preprocessing.R	Cleans the raw tweet text.	tweets_cleaned.csv
   03_data_preparation_for_mapreduce.R	Prepares the cleaned data for the MapReduce jobs.	(Intermediate step)
   04_mapreduce_simulation.R	Runs the three core MapReduce jobs for word count, sentiment, and user stats.	mapreduce_word_count.csv, mapreduce_daily_sentiment.csv, mapreduce_user_stats.csv
   05_sentiment_analysis.R (Optional)	Performs a more advanced sentiment and emotion analysis using different lexicons.	sentiment_results.csv

   After running these, your project directory will contain all the .csv files needed for the dashboard.

Step 3: Set Up the Python Environment

It is highly recommended to use a Python virtual environment to manage dependencies.

# Create a virtual environment named 'venv'
python -m venv venv

# Activate the virtual environment
# On Windows:
.\venv\Scripts\activate
# On macOS/Linux:
# source venv/bin/activate

# Install the required Python packages
pip install -r requirements.txt

Step 4: Launch the Dashboard

With the data generated and the Python environment ready, you can now start the web application.

python full_dashboard.py

You will see a message in your terminal with a local URL. Open your web browser and navigate to:

http://127.0.0.1:8050/

You should now see the interactive Social Media Analysis Dashboard!

---

📂 Project Structure

SocialMediaProject/
├── scripts/
│   ├── 01_generate_sample_data.R   # Generates raw sample data
│   ├── 01_preprocessing.R          # Cleans the raw data
│   ├── 03_data_preparation_for_mapreduce.R # Prepares data for MapReduce
│   ├── 04_mapreduce_simulation.R   # Runs the core MapReduce jobs
│   └── 05_sentiment_analysis.R     # Performs advanced sentiment analysis
├── .gitignore                      # Specifies files for Git to ignore
├── full_dashboard.py               # The main Python Dash application
├── README.md                       # This file
├── requirements.txt                # Python dependencies
└── raw_tweets.csv                  # (Generated by R script)
└── *.csv                           # (Generated by R scripts)