This project presents an in-depth exploratory data analysis (EDA) of a dataset containing over 114,000 tracks from Spotify, sourced from Kaggle. The primary goal was to move beyond surface-level metrics and dive into the rich audio features that define a song's character.
Instead of building a predictive machine learning model, this study focuses on the crucial preceding step: identifying and understanding the dependencies between musical attributes. We sought to answer the fundamental question: How do objective audio features like energy, valence, and danceability relate to each other and influence a song's perceived success and potential for recommendation?
This work was completed as a data analysis project at Polytech Nice Sophia, focusing on extracting actionable insights from a real-world open dataset.
Spotify's recommendation engine is renowned for its ability to create seamless and often surprisingly relevant musical transitions. Our objective was to uncover the underlying data relationships that might enable such recommendations. We hypothesized that these transitions are not based on genre alone, but on a deeper "contextual similarity" rooted in a combination of audio features.
This analysis aims to:
- Model the relationships between key audio features.
- Identify the characteristics of popular songs.
- Simulate the kind of dependency analysis that could inform a sophisticated recommendation algorithm.
Our analysis revealed several key dependencies, moving from simple linear relationships to more complex interaction effects.
By analyzing our engineered features against song popularity, we identified a clear trend: popular tracks often have specific characteristics. They tend to be faster, louder, and have medium energy and danceability, suggesting a "sweet spot" for widespread appeal.
| Characteristics of Popular Songs |
| Feature | Popular if... | Interpretation :--- | :--- | :--- **Mood** | Sad | Emotional songs resonate, even if not "happy". **Danceability**| Medium | A track that is versatile and not overly complex. **Energy** | Medium | An optimal balance for daily listening—not too calm, not too intense. **Tempo** | Fast | Upbeat rhythms are engaging and align with current trends. **Loudness** | Loud | Higher volume often correlates with professional production and perceived energy. |
We first explored direct relationships between audio features.
- A strong, positive linear relationship was identified between
energyandloudness(Adjusted R² ≈ 0.58). This makes intuitive sense: louder tracks are perceived as more energetic. - A more complex, non-linear relationship was observed between
danceabilityandenergy. A polynomial regression revealed a parabolic trend (Adjusted R² ≈ 0.15), suggesting that the most danceable songs often have a moderate level of energy.
| Linear Regression: Energy vs. Loudness | Polynomial Regression: Danceability vs. Energy |
 |
 |
| As loudness increases, so does energy. | The inverted U-shape shows a peak in danceability at mid-range energy levels. |
We then investigated interaction effects to test if the relationship between two variables is moderated by a third. This revealed that a song's attributes do not exist in a vacuum; context is key.
Our most significant finding came from the model loudness ~ instrumentalness * energy_level. This interaction model explains 61.5% of the variance in loudness, a massive 41% improvement in explanatory power over a simple model that ignores the interaction.
Key Insight: The relationship between how instrumental a track is and its volume is heavily dependent on its energy level. This effect is much more pronounced in high-energy tracks (e.g., EDM) compared to low-energy tracks (e.g., ambient).
| Other Significant Interactions Uncovered |
| Model | Interaction R² (adj) | Improvement vs. Simple Model (ΔR²) | Key Insight :--- | :---: | :---: | :--- `loudness ~ valence * energy_level` | **52.3%** | **+44.0%** | The connection between a song's mood (`valence`) and its volume is significantly stronger in high-energy tracks. `energy ~ acousticness * mood_category` | **58.4%** | **+4.4%** | `Acousticness` is a much better predictor of low energy in 'sad' songs than in 'happy' ones. `loudness ~ acousticness * energy_level` | **52.6%** | **+17.6%** | How acoustic a song is impacts its loudness differently depending on its energy level. |
This exploratory analysis confirms that the relationships between a song's audio attributes are complex, often non-linear, and highly contextual.
- Popularity is multi-faceted: It's a specific combination of attributes that correlates with success.
- Dependencies are key to similarity: A recommendation engine likely relies on these inter-feature relationships to find contextually similar tracks.
- Interactions are powerful: The most valuable insights emerge when analyzing how variables influence each other. A model that ignores interactions (like
energy_levelmoderating theloudness-instrumentalnesslink) misses a huge part of the story and loses significant explanatory power.
This study demonstrates how a rigorous EDA process is vital for understanding the fundamental dynamics of a dataset before any machine learning is applied.
The project script follows a structured data processing and analysis pipeline:
- Data Loading & Initial Exploration: Reading the CSV file, displaying initial stats, and checking for missing values.
- Data Cleaning: Imputing missing values using robust methods (median for numerical, mode for categorical).
- Exploratory Data Analysis (EDA): Visualizing distributions, correlation matrices, and relationships between variables.
- Feature Engineering: Creating new, interpretable categorical variables (
mood_category,energy_level, etc.) from raw audio features. - Statistical Modeling: Splitting data and training Linear, Polynomial, and Interaction Regression models to quantify relationships and evaluate performance with metrics like R² and RMSE.
- Google Colab
-
Download .ipynb files to the script folder
-
Import these files into Google Colab and run them
- Place your dataset (e.g.,
dataset.csv) inside thedata/directory. - Configure and run the analysis script located in the
scripts/directory. The script will perform the full analysis pipeline, print statistical results, and save visualizations.
- Advanced Cleaning: Outlier detection and more sophisticated imputation techniques.
- Advanced Modeling: Testing non-linear models (Random Forest, Gradient Boosting) to capture even more complex relationships.
- Cross-Validation: Implementing K-Fold Cross-Validation for more robust model evaluation.
- Residual Analysis: Formally checking the assumptions of the linear regression models.
- Data Analysis: Pandas, NumPy
- Data Visualization: Matplotlib, Seaborn
- Statistical Modeling: Scikit-learn
- File Encoding Detection: Chardet
This project is licensed under the MIT License. See the LICENSE file for more details.