Machine Learning Experiments Repository

This repository contains a collection of machine learning projects covering various fundamental concepts and techniques. Each project demonstrates different aspects of machine learning including classification, regression, regularization, and language modeling.

Repository Structure

The repository is organized into four main projects:

Requirements

Each project has specific dependencies, but the core requirements can be downloaded by running the below command in the root directory (alternatively see each individual folder’s README/requirements.txt to download that specific folder’s requirements only):

pip install -r requirements.txt

For the language modeling project, additional requirements are provided in its dedicated requirements.txt file.

Project Components

1. Binary Classification with Logistic Regression and SVM
   Demonstrates implementation of binary classification models using logistic regression and support vector machines (SVM) on 2D data.
   Key Features:

   • Basic logistic regression with linear decision boundaries
   • SVM implementation with parameter tuning (C values)
   • Feature engineering by adding squared terms
   • Enhanced visualizations with decision boundaries
   • Model accuracy comparison against baseline classifiers
     Dataset: my_sample_data.csv with two features and binary target values (1, -1)

2. Binary Classification Techniques
   Explores various binary classification algorithms with comparison of their performance characteristics.
   Key Features:

   • Baseline classification models for performance benchmarking
   • K-Nearest Neighbors implementation with tunable k values
   • Logistic regression with polynomial features
   • ROC curve analysis and confusion matrices
   • Cross-validation for hyperparameter tuning
     Datasets: sample_binary_data_1.csv and sample_binary_data_2.csv

3. Polynomial Regression with Regularization
   Implements polynomial regression models with Ridge and Lasso regularization techniques.
   Key Features:

   • Polynomial feature generation
   • Lasso regression analysis and visualization
   • Ridge regression analysis and visualization
   • Cross-validation for regularization parameter tuning
   • 3D visualization of prediction surfaces
     Dataset: sample_data.csv with two features and continuous target values

4. Language Modeling with GPT Implementation
   Contains a GPT-like language model implementation for text generation and analysis.
   Key Features:

   • Transformer-based model architecture
   • Text preprocessing and tokenization
   • Transfer learning between different text corpora
   • Skip connection analysis
   • Bias analysis visualization
     Datasets:
     • input_childSpeech_trainingSet.txt
     • input_childSpeech_testSet.txt
     • input_shakespeare.txt

Key Findings

Classification Models

• Feature engineering significantly improves classification performance
• Higher regularization (lower C values) creates simpler decision boundaries with better generalization
• KNN performance varies significantly with choice of k value
• Polynomial features enable capturing non-linear relationships

Regression Models

• Degree 5 polynomials capture complex non-linear relationships in the data
• Lasso tends to produce sparse models by setting some coefficients exactly to zero
• Ridge regression shrinks all coefficients but typically keeps them non-zero
• Cross-validation reveals the bias-variance tradeoff when selecting regularization strength

Language Modeling

• Transfer learning enables applying models trained on one corpus to another domain
• Skip connections impact model performance and gradient flow
• Potential biases exist in model predictions based on training data

Visualizations

Each project includes extensive visualizations to help understand model behavior:

• Classification Projects: Scatter plots with decision boundaries, ROC curves, confusion matrices
• Regression Project: 3D scatter plots, prediction surfaces, error plots, coefficient value tables
• Language Modeling: Data statistics plots, bias analysis visualizations, architecture diagrams

How to Run

Navigate to any project directory and run individual scripts:

# For example, to run data visualization in the binary classification project
cd Binary_Classification_with_Logistic_Regression_and_SVM
python data_visualization.py

# To run the GPT implementation
cd Language_Modeling_with_GPT_Implementation
python gpt.py

python data_visualization.py

For more detailed information about each project, refer to the README files in their respective directories.

License

This project is open-source and available for educational purposes.