Ensemble Classification Project

Project Overview

This project explores ensemble classification methods by building a custom implementation of a model ensemble based on:

• Training multiple classifiers on bootstrap samples from the training dataset.
• Using a random subset of attributes for each model.
• Combining predictions through majority voting.

The results are compared against:

• Single classifiers (e.g., DecisionTreeClassifier, GaussianNB)
• Standard ensemble methods available in Python/R (BaggingClassifier, RandomForestClassifier)

Additionally, two approaches to probability prediction are evaluated:

1. predict_proba → Averaging probabilities returned by individual models.
2. predict_proba2 → Estimating probabilities based on the frequency of class votes.

---

Datasets

1. Car Evaluation Dataset

• Source: UCI ML Repository
• Attributes: 6 categorical features.
• Target: Car evaluation (unacc, acc, good, v-good).
• Imbalanced distribution (70% in unacc).
• Preprocessed using ordinal encoding.

2. Airline Passenger Satisfaction Dataset

• Source: Kaggle
• Attributes: 22 features (categorical + continuous).
• Target: Passenger satisfaction (satisfied, neutral or dissatisfied).
• Balanced dataset (~43% satisfied).
• Preprocessing included binning continuous values, categorical mapping, and handling missing values.

---

Custom Ensemble Method

A dedicated class ModelsBand implements the custom ensemble approach.

Core Features

• Supports DecisionTreeClassifier, GaussianNB, and CategoricalNB.
• Bootstrap sampling for diversity in training data.
• Random attribute subset selection for each model.
• Voting-based prediction and two probability estimation methods.
• Compatible with scikit-learn’s cross_validate for benchmarking.

Example: Creating and Training the Ensemble

from sklearn.tree import DecisionTreeClassifier
from sklearn.naive_bayes import GaussianNB, CategoricalNB
from models_band import ModelsBand  # if saved into a module

# Load Car dataset
df_car_data = pd.read_csv("/content/input_ready/CarEvaluationDataSet/carData.csv")
Y_car_data = df_car_data["Evaluation"]
X_car_data = df_car_data.drop(columns=["Evaluation"])

# Train ensemble
ensemble = ModelsBand(
    model="DecisionTreeClassifier",
    model_size=10,
    bootstrap_size=500,
    attributes_size=4,
    column_y="Evaluation"
)
ensemble.fit(X_car_data, Y_car_data)
predictions = ensemble.predict(X_car_data)

Probability Estimation

• predict_proba: averages probabilities from each model.
• predict_proba2: estimates probabilities from class vote frequencies.

---

Helper Functions

Several utility functions were implemented to assist in experimentation:

• print_scores(scores, **kwargs) – displays cross-validation results.
• itarate_over_args(kwargs), generate_new_args(args, best_args, key, value) – used for parameter iteration.
• benchmark(X, Y, delta, **kwargs) – automatic parameter search using cross-validation with accuracy as the metric.

Example Parameters Tested

• Car Dataset:

  • bootstrap_size = [100, 1000]
  • attributes_size = [3, 4, 5, 6]
  • model_size = [5, 7, 9, ..., 20]

• Airline Dataset:

  • bootstrap_size = [100, 1000, 10000, 100000]
  • attributes_size = [11, 13, ..., 22]
  • model_size = [5, 7, 9, ..., 20]

---

Baseline Experiments

Experiment 1: DecisionTreeClassifier

• CarEvaluationDataSet: accuracy = 79.4%
• AirlinePassengerDataSet: accuracy = 94.5%

These serve as baselines for ensemble comparisons.

---

Experiment 2: DecisionTreeClassifier vs Ensemble

This experiment compares a single DecisionTreeClassifier with an ensemble of decision trees implemented via the ModelsBand class. The main goal is to check whether the ensemble generalizes better than a single tree.

CarEvaluationDataSet

To find optimal ensemble parameters, the benchmark function was used. Three parameters were optimized in sequence:

1. model_size (number of models in ensemble)
2. attributes_size (number of features per bootstrap sample)
3. bootstrap_size (number of rows per bootstrap sample)

The optimization used delta = 0.01 (1%), meaning accuracy improvements below 1% were ignored in favor of faster computation.

Optimal parameters found:

{'model': 'DecisionTreeClassifier',
 'model_size': 5,
 'attributes_size': 6,
 'bootstrap_size': 1000,
 'column_y': 'Evaluation'}

Result: accuracy = 86.2%

• This is ~7% higher than a single decision tree (79.4%).

---

AirlinePassengerDataSet

The same optimization process was applied.

Optimal parameters found:

{'model': 'DecisionTreeClassifier',
 'model_size': 19,
 'attributes_size': 11,
 'bootstrap_size': 10000,
 'column_y': 'satisfaction'}

Result: accuracy = 94.8%

• This is ~0.3% higher than a single decision tree (94.5%).

---

Summary of Experiment 2

• On the Car dataset, the ensemble showed a significant improvement (~7%).
• On the Airline dataset, the improvement was modest (~0.3%) but still confirmed that ensembles can slightly enhance generalization.

---

Experiment 3: GaussianNB vs Ensemble

Goal: Compare a single GaussianNB classifier with an ensemble of GaussianNB models to test whether bootstrapping and attribute subsampling improve results.

Results:

• CarEvaluationDataSet:

  • Single model: ~76.6%
  • Ensemble: ~76.7%

• AirlinePassengerDataSet:

  • Single model: ~87.5%
  • Ensemble: ~87.5%

Conclusion: The ensemble provided no noticeable improvement. GaussianNB is stable, and additional sampling did not increase classification accuracy.

---

Experiment 4: CategoricalNB vs Ensemble

Goal: Evaluate CategoricalNB on categorical data and check whether the ensemble improves performance.

Results:

• CarEvaluationDataSet:

  • Single model: ~81.3%
  • Ensemble: ~81.3%

• AirlinePassengerDataSet:

  • Single model: ~82.8%
  • Ensemble: ~82.8%

Conclusion: Similar to GaussianNB, the ensemble did not provide any benefits. Results were nearly identical for both approaches.

---

Experiment 5: BaggingClassifier

Goal: Compare the custom ensemble of decision trees with the built-in BaggingClassifier.

Results:

• CarEvaluationDataSet:

  • BaggingClassifier: ~86.3%
  • Custom ensemble: ~86.2%

• AirlinePassengerDataSet:

  • BaggingClassifier: ~94.7%
  • Custom ensemble: ~94.8%

Conclusion: Our implementation achieved almost identical results to BaggingClassifier, confirming the correctness of our approach.

---

Experiment 6: RandomForestClassifier

Goal: Compare the decision tree ensemble with RandomForestClassifier, which introduces additional randomness in feature selection during tree construction.

Results:

• CarEvaluationDataSet:

  • RandomForest: ~92.7%
  • Custom ensemble: ~86.2%

• AirlinePassengerDataSet:

  • RandomForest: ~96.1%
  • Custom ensemble: ~94.8%

Conclusion: RandomForest significantly outperformed the custom ensemble. This is due to the additional randomness in feature selection at each split, showing the advantage of more advanced mechanisms over plain bagging.

---

Experiment 7: Comparing predict_proba and predict_proba2

Goal: Evaluate which probability estimation method performs better:

• predict_proba: average of probabilities returned by individual models.
• predict_proba2: proportion of class votes across the ensemble.

Results:

• CarEvaluationDataSet:

  • predict_proba: ~86.2%
  • predict_proba2: ~86.2%

• AirlinePassengerDataSet:

  • predict_proba: ~94.8%
  • predict_proba2: ~94.8%

Conclusion: Both methods produced identical results. In this case, the choice of probability calculation method did not affect classification accuracy.