question on exercise03 from phases/01-math-foundations/10-dimensionality-reduction

[jubeater]

PCA Elbow Question

as the exercise quote:

Take a dataset with 50 features where only 5 are informative (generate one with sklearn.datasets.make_classification). Apply PCA and check whether the explained variance curve correctly identifies that the data is effectively 5-dimensional.

I ran PCA on a synthetic dataset created with sklearn.datasets.make_classification using:

• n_samples = 5000
• n_features = 50
• n_informative = 5

Then I plotted the cumulative explained variance ratio from PCA.

What I observed

• The curve increases smoothly and looks almost linear.
• I do not see a clear elbow near 5 components.
• I also tried sklearn PCA and got the same result, so the issue does not seem to be my PCA implementation.

What I want to ask

Why does the explained variance curve look so linear in this case?

Is it because:

• n_informative = 5 only means 5 features are useful for predicting the label, not that the data is actually 5-dimensional in the PCA sense?
• PCA measures variance, not label informativeness(only use X not y in my case)?

Are there ways to generate or transform the dataset so that PCA shows a clearer elbow near 5 components like different make_classification settings ?

If not, what method could I use to be able to find the number of informative features is 5 from the generated datasets?

Current cumulative variance output

[0.79001528 0.80883941 0.81594031 0.82063076 0.82527467 0.82985169
 0.83439929 0.83889553 0.8433546  0.84779575 0.85217141 0.85650316
 0.8607992  0.86506236 0.86929372 0.87349747 0.87768152 0.88182503
 0.88595956 0.89007338 0.89415834 0.89822124 0.90224452 0.90625165
 0.91022895 0.91418556 0.91811477 0.92201461 0.92588088 0.92970524
 0.93349004 0.9372697  0.94101989 0.94474171 0.94843535 0.95211684
 0.95578337 0.9594258  0.96302392 0.96661829 0.97016817 0.97367212
 0.97716275 0.98063472 0.98406645 0.98743854 0.99074949 0.99403477
 0.99727858 1.        ]

Code I ran using sklearn

import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import make_classification
from sklearn.decomposition import PCA

X_sample, y_sample = make_classification(
    n_samples=5000,
    n_features=50,
    n_informative=5,
    n_redundant=0,
    n_repeated=0,
    n_clusters_per_class=1,
    class_sep=10.0,
    flip_y=0.0,
    shuffle=False,
    random_state=42,
)

pca_sample = PCA(n_components=50)
pca_sample.fit(X_sample)

cum_ratio = np.cumsum(pca_sample.explained_variance_ratio_)
print(f"list of the cum ratio: {cum_ratio}")
plt.figure(figsize=(8, 5))
plt.plot(range(1, 51), cum_ratio, marker="o")
plt.axhline(0.95, color="red", linestyle="--", label="95% variance")
plt.xlabel("Number of components")
plt.ylabel("Cumulative explained variance")
plt.title("PCA Elbow Plot")
plt.grid(True)
plt.legend()
plt.show()

[skyrain888]

@claude 有问题吗