LatentDirichletAllocation (LDA) score grows negatively, while it seems that GridSearchCV would want to make it grow

[guillaume-chevalier]

Description

The score of the LatentDirichletAllocation (LDA) seems inverted. The GridSearchCV seems to maximize score, but for the LDA it seems that the score is better if it is more negative.

Steps/Code to Reproduce

1. Build a Pipeline() from a CountVectorizer() and an LDA()
2. Pass the pipeline to a GridSearchCV()
3. The thing doesn't seem to work, and after manual inspection, the score seems to increase in the wrong direction (negatively) instead that it should be positively increasing.

I also tried to implement a class inheriting the LDA to negate the score, but the score is internally used in the LDA class itself, so overriding that seemed to change nothing at all. So either no learning is made for a bad Grid Search, or either no learning is made because of an early stopping of a score not improving the way it was.

Expected Results

If I'm wrong, the documentation should be clearer on wheter or not the GridSearchCV does reduce or increase the score. Also, there should be a better description of the directions in which the score and perplexity changes in the LDA. Obviously normally the perplexity should go down. But the score goes down with the perplexity going down too. And I'd expect a "score" to be a metric going better the higher it is.

Actual Results

# Tried to print this with a few different optimization round, the score grows negatively with a reduction of perplexity. 
print(lda.score(train), lda.perplexity(train))
# -182.51029019702543   27.615270885701467
# -52.05494670051718    41.19061662552155   
# -41.973876576664225  189.9450029464927    
# -32.432059761463556 3320.9791408284505    

Versions

Linux-4.17.0-amd64-x86_64
Python 3.6.6 (default, Jun 27 2018, 14:44:17) 
[GCC 8.1.0]
NumPy 1.14.0
SciPy 1.1.0
Scikit-Learn 0.19.2

[jnothman]

Do you find that score and perplexity are always inversely related?

[guillaume-chevalier]

In other words, when the perplexity is less positive, the score is more negative.

Normally, perplexity needs to go down. Making it go down makes the score go down too. I believe that the GridSearchCV seeks to maximize the score. Thus it doesn't work. I tried to multiply by -1 the output score by overriding the method, but it didn't work either, because the score method is used internally by the LDA algorithm for something. If I remember well, the score was used internally for something like early stopping, so inverting score as a quick fix didn't work.

If I did no mistakes here and if my assumptions are right, then the score would need a refactoring for the LDA to be useable by GridSearchCV. And/Or else providing good alternative metrics for scoring this unsupervised algorithm would be a good thing too.

[miguel2488]

I'm also having this problem. I can't optimize a sklearn LDA model using anything similar to grid search. Hyperband won't work, skopt won't work either, and all the autoML tools like TPOT, auto-sklearn, etc, won't do the job as they don't work on unsupervised learning. I really think this a problem that needs to be tackled. Any help on how to fix this will be much appreciated. I need to optimize an LDA model.

If you know any optmization techniques imilar to GridSearchCV that could work on LDA, or if there's some custom scoring function that could be passed the GridSearchCV to make it work on LDA, please post it.

Thank you very much for your attention

[ogrisel]

The docstring of LatentDirichletAllocation.score states:

Calculate approximate log-likelihood as score.

And indeed the .score method of estimators in scikit-learn should always be "higher is better". So I think this is a bug and this method should be updated to return the average negative log likelihood (the average, instead of sum, is useful to make this invariant to the sample size in expectation which can be useful to compare the training set score and the test score even if they do not have the same size).

[PavloFesenko]

And indeed the .score method of estimators in scikit-learn should always be "higher is better". So I think this is a bug and this method should be updated to return the average negative log likelihood

@ogrisel, isn't the model performing better when the log likelihood is maximized (and hence, the log loss is minimized)? Then the method score is defined correctly, isn't it?

What I find weird in the example provided by @guillaume-chevalier is that perplexity is smaller for more negative log likelihood while according to the documentation and the code this shouldn't be the case:

Perplexity is defined as exp(-1. * log-likelihood per word)

I haven't tested the code myself but I can have a look at this issue.

[PavloFesenko]

I haven't tried GridSearchCV but since the below test ran well, I believe that GridSearchCV would too.

First, I initialized the dataset from the documentation on Latent Dirichlet Allocation:

from sklearn.decomposition import LatentDirichletAllocation
from sklearn.datasets import make_multilabel_classification
​
X, _ = make_multilabel_classification(n_classes=5, random_state=0)

Then I ran the following code twice for different max_iter and got the results below:

lda = LatentDirichletAllocation(n_components=5, random_state=0, max_iter=0)
lda.fit(X)
print(f'score {lda.score(X)}, perplexity {lda.perplexity(X)}')

• max_iter=0: score -17749.09763631513, perplexity 37.78455714707963
• max_iter=10: score -14899.469022580823, perplexity 21.089949665885182

As you can clearly see, larger score (log likelihood) corresponds to smaller perplexity which is in contradiction to the author's original example. @guillaume-chevalier or @miguel2488, could you please provide your code where GridSearchCV wasn't working with LatentDirichletAllocation?

[guillaume-chevalier]

Whoa, 4 years have passed since I asked this! Given the popularity of Scikit-Learn, I find it unusual that there is still some mystery related to this question.

For the info, here is a link to my corresponding stack overflow question as well:

• https://stackoverflow.com/questions/51735011/should-the-perplexity-or-score-go-up-or-down-in-the-lda-implementation-of

Here is what I was coding when I raised the present GitHub issue:

• https://github.com/ArtificiAI/Multilingual-Latent-Dirichlet-Allocation-LDA

More especially, it might be the following version of these files at a specific commit:

• https://github.com/ArtificiAI/Multilingual-Latent-Dirichlet-Allocation-LDA/blob/d43f8049577ccd715eb73cc233c9d026bfe8edc5/LDA-Pipeline-Tutorial.ipynb
• https://github.com/ArtificiAI/Multilingual-Latent-Dirichlet-Allocation-LDA/blob/d43f8049577ccd715eb73cc233c9d026bfe8edc5/app/logic/lda.py

This commit is 2 days after opening the present issue. The previous commit I did in the same project could be of interest as well, that was the same day of asking the question as well.

Note: I will probably not reply to more comments on the present issue and on StackOverflow on this topic, and let the community manage this if it needs to be solved furthermore. The response to my StackOverflow question is also only partially satisfying to me in the context I was asking it. I could accept a better StackOverflow answer or an edit to the current answer to clarify it.