[MRG+2] switch to multinomial composition for mixture sampling

[ljwolf]

Reference Issue

fixes #7701

What does this implement/fix? Explain your changes.

This changes the way mixture models construct the composition of new samples. Specifically, any subclass deriving fromBaseMixture.sample is affected.

Instead of rounding the composition vector from weights * n_samples, this draws the composition vector from a multinomial distribution. Thus, samples return are guaranteed to have the number of observations requested by the user. However, the composition of the sample is now stochastic.

In addition, this adds tests to ensure that n_samples are returned when mixture.sample(n_samples) is called.

This may affect scaling for mixture models with a very large number of dimensions, since the multinational composition draw may be slow. But, this composition draw only occurs once during sampling.

Any other comments?

None. Thanks for the great package!

[tguillemot]

Can you add a test with the example you've put in #7701 on the function test_sample in mixture/test_gaussian_mixture.py ?
@ljwolf Thanks.

[tguillemot]

Can you add a test with the example you've put in #7701 on the function test_sample in mixture/test_gaussian_mixture.py ?
@ljwolf Thanks.

[lesteve]

This may affect scaling for mixture models with a very large number of dimensions, since the multinational composition draw may be slow.

This may be a problem, not sure.

This reminds me the issue we had in StratifiedShuffleSplit #6472 where drawing samples for each group did not add up to the total number of samples. Although I didn't really understand the details I believe @amueller used some kind of approximation to avoid randomly sampling.

[lesteve]

This may affect scaling for mixture models with a very large number of dimensions, since the multinational composition draw may be slow.

This may be a problem, not sure.

This reminds me the issue we had in StratifiedShuffleSplit #6472 where drawing samples for each group did not add up to the total number of samples. Although I didn't really understand the details I believe @amueller used some kind of approximation to avoid randomly sampling.

[tguillemot]

This may affect scaling for mixture models with a very large number of dimensions, since the multinational composition draw may be slow.

It's not a problem. The function sample is not use during the fitting process.
Moreover, as weights is linked to the number of component and GaussianMixture is not designed to deal with a lot of component (less than 1e6), the time is really OK.

On my computer, for a weights of shape = (1000000, ) :

%timeit np.random.multinomial(100000000, weights).astype(int)
10 loops, best of 3: 135 ms per loop

[tguillemot]

This may affect scaling for mixture models with a very large number of dimensions, since the multinational composition draw may be slow.

It's not a problem. The function sample is not use during the fitting process.
Moreover, as weights is linked to the number of component and GaussianMixture is not designed to deal with a lot of component (less than 1e6), the time is really OK.

On my computer, for a weights of shape = (1000000, ) :

%timeit np.random.multinomial(100000000, weights).astype(int)
10 loops, best of 3: 135 ms per loop

[jnothman]

I think this is an acceptable use of rng.multinomial. Please add a test.

[amueller]

Can you please add a non-regression test? Otherwise looks good. Thanks for the fix!

[amueller]

Can you please add a non-regression test? Otherwise looks good. Thanks for the fix!

[ljwolf]

Yes, will do.

[ljwolf]

Yes, will do.

[ljwolf]

I don't think I have the correct permissions to restart the travis build, but it should be passing. Can a maintainer trigger a rebuild?

[ljwolf]

I don't think I have the correct permissions to restart the travis build, but it should be passing. Can a maintainer trigger a rebuild?

[jnothman]

As long as the test runs quite quickly, this LGTM.

[jnothman]

As long as the test runs quite quickly, this LGTM.

[jnothman]

Please update what's new

[jnothman]

Please update what's new

[ljwolf]

I just swapped to the range(1,50)construct and added the fact that the test was added to the original statement. That Travis report looks pretty strange... tons of errors that look like build errors?

[ljwolf]

I just swapped to the range(1,50)construct and added the fact that the test was added to the original statement. That Travis report looks pretty strange... tons of errors that look like build errors?

[jnothman]

Could you please try updating master and rebasing on it?

[jnothman]

Could you please try updating master and rebasing on it?

[ljwolf]

Alright, I've rebased to master & the tests are off to the races.

[ljwolf]

Alright, I've rebased to master & the tests are off to the races.

[tguillemot]

LGTM

[tguillemot]

LGTM

[lesteve]

I used n_components=3 for the test to test the actual regression seen in #7701.

As part of this it seems that n_features and n_components were swapped in a few places. @tguillemot can you quickly check whether what I changed makes sense.

[lesteve]

I used n_components=3 for the test to test the actual regression seen in #7701.

I forgot to say it explicitly: I used the recent github features that allows people with enough admin rights to push into PR branches. @ljwolf I hope you don't mind :-).

[lesteve]

I used n_components=3 for the test to test the actual regression seen in #7701.

I forgot to say it explicitly: I used the recent github features that allows people with enough admin rights to push into PR branches. @ljwolf I hope you don't mind :-).

[tguillemot]

@lesteve Sorry for these mistakes. It's n_components indeed.

[tguillemot]

@lesteve Sorry for these mistakes. It's n_components indeed.

[amueller]

LGTM

[amueller]

LGTM

[ljwolf]

@lesteve nope, that's what they're there for!

[ljwolf]

@lesteve nope, that's what they're there for!

[lesteve]

AppVeyor is taking quite some time, this PR should be merged if it comes back green.

[lesteve]

AppVeyor is taking quite some time, this PR should be merged if it comes back green.

[lesteve]

Merged, thanks a lot!

[lesteve]

Merged, thanks a lot!

[amueller]

@lesteve for future reference, please use the squash and merge feature, that makes cherry-picking much simpler.

[amueller]

@lesteve for future reference, please use the squash and merge feature, that makes cherry-picking much simpler.

[lesteve]

Oops sorry for that, I always try to remember to do squash and merge but it looks like I missed this one.

[lesteve]

Oops sorry for that, I always try to remember to do squash and merge but it looks like I missed this one.

[lesteve]

Looks like you can only allow squash and merge if you want to:
https://github.com/blog/2141-squash-your-commits

The settings are available from: https://github.com/scikit-learn/scikit-learn/settings

Should we do that?

[lesteve]

Looks like you can only allow squash and merge if you want to:
https://github.com/blog/2141-squash-your-commits

The settings are available from: https://github.com/scikit-learn/scikit-learn/settings

Should we do that?

[amueller]

@lesteve yes. I enabled that. Hm though sometimes we have multiple authors? Well, let's see when the first person complains. We can always go back. Thanks for finding that.

[amueller]

@lesteve yes. I enabled that. Hm though sometimes we have multiple authors? Well, let's see when the first person complains. We can always go back. Thanks for finding that.

[jnothman]

Whoops, we'd forgotten to tag this for 0.18.1. Done now.

[jnothman]

Whoops, we'd forgotten to tag this for 0.18.1. Done now.

[amueller]

hm sorry didn't make it into 0.18.1. My bad :(

[amueller]

hm sorry didn't make it into 0.18.1. My bad :(