cvpartition hasn't the same behavior of matlab version

[achengli]

Hello,

I was using cvpartition when I saw that cvpartition in Octave using KFold = 10 is giving groups of random sizes while with matlab the result was groups of same size or almost the same size when the number of observations is not divisible by the number of partitions.

The code I tried:

rng (1); % for reproducibility
y = randi (30,1,50);
cpart = cvpartition (y, 'KFold', 10);

Stratified k-fold cross validation partition
          NumObservations: 50
              NumTestSets: 10
                TrainSize: [26 48 48 45 50 42 45 48 48 50]
                 TestSize: [24 2 2 5 0 8 5 2 2 0]
                 IsCustom: 0
                IsGrouped: 0
             IsStratified: 1

TrainSize and TestSize have arbitrary number of observations. Doing the same with matlab retrieves the following:

K-fold cross validation partition
   NumObservations: 50
       NumTestSets: 10
         TrainSize: 45  45  45  45  45  45  45  45  45  45
          TestSize: 5  5  5  5  5  5  5  5  5  5
          IsCustom: 0

I copied the y variable literally in matlab

The number of observations of TrainSize and TestSize is the same for each group.

[pr0m1th3as]

According to MATLAB documentation, a vector as 1st input make stratified option true by default. Apparently, your results tell otherwise. This doesn't mean of course that Octave implementation is correct, but MATLAB's output also suggest that there is some undocumented behavior in their implementation as well.
I am away ATM, so I can't really test much or make any changes (hopefully, by next week I will try and fix this). Can you test the same vector in Octave with

cpart = cvpartition (y, 'KFold', 10, 'stratify', false)

as well as

cpart = cvpartition (50, 'KFold', 10)

[achengli]

I understand, as you, I think that the Matlab documentation in this case is wrong or it's considering stratify in the opposite meaning of the parameter setting.

The code you suggested is working as I was looking. Thanks!

[pr0m1th3as]

I am reopening, since the initial Octave results seem way off. I know that stratification on a vector of random numbers is meaningless, but I think that this needs a bit further investigation.
Can you test in MATLAB the following, please?

cpart = cvpartition (y, 'KFold', 10, 'stratify', true)

[achengli]

That's fine.

This is the result on matlab:

cpart = 

K-fold cross validation partition
   NumObservations: 50
       NumTestSets: 10
         TrainSize: 45  45  45  45  45  45  45  45  45  45
          TestSize: 5  5  5  5  5  5  5  5  5  5
          IsCustom: 0

Using the same y used before

[pr0m1th3as]

So, I assume that MATLAB handles numeric vectors as unstratifiable no matter what. Can you test with

y = [ones(25,1); zeros(25,1)];

[achengli]

cpart = 

K-fold cross validation partition
    NumObservations: 50
        NumTestSets: 10
          TrainSize: [45 45 45 45 45 45 45 45 45 45]
           TestSize: [5 5 5 5 5 5 5 5 5 5]
           IsCustom: 0
          IsGrouped: 0
       IsStratified: 1

[achengli]

However, repeating with random y (y = randi (30,1,50)), throws a warning. Changing stratify to false also splits the samples in same size groups.

stratify=true:

cpart = cvpartition (y, 'KFold', 10, 'stratify', true)

Warning: One or more of the unique class values in the stratification variable is not present in one or more
folds. For classification problems, either remove this class from the data or use N instead of the
stratification variable to obtain nonstratified partitions. For regression problems with continuous
response, use N. 
> In [internal.stats.cvpartitionImpl>stra_kfoldcv](matlab:matlab.lang.internal.introspective.errorDocCallback('internal.stats.cvpartitionImpl>stra_kfoldcv', '/MATLAB/toolbox/stats/stats/+internal/+stats/cvpartitionImpl.m', 439)) ([line 439](matlab: opentoline('/MATLAB/toolbox/stats/stats/+internal/+stats/cvpartitionImpl.m',439,0)))
In [internal.stats/cvpartitionImpl/rerandom](matlab:matlab.lang.internal.introspective.errorDocCallback('internal.stats/cvpartitionImpl/rerandom', '/MATLAB/toolbox/stats/stats/+internal/+stats/cvpartitionImpl.m', 338)) ([line 338](matlab: opentoline('/MATLAB/toolbox/stats/stats/+internal/+stats/cvpartitionImpl.m',338,0)))
In [internal.stats/cvpartitionInMemoryImpl](matlab:matlab.lang.internal.introspective.errorDocCallback('internal.stats/cvpartitionInMemoryImpl', '/MATLAB/toolbox/stats/stats/+internal/+stats/cvpartitionInMemoryImpl.m', 232)) ([line 232](matlab: opentoline('/MATLAB/toolbox/stats/stats/+internal/+stats/cvpartitionInMemoryImpl.m',232,0)))
In [cvpartition](matlab:matlab.lang.internal.introspective.errorDocCallback('cvpartition', '/MATLAB/toolbox/stats/stats/cvpartition.m', 122)) ([line 122](matlab: opentoline('/MATLAB/toolbox/stats/stats/cvpartition.m',122,0)))

cpart = 

K-fold cross validation partition
    NumObservations: 50
        NumTestSets: 10
          TrainSize: [45 45 45 45 45 45 45 45 45 45]
           TestSize: [5 5 5 5 5 5 5 5 5 5]
           IsCustom: 0
          IsGrouped: 0
       IsStratified: 1

With stratify=false:

cpart = cvpartition (y, 'KFold', 10, 'stratify', false)

cpart = 

K-fold cross validation partition
    NumObservations: 50
        NumTestSets: 10
          TrainSize: [45 45 45 45 45 45 45 45 45 45]
           TestSize: [5 5 5 5 5 5 5 5 5 5]
           IsCustom: 0
          IsGrouped: 0
       IsStratified: 0`

[pr0m1th3as]

So, when first input is a vector it uses stratified = true by default, but if the vector cannot be properly stratified (as is the case with random numbers) it produces a warning that certain classes are missing from some folds, but nevertheless, the returned cvpartition is stratified (to the extend that this is possible). In every case, the train and test sizes are equal. Thus, my cvpartition implementation is certainly wrong in this case.

[pr0m1th3as]

Another test, please

y =  [ones(26,1); zeros(26,1)];
cpart = cvpartition (y, 'KFold', 10)

[achengli]

cpart = 

K-fold cross validation partition
    NumObservations: 52
        NumTestSets: 10
          TrainSize: [47 46 46 47 47 47 47 47 47 47]
           TestSize: [5 6 6 5 5 5 5 5 5 5]
           IsCustom: 0
          IsGrouped: 0
       IsStratified: 1

Another test, please

y =  [ones(26,1); zeros(26,1)];
cpart = cvpartition (y, 'KFold', 10)

[pr0m1th3as]

Revisiting this issue after a while!

My initial thought about how to handle non-stratifiable vectors when stratify is true was a bit off. Apparently, MATLAB emits a warning and then defaults to stratify=false. So it handles the vector as if a scalar was given. I still need to figure out and test equally balanced stratification.

Second thing that is still missing from cvpartition is the ability to use an integer value other than 1 for leaveout partitioning. I assume that when leaveout != 1, then the specified number needs to be a divisor of the total number of samples N. Can someone test this, please? tagging @Sonu0305 for help 😅

[Sonu0305]

Hi @pr0m1th3as,
No problem for that!😄
I have uploaded a patch below which can work, but can you test it? (In the patch, I tried addding code that was required.)
changes.patch