Provide more flexible holdouts for train-test extraction

[ghost]

(originally reported in Trac by @elehack on 2011-03-25 16:04:08)

In addition to our fractional holdout scheme for splitting query and probe sets for user-based crossfold evaluation, it would be useful to support leave-''N''-out and retain-''N'' protocols.

All that's required to finish this is support for retain-N protocols.

Note: This issue has been automatically migrated from Bitbucket
Created by grouplens on 2013-02-01T21:55:29.120420+00:00, last updated: 2013-03-05T18:29:43.403522+00:00

[ghost]

Merged in LingfeiH/lenskit (pull request #89, closes #33)

Add method and class to support the retain-N protocol

→ <<cset 86d5dbf711fa>>

Note: This comment has been automatically migrated from Bitbucket
Created by @elehack on 2013-03-05T18:29:43.398723+00:00, last updated: None

[ghost]

There are two pieces to this:

• add a setRetain(int) method to CrossfoldCommand that sets up a retain-N partition
• create a new partition class, or extend CountPartition, to support the retain-N protocol

I think it is probably best to rename CountPartition to LastNPartition, and create a FirstNPartition to implement the new logic. These classes are not part of the public interface, so it is fine to reimplement them.

We should probably also improve the JavaDoc for setHoldout and setRetain to be clear, and create a new setHoldoutFraction method (and deprecate setHoldout(double)) to make things all clear.

Note: This comment has been automatically migrated from Bitbucket
Created by @elehack on 2013-02-28T20:44:39.674033+00:00, last updated: None

[ghost]

Reducing scope to remove overlap with #205.

Note: This comment has been automatically migrated from Bitbucket
Created by @elehack on 2013-02-05T23:06:38.018883+00:00, last updated: None

[ghost]

Michael Ekstrand michael@elehack.net on 2011-10-18 13:55:03 said:

In [29e4320947109e148b63e35624964c80254f5179]:

Support fractional holdouts (refs #33).

* Re-implement holdout system in terms of orders and partition methods
* Add fractional holdout & parse percentages in config file

Note: This comment has been automatically migrated from Bitbucket
Created by grouplens on 2013-02-01T21:55:29.835406+00:00, last updated: None

[ghost]

ekstrand on 2011-05-24 18:10:18 said:

We now do leave-N-out. We still need more flexibility, but it's a start.

Note: This comment has been automatically migrated from Bitbucket
Created by grouplens on 2013-02-01T21:55:29.462014+00:00, last updated: None