A bug in the cross_validation function

[pyaf]

Hi,

Currently, cross_validation function shuffles data before every step of k fold cross-validation [here]. Which is wrong because of this the test set at each step is not unique (when compared to all the other steps) which gives us the wrong k fold cross validation perfomance score.

Instead, we should shuffle the dataset before performing k fold cross validation (rather than doing the same at each step of k fold cross-validation).

Let me know your thoughts.

[ad71]

The implementation for the cross-validation algorithm has been up for debate for quite some time now. Refer to this issue and this issue for details. It needs to be re-implemented but we haven't reached a conclusion yet. This will probably be updated in the newer versions of AIMA.
However, I think we can implement a function with a different name for the time-being, that works as we expect cross_validation to.

[pyaf]

I see a # TODO: The function cross_validation_wrapper needs to be fixed. (The while loop runs forever!) before cross_validation_wrapper. Is this the issue here? Like what exactly is the debate here?

[ad71]

The debate here is that the conventional meaning of cross_validation has changed since the time AIMA was first written. The pseudocode in AIMA does different things than what cross_validation in scikit-learn does for example. We are unsure as to what the goal of the cross_validation function in the repo is, to implement the pseudocode from AIMA or the more commonly accepted implementations in modern libraries.

[ashishgit7]

@pyaf sir I made a PR related 995 issue now what else we can do here

[Pihu1998]

Hi! I would like to know whether this issue is resolved.

[ashishgit7]

@pyaf can you review my PR

[pyaf]

@hackerashish25 you should ask aima-python mentors for that :)