Using Random Forest classifier in batch mode

[vahuja4]

Hello there,

I have a scenario in which I need to train a random forest classifier in the following manner:

Step1: Train on 20k datapoints just once
Step2: Train on 20 datapoints. This will repeat every week.

Can someone please tell me how this can be achieved using river-ml.

Sincerely,
Vishal

[MaxHalford]

Can you please tell us what you've tried? You should be able to do this in a straightforward manner by using the learn_one method. I'd be happy to help out, but I have no idea what you've already tried out.

Meta: the project is called River, not river-ml :)

[ElisonSherton]

Hi Max,

Good Day!

I have consulted with Vishal and my other colleagues and I can give you the notebook and dataset here as below.

https://drive.google.com/drive/folders/13SIDHxDG9A3BC6rkRNjqKGJ75QOVqDHs?usp=sharing

There are two files of interest here:

1. Using River ML For Adaptive Fitting a Random Forest.ipynb: This is the notebook used for training
2. dress_neckline_object.pkl: This is the dataset which is utilized in the notebook for training.

You could ignore the two html files that I sent yesterday which are also present in this same folder. The environment details are as follows:

I am using an ubuntu 18.04 LTS distribution as my Operating System. Hope this helps to investigate further for you.

Thanks & Regards,
Vinayak

[vahuja4]

Also Max, please look at these two questions of ours:

1. Is this a good use-case for using river?
2. Does the adaptive random forest adapt to distribution shift in the production data?

[MaxHalford]

Ah, I think I know. Can you try shuffling the training data? Online models tend to overfit on recent data if the data is ordered in a deceptive way. In this case I believe your training data is ordered by class.

Is this a good use-case for using river?

To be honest I would be inclined to say no. I don't know how much data you have, but you can probably afford to retrain a batch model every week.

Does the adaptive random forest adapt to distribution shift in the production data?

Yes! That's one of the main benefits of using an online model.

[ElisonSherton]

Hey Max,

Good Day!

Thanks for that suggestion of yours. Now that I run the code, I get a good concurrence between the results of train set when done dynamically and when done all at once as could be seen in the html code file below.

https://drive.google.com/file/d/10PTUs3MgO1WQpKS55JqL6guvNBlTTduy/view?usp=sharing

One question which I have Max is that let's say the incoming stream data is only of a single class; i.e. out of the 11 classes, I get 10 datapoints successively with label of 1 then will there be overfitting wrt a class?

Because what I can make out of this exercise is that when I curate the input in such a way that the same class' datapoints are iteratively feeded to the algorithm the accuracy as seen in the previous notebook is 98%. But it's generalizability goes for a toss in that case. Whereas when data is curated in such a way that the incoming datapoints are random or follow some kind of equiprobable distribution, then the generalizability is pretty decent.

Thanks & Regards,
Vinayak.

[MaxHalford]

Basically, you want to train the model with data ordered as it would be in production. The goal is to simulate as closely as possible a production scenario. It's likely that the data order is shuffled with respect to the classes in production so you should do the same thing for training.