Add TabPFN #3270
Add TabPFN #3270
Conversation
Innixma
added
enhancement
|
Hello @noahho and @SamuelGabriel, I am looking to add TabPFN to AutoGluon for the upcoming v0.8 release on June 13th! If you had time, it would be great if you could give a quick look at the PR and mention any suggestions/changes you'd recommend. In particular, I'd be curious your thoughts on the |
| ) | ||
| return X, y | ||
|
|
||
| # TODO: Should we fillna 0? what about -1 or mean? Does TabPFN automatically fill missing values? |
There was a problem hiding this comment.
TabPFNClassifier can handle nans and fills them more or less with the mean of the feature. So, I think it is best to just hand the nans over.
There was a problem hiding this comment.
Great! I did a quick test and it appears to work without doing the fillna, with comparable performance, so I've updated the PR to not fillna and instead pass the NaNs to TabPFN directly.
|
|
||
| # TODO: Make sample_rows generic | ||
| if sample_rows is not None and len(X) > sample_rows: | ||
| X, y = self._subsample_train(X=X, y=y, num_rows=sample_rows) |
There was a problem hiding this comment.
We have a bit of data on this. The best strategy is to something like have multiple trees with a TabPFN in each leaf, but this is not too much worse in our experience.
There was a problem hiding this comment.
That makes sense, I had a similar assumption as to what would probably work best.
For now, I've kept as is since doing the TabPFN Forest approach is very similar to using bagging in AutoGluon on TabPFN. I'd expect TabPFN_Forest to be slightly better, but will keep things simple for now. If you do have data that shows a compelling performance improvement I'd definitely take another look.
If TabPFN can do this logic internally, that could be a very nice addition. For example, when N_ensemble_configurations=10, each TabPFN in N could use a different sample from the train data input, which should maximize diversity.
|
Hey :) I have left two comments on your points. It looks good in general, I think, besides the 0 imputation strategy.
Generally I think it is important to know that the main costs of TabPFN are incurred with loading the model to (GPU) memory at the first construction of the TabPFNClassifer and in the fit function, where the full forward pass happens.
Best,
Sam
… On 4. Jun 2023, at 05:47, Nick Erickson ***@***.***> wrote:
Hello @noahho <https://github.com/noahho> and @SamuelGabriel <https://github.com/SamuelGabriel>, I am looking to add TabPFN to AutoGluon for the upcoming v0.8 release on June 13th!
If you had time, it would be great if you could give a quick look at the PR and mention any suggestions/changes you'd recommend.
In particular, I'd be curious your thoughts on the sample_rows value and how to best handle missing values.
—
Reply to this email directly, view it on GitHub <#3270 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACK7PSLIGCRM6UZ4QVS2TGTXJQALRANCNFSM6AAAAAAYZVQF3A>.
You are receiving this because you were mentioned.
|
|
Job PR-3270-3ed4522 is done. |
|
Job PR-3270-833da90 is done. |
|
Job PR-3270-a15baab is done. |
|
Job PR-3270-94ad668 is done. |
|
Job PR-3270-fa2e75b is done. |
|
Job PR-3270-34c9d1d is done. |
|
Job PR-3270-ffd396d is done. |
|
Job PR-3270-d021ba8 is done. |
* 'master' of https://github.com/awslabs/autogluon: (24 commits) [WIP] 0.8.0 release notes (autogluon#3303) Add model keys doc (autogluon#3321) Fix NaN warning in np.array(X) (autogluon#3315) [Draft] Upgrade networkx to 3.x (autogluon#3317) Add calibrate_decision_threshold tutorial (autogluon#3316) [Doc] AutoMM FAQ Updates (autogluon#3314) Update to v0.8.0 (autogluon#3313) Add Experimental Zeroshot HPO (autogluon#3312) Update GPU installation guide to use CUDA 11.7 (autogluon#3306) [Tutorial]Update tutorials for object detection (autogluon#3305) [timeseries] Update documentation (autogluon#3297) Update mac cpu install instructions (autogluon#3280) Add docstring for hyperparameter_tune_kwargs (autogluon#3307) [Doc] Add Search Space Page (autogluon#3311) Fewshot learning predict proba (autogluon#3267) Fix log to file Windows tests (autogluon#3302) Add missing doc pages (autogluon#3304) Add calibrate_decision_threshold (autogluon#3298) continuous training tutorial update (autogluon#3300) Add TabPFN (autogluon#3270) ...
Issue #, if available:
#2806
Description of changes:
TODO:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.