GPU usage discussion

[HuyLeDinhGia]

Hi team,
Thanks for your valuable project. It will be better if we have a separate part for GPU usage discussion. In this part, you could estimate the VRAM requirement using the training size, feature size, test points size, etc. This will help the users know how much VRAM they need for their project, and with this estimation, you can print out the warning for users before they infer. Besides, as the VRAM size increases, it should be explained how we can use multiple GPUs with the model; in fact, support for multiple GPU compatibility is needed.

Regards,
Huy

[NicoMaker]

When participating in discussions on the PriorLabs/TabPFN repository regarding GPU usage, it is best to keep your technical points direct and clear. Here is a simple guide to framing your post:
Key Points for Your Discussion
Specify your hardware: State the GPU you are using (e.g., "I am running TabPFN on an NVIDIA A100/H100"). This helps the community understand your performance baseline �.
Mention your workload: Briefly describe the size of your datasets (e.g., "I am processing datasets with ~5k rows and 100 features"). This clarifies if you are hitting memory limits �.
Ask about trade-offs: If you are deciding between local hosting and their API, ask about the benefits for your specific use case. For example: "Is local GPU inference recommended for sensitive data compared to the Prior Labs cloud API?" �.
Suggested Message Template
You can adapt this simple structure for a new discussion or issue:
Subject: GPU optimization for local deployment
Hi everyone,
I am working on integrating TabPFN into our project and would like to optimize it for local GPU usage.
Setup: Running on [Your GPU Model] with [Amount of VRAM].
Goal: Efficient inference for datasets of size [Number of rows/features].
I am currently documenting our architecture choices using ADRs. Could you share any best practices for memory management or device configuration to ensure stable performance for production?
Any advice on local vs. API deployment for this scale would be appreciated!
Why this works:
Clear Context: You provide the "who, what, and where," which makes it easier for maintainers to give you a helpful answer �.
Professional Tone: Mentioning your ADR workflow shows you are serious about governance and project structure, which often encourages more detailed, high-quality responses from open-source maintainers �.

[BernalFA]

Hello everyone!

I am testing TabPFN for one of our projects on molecular property prediction. I decided to go for fine-tuning but I could not make it work.

Setup: model v2.6 on Nvidia GeForce RTX 2080 with 11GB of memory.
Goal: fine-tune the model to my dataset (training 1000 samples with 386 features; separate validation set of 220 samples).

I am not an expert in PyTorch but have been able to use the TabPFNRegressor on the same dataset and also hyperparameter tuning of chemprop models on the same hardware without problems. However, when I try to fit the FinetunedTabPFNRegressor, it fails due to memory issues.

Here a snippet of my code:

from tabpfn.finetuning import FinetunedTabPFNRegressor
from tabpfn.model_loading import ModelSource, prepend_cache_path

# As FinetunedTabPFNRegressor uses v2.5 by default, I set v2.6 manually
model_path = prepend_cache_path(
    ModelSource.get_regressor_v2_6().default_filename
)
model = FinetunedTabPFNRegressor(
    extra_regressor_kwargs={
        "model_path": model_path
    }
)

model.fit(train[cols], train.y, X_val=val[cols], y_val=val.y)

Output:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 94.00 MiB. GPU 0 has a total capacity of 10.57 GiB of which 42.06 MiB is free. Including non-PyTorch memory, this process has 10.52 GiB memory in use. Of the allocated memory 10.14 GiB is allocated by PyTorch, and 197.79 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)

It is important to note that there are NO processes running on the GPU card (in fact the hardware has 4 cards all free).

Does anyone have any advice on how to properly use the FinetunedTabPFNRegressor on such a small dataset without running into out-of-memory issues? I would really appreciate your help.