Pool Me Wisely: On the Effect of Pooling in Transformer-Based Models

This repository is the official implementation of our paper "Pool Me Wisely: On the Effect of Pooling in Transformer-Based Models".

In the paper, we consider different modalities, namely Computer Vision, NLP and Time Series. This repository is therefore divided into three parts.

Requirements

Code is written in Python and we consider different modalities, which each depends on a set of requirements that should be check individually.

Setup

The code's folder is divided into a corresponding folder for each modality:

• computer_vision: contains the full implementation of the experiments considered for the CV tasks.
• NLP: contains the full implementation of the experiments considered for the NLP tasks.
• time_series: contains the full implementation of the experiments considered for the Time Series tasks.

Each folder and modality have its corresponding folder setup that should also be respected and is simply explained in the corresponding ReadMe of each folder.

Citing

Upon using this repository for your work, or finding our proposed analysis useful for your research, please consider citing our paper this paper:

@inproceedings{
ennadir2025pool,
title={Pool Me Wisely: On the Effect of Pooling in Transformer-Based Models},
author={Sofiane ENNADIR and Levente Z{\'o}lyomi and Oleg Smirnov and Tianze Wang and John Pertoft and Filip Cornell and Lele Cao},
booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},
year={2025},
url={https://openreview.net/forum?id=8uhXfdSJmA}
}

For any additional questions/suggestions you might have about the code and/or the proposed analysis, please contact: sofiane.ennadir@king.com