Nested Transformer: A Solution to Kaggle AI4Code Competition

We called our network architecture "Nested Transformers", since it contains two levels of transformers:

• Cell transformer: A transformer encoder to encode cell tokens, this can be a typical transformer for NLP.
• Notebook transformer: A transformer encoder with cell*cell self-attention to learn the interaction among cells.

A brief description of the solution is provided here.

Training Steps

• Check and modify env.sh
• Create relevant directories and place datasets to the RAW_DIR specified in env.sh
• Check configs/train.sh to adjust the hyperparameters
• Next, follow the steps below:

# install the src module
pip install -e .

# set environment variables
source env.sh

# data preprocessing
python preprocess.py

# training
sh configs/train.sh