Pocket Former

A playground for implementing transformer architectures.

Usage

Wiki Dataset

Download the dataset to data/wiki-data.txt:

python dataset.py download

Process the dataset from data/wiki-data.txt:

python dataset.py preprocess

Sample data from the dataset:

python dataset.py sample

The Model

Train the model:

python transformer.py --train

Inference the model:

python transformer.py --prompt "Hello World"

Train model with customer hyperparameters:

python transformer.py --bs 32 --seq_len 16 --hidden_dim 512 --num_heads 8  --train

Move Group Dataset

Move groups are saved to data/mg/ Each move group has a data.json

Download and preprocess the first 20 move groups:

python mg.py download 20

Then download all the videos

python mg.py download_videos

Then process all the frames

python mg.py process_frames

View info for mg with id 0:

python mg.py mg 0 print

Download a video for mg with id 0:

python mg.py mg 0 video

Process all the frames for mg with id 0:

python mg.py mg 0 process

Launch the labeler

python mg-labeler.py