The shell scripts started with run_* in zemi/training and zemi/eval are lower level scripts that will call training and evaluation .py files. For conveniently running multiple experiments, we provide higher level scripts, such as zemi_base.sh, which will call lower level shell scripts for both training and evalution. The comments for each positional arguments can be found in each of the higher level scripts, such as zemi_base.sh. For Zemi and FiD models, we will further include a config file from perceiver_configs/ which is used for config the model architecture, such as the number of augmentation, latent size, etc (detailed in the following section)
The json file in perceiver_configs/ directory contains the main configuration for Zemi and FiD model architecture. The following table shows the description of each fields.
| Field Name | Description |
|---|---|
| dim | hidden dim; 768 for base, 1024 for large |
| num_latents | size of the latent query vector; default = 64 |
| num_aug_sources | maximum num of augmentations to be considered for each instance |
| depth | num of layers of the perceiver resampler, default = 1 |
| heads | num of heads of the perceiver resampler, default = 8 |
| dim_head | dim for each head, default = 64 |
| ff_mult | FFN layer scaling ratio, default = 2 |
| xattn_ff_mult | Cross-attn FFN layer scaling ratio, default = 4 |
| freeze_lm | If freeze the backbone language model, default = False |
| cross_attn_every | * for future extension, can be ignored, default = 1 |
| only_attend_immediate_media | * for future extension, can be ignored, default = False |
| num_xattn_layers | * for future extension, can be ignoredm default = 1 |
- base:
bash ./training/no_aug_base.sh - large:
bash ./training/no_aug_large.sh
- base:
bash ./training/concat_base.sh - large:
bash ./training/concat_large.sh
- base:
bash ./training/fid_base.sh - large:
bash ./training/fid_large.sh
- base:
bash ./training/zemi_base.sh - large:
bash ./training/zemi_large.sh
We provide the script for post-processing the evaluation results that will be outputed in zemi/output/eval. Please refer to the lines with #TODO instructions for using the following script which finds the overall best checkpoint across all tasks and return the averaged scores for each task:
zemi/output/eval/visualize_multitask_results.py