[2024-03-19 03:20:08,223] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Loading data...
Loaded 900 examples from 9 languages: ['english', 'arabic', 'bengali', 'finnish', 'indonesian', 'korean', 'russian', 'swahili', 'telugu']
Loading model and tokenizer...

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:03<00:03,  3.71s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:05<00:00,  2.37s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:05<00:00,  2.57s/it]
Loading PEFT /flan_v2_7b_sft_rigl_r-64_lr-1e-5_wd-30_raw_codes
model.embed_tokens.weight: torch.bfloat16, torch.Size([32000, 4096]), False
model.layers.0.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.0.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.0.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.0.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.0.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.0.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.0.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.0.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.0.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.1.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.1.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.1.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.1.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.1.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.1.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.1.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.1.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.1.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.2.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.2.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.2.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.2.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.2.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.2.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.2.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.2.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.2.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.3.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.3.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.3.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.3.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.3.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.3.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.3.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.3.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.3.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.4.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.4.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.4.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.4.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.4.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.4.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.4.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.4.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.4.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.5.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.5.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.5.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.5.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.5.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.5.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.5.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.5.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.5.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.6.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.6.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.6.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.6.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.6.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.6.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.6.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.6.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.6.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.7.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.7.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.7.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.7.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.7.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.7.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.7.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.7.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.7.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.8.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.8.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.8.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.8.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.8.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.8.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.8.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.8.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.8.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.9.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.9.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.9.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.9.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.9.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.9.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.9.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.9.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.9.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.10.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.10.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.10.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.10.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.10.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.10.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.10.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.10.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.10.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.11.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.11.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.11.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.11.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.11.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.11.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.11.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.11.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.11.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.12.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.12.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.12.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.12.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.12.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.12.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.12.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.12.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.12.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.13.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.13.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.13.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.13.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.13.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.13.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.13.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.13.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.13.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.14.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.14.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.14.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.14.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.14.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.14.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.14.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.14.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.14.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.15.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.15.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.15.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.15.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.15.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.15.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.15.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.15.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.15.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.16.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.16.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.16.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.16.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.16.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.16.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.16.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.16.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.16.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.17.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.17.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.17.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.17.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.17.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.17.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.17.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.17.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.17.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.18.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.18.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.18.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.18.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.18.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.18.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.18.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.18.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.18.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.19.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.19.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.19.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.19.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.19.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.19.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.19.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.19.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.19.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.20.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.20.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.20.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.20.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.20.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.20.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.20.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.20.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.20.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.21.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.21.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.21.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.21.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.21.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.21.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.21.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.21.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.21.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.22.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.22.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.22.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.22.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.22.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.22.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.22.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.22.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.22.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.23.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.23.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.23.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.23.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.23.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.23.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.23.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.23.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.23.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.24.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.24.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.24.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.24.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.24.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.24.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.24.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.24.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.24.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.25.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.25.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.25.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.25.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.25.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.25.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.25.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.25.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.25.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.26.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.26.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.26.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.26.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.26.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.26.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.26.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.26.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.26.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.27.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.27.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.27.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.27.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.27.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.27.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.27.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.27.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.27.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.28.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.28.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.28.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.28.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.28.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.28.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.28.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.28.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.28.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.29.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.29.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.29.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.29.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.29.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.29.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.29.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.29.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.29.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.30.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.30.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.30.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.30.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.30.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.30.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.30.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.30.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
Using pad_token, but it is not set yet.
model.layers.30.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.31.self_attn.q_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.31.self_attn.k_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.31.self_attn.v_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.31.self_attn.o_proj.weight: torch.bfloat16, torch.Size([4096, 4096]), False
model.layers.31.mlp.gate_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.31.mlp.up_proj.weight: torch.bfloat16, torch.Size([11008, 4096]), False
model.layers.31.mlp.down_proj.weight: torch.bfloat16, torch.Size([4096, 11008]), False
model.layers.31.input_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.layers.31.post_attention_layernorm.weight: torch.bfloat16, torch.Size([4096]), False
model.norm.weight: torch.bfloat16, torch.Size([4096]), False
lm_head.weight: torch.bfloat16, torch.Size([32000, 4096]), False

Generating Completions:   0%|          | 0/900 [00:00<?, ?it/s]/root/anaconda3/envs/sft/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:381: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.6` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.
  warnings.warn(
/root/anaconda3/envs/sft/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:386: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.9` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`.
  warnings.warn(

Generating Completions:   1%|          | 5/900 [00:01<05:09,  2.89it/s]
Generating Completions:   1%|          | 10/900 [00:02<02:54,  5.09it/s]
Generating Completions:   2%|▏         | 15/900 [00:02<02:15,  6.51it/s]
Generating Completions:   2%|▏         | 20/900 [00:03<01:46,  8.29it/s]
Generating Completions:   3%|▎         | 25/900 [00:03<01:35,  9.19it/s]
Generating Completions:   3%|▎         | 30/900 [00:03<01:31,  9.49it/s]
Generating Completions:   4%|▍         | 35/900 [00:04<01:25, 10.11it/s]
Generating Completions:   4%|▍         | 40/900 [00:04<01:18, 10.93it/s]
Generating Completions:   5%|▌         | 45/900 [00:05<01:22, 10.36it/s]
Generating Completions:   6%|▌         | 50/900 [00:05<01:24, 10.05it/s]
Generating Completions:   6%|▌         | 55/900 [00:06<01:31,  9.26it/s]
Generating Completions:   7%|▋         | 60/900 [00:06<01:28,  9.47it/s]
Generating Completions:   7%|▋         | 65/900 [00:07<01:27,  9.49it/s]
Generating Completions:   8%|▊         | 70/900 [00:07<01:23,  9.97it/s]
Generating Completions:   8%|▊         | 75/900 [00:08<01:21, 10.13it/s]
Generating Completions:   9%|▉         | 80/900 [00:08<01:17, 10.64it/s]
Generating Completions:   9%|▉         | 85/900 [00:09<01:17, 10.48it/s]
Generating Completions:  10%|█         | 90/900 [00:09<01:24,  9.62it/s]
Generating Completions:  11%|█         | 95/900 [00:10<01:17, 10.40it/s]
Generating Completions:  11%|█         | 100/900 [00:10<01:12, 11.03it/s]
Generating Completions:  12%|█▏        | 105/900 [00:13<03:12,  4.14it/s]
Generating Completions:  12%|█▏        | 110/900 [00:16<04:18,  3.06it/s]
Generating Completions:  13%|█▎        | 115/900 [00:19<05:07,  2.55it/s]
Generating Completions:  13%|█▎        | 120/900 [00:21<05:25,  2.40it/s]
Generating Completions:  14%|█▍        | 125/900 [00:23<05:02,  2.57it/s]
Generating Completions:  14%|█▍        | 130/900 [00:25<05:20,  2.41it/s]
Generating Completions:  15%|█▌        | 135/900 [00:27<05:19,  2.39it/s]
Generating Completions:  16%|█▌        | 140/900 [00:30<05:38,  2.24it/s]
Generating Completions:  16%|█▌        | 145/900 [00:31<05:02,  2.50it/s]
Generating Completions:  17%|█▋        | 150/900 [00:33<04:52,  2.57it/s]
Generating Completions:  17%|█▋        | 155/900 [00:35<04:49,  2.57it/s]
Generating Completions:  18%|█▊        | 160/900 [00:37<05:16,  2.34it/s]
Generating Completions:  18%|█▊        | 165/900 [00:39<04:57,  2.47it/s]
Generating Completions:  19%|█▉        | 170/900 [00:42<05:28,  2.22it/s]
Generating Completions:  19%|█▉        | 175/900 [00:46<06:23,  1.89it/s]
Generating Completions:  20%|██        | 180/900 [00:49<06:49,  1.76it/s]
Generating Completions:  21%|██        | 185/900 [00:51<05:59,  1.99it/s]
Generating Completions:  21%|██        | 190/900 [00:53<05:52,  2.01it/s]
Generating Completions:  22%|██▏       | 195/900 [00:55<05:36,  2.09it/s]
Generating Completions:  22%|██▏       | 200/900 [00:58<06:11,  1.89it/s]
Generating Completions:  23%|██▎       | 205/900 [01:01<06:25,  1.80it/s]
Generating Completions:  23%|██▎       | 210/900 [01:04<06:20,  1.81it/s]
Generating Completions:  24%|██▍       | 215/900 [01:07<06:06,  1.87it/s]
Generating Completions:  24%|██▍       | 220/900 [01:10<06:42,  1.69it/s]
Generating Completions:  25%|██▌       | 225/900 [01:13<06:19,  1.78it/s]
Generating Completions:  26%|██▌       | 230/900 [01:17<07:25,  1.50it/s]
Generating Completions:  26%|██▌       | 235/900 [01:20<07:10,  1.55it/s]
Generating Completions:  27%|██▋       | 240/900 [01:24<07:07,  1.54it/s]
Generating Completions:  27%|██▋       | 245/900 [01:26<06:39,  1.64it/s]
Generating Completions:  28%|██▊       | 250/900 [01:30<07:16,  1.49it/s]
Generating Completions:  28%|██▊       | 255/900 [01:35<07:48,  1.38it/s]
Generating Completions:  29%|██▉       | 260/900 [01:37<07:16,  1.47it/s]
Generating Completions:  29%|██▉       | 265/900 [01:41<07:18,  1.45it/s]
Generating Completions:  30%|███       | 270/900 [01:44<07:04,  1.49it/s]
Generating Completions:  31%|███       | 275/900 [01:47<06:32,  1.59it/s]
Generating Completions:  31%|███       | 280/900 [01:49<06:10,  1.67it/s]
Generating Completions:  32%|███▏      | 285/900 [01:53<06:18,  1.62it/s]
Generating Completions:  32%|███▏      | 290/900 [01:55<05:57,  1.71it/s]
Generating Completions:  33%|███▎      | 295/900 [01:59<06:11,  1.63it/s]
Generating Completions:  33%|███▎      | 300/900 [02:02<06:21,  1.57it/s]
Generating Completions:  34%|███▍      | 305/900 [02:03<04:50,  2.05it/s]
Generating Completions:  34%|███▍      | 310/900 [02:04<04:20,  2.26it/s]
Generating Completions:  35%|███▌      | 315/900 [02:05<03:31,  2.76it/s]
Generating Completions:  36%|███▌      | 320/900 [02:06<02:45,  3.51it/s]
Generating Completions:  36%|███▌      | 325/900 [02:06<02:14,  4.27it/s]
Generating Completions:  37%|███▋      | 330/900 [02:07<01:58,  4.79it/s]
Generating Completions:  37%|███▋      | 335/900 [02:09<02:21,  4.01it/s]
Generating Completions:  38%|███▊      | 340/900 [02:10<02:20,  3.98it/s]
Generating Completions:  38%|███▊      | 345/900 [02:11<01:59,  4.64it/s]
Generating Completions:  39%|███▉      | 350/900 [02:12<01:50,  4.99it/s]
Generating Completions:  39%|███▉      | 355/900 [02:12<01:38,  5.55it/s]
Generating Completions:  40%|████      | 360/900 [02:13<01:37,  5.55it/s]
Generating Completions:  41%|████      | 365/900 [02:14<01:32,  5.77it/s]
Generating Completions:  41%|████      | 370/900 [02:15<01:32,  5.70it/s]
Generating Completions:  42%|████▏     | 375/900 [02:16<01:31,  5.76it/s]
Generating Completions:  42%|████▏     | 380/900 [02:16<01:22,  6.29it/s]
Generating Completions:  43%|████▎     | 385/900 [02:17<01:22,  6.26it/s]
Generating Completions:  43%|████▎     | 390/900 [02:19<01:42,  5.00it/s]
Generating Completions:  44%|████▍     | 395/900 [02:19<01:25,  5.94it/s]
Generating Completions:  44%|████▍     | 400/900 [02:20<01:19,  6.26it/s]
Generating Completions:  45%|████▌     | 405/900 [02:21<01:18,  6.28it/s]
Generating Completions:  46%|████▌     | 410/900 [02:22<01:45,  4.66it/s]
Generating Completions:  46%|████▌     | 415/900 [02:24<02:11,  3.68it/s]
Generating Completions:  47%|████▋     | 420/900 [02:25<01:49,  4.38it/s]
Generating Completions:  47%|████▋     | 425/900 [02:26<01:32,  5.13it/s]
Generating Completions:  48%|████▊     | 430/900 [02:26<01:19,  5.94it/s]
Generating Completions:  48%|████▊     | 435/900 [02:28<01:40,  4.62it/s]
Generating Completions:  49%|████▉     | 440/900 [02:28<01:21,  5.63it/s]
Generating Completions:  49%|████▉     | 445/900 [02:29<01:09,  6.51it/s]
Generating Completions:  50%|█████     | 450/900 [02:30<01:33,  4.79it/s]
Generating Completions:  51%|█████     | 455/900 [02:31<01:21,  5.44it/s]
Generating Completions:  51%|█████     | 460/900 [02:32<01:10,  6.23it/s]
Generating Completions:  52%|█████▏    | 465/900 [02:32<01:07,  6.49it/s]
Generating Completions:  52%|█████▏    | 470/900 [02:33<01:01,  7.04it/s]
Generating Completions:  53%|█████▎    | 475/900 [02:33<00:56,  7.48it/s]
Generating Completions:  53%|█████▎    | 480/900 [02:35<01:21,  5.17it/s]
Generating Completions:  54%|█████▍    | 485/900 [02:36<01:08,  6.05it/s]
Generating Completions:  54%|█████▍    | 490/900 [02:37<01:24,  4.85it/s]
Generating Completions:  55%|█████▌    | 495/900 [02:38<01:24,  4.81it/s]
Generating Completions:  56%|█████▌    | 500/900 [02:39<01:14,  5.35it/s]
Generating Completions:  56%|█████▌    | 505/900 [02:42<02:02,  3.22it/s]
Generating Completions:  57%|█████▋    | 510/900 [02:44<02:04,  3.13it/s]
Generating Completions:  57%|█████▋    | 515/900 [02:46<02:31,  2.53it/s]
Generating Completions:  58%|█████▊    | 520/900 [02:48<02:16,  2.78it/s]
Generating Completions:  58%|█████▊    | 525/900 [02:50<02:14,  2.79it/s]
Generating Completions:  59%|█████▉    | 530/900 [02:51<01:57,  3.14it/s]
Generating Completions:  59%|█████▉    | 535/900 [02:52<02:00,  3.04it/s]
Generating Completions:  60%|██████    | 540/900 [02:54<01:54,  3.14it/s]
Generating Completions:  61%|██████    | 545/900 [02:55<01:51,  3.19it/s]
Generating Completions:  61%|██████    | 550/900 [02:57<01:56,  3.02it/s]
Generating Completions:  62%|██████▏   | 555/900 [02:59<01:52,  3.07it/s]
Generating Completions:  62%|██████▏   | 560/900 [03:01<02:11,  2.58it/s]
Generating Completions:  63%|██████▎   | 565/900 [03:03<01:56,  2.86it/s]
Generating Completions:  63%|██████▎   | 570/900 [03:04<01:49,  3.01it/s]
Generating Completions:  64%|██████▍   | 575/900 [03:06<01:42,  3.18it/s]
Generating Completions:  64%|██████▍   | 580/900 [03:08<01:47,  2.97it/s]
Generating Completions:  65%|██████▌   | 585/900 [03:09<01:38,  3.21it/s]
Generating Completions:  66%|██████▌   | 590/900 [03:12<02:00,  2.57it/s]
Generating Completions:  66%|██████▌   | 595/900 [03:13<01:44,  2.92it/s]
Generating Completions:  67%|██████▋   | 600/900 [03:14<01:40,  2.99it/s]
Generating Completions:  67%|██████▋   | 605/900 [03:15<01:25,  3.45it/s]
Generating Completions:  68%|██████▊   | 610/900 [03:17<01:20,  3.62it/s]
Generating Completions:  68%|██████▊   | 615/900 [03:18<01:13,  3.87it/s]
Generating Completions:  69%|██████▉   | 620/900 [03:19<01:17,  3.63it/s]
Generating Completions:  69%|██████▉   | 625/900 [03:20<01:02,  4.38it/s]
Generating Completions:  70%|███████   | 630/900 [03:21<00:54,  4.95it/s]
Generating Completions:  71%|███████   | 635/900 [03:21<00:47,  5.63it/s]
Generating Completions:  71%|███████   | 640/900 [03:22<00:41,  6.30it/s]
Generating Completions:  72%|███████▏  | 645/900 [03:22<00:36,  6.97it/s]
Generating Completions:  72%|███████▏  | 650/900 [03:23<00:41,  6.01it/s]
Generating Completions:  73%|███████▎  | 655/900 [03:24<00:35,  6.85it/s]
Generating Completions:  73%|███████▎  | 660/900 [03:26<00:48,  4.91it/s]
Generating Completions:  74%|███████▍  | 665/900 [03:27<00:48,  4.81it/s]
Generating Completions:  74%|███████▍  | 670/900 [03:27<00:43,  5.26it/s]
Generating Completions:  75%|███████▌  | 675/900 [03:28<00:39,  5.74it/s]
Generating Completions:  76%|███████▌  | 680/900 [03:30<00:50,  4.39it/s]
Generating Completions:  76%|███████▌  | 685/900 [03:30<00:42,  5.09it/s]
Generating Completions:  77%|███████▋  | 690/900 [03:31<00:37,  5.67it/s]
Generating Completions:  77%|███████▋  | 695/900 [03:32<00:32,  6.23it/s]
Generating Completions:  78%|███████▊  | 700/900 [03:32<00:28,  6.91it/s]
Generating Completions:  78%|███████▊  | 705/900 [03:33<00:27,  7.04it/s]
Generating Completions:  79%|███████▉  | 710/900 [03:34<00:25,  7.39it/s]
Generating Completions:  79%|███████▉  | 715/900 [03:34<00:23,  7.86it/s]
Generating Completions:  80%|████████  | 720/900 [03:35<00:22,  8.05it/s]
Generating Completions:  81%|████████  | 725/900 [03:35<00:21,  7.95it/s]
Generating Completions:  81%|████████  | 730/900 [03:36<00:20,  8.19it/s]
Generating Completions:  82%|████████▏ | 735/900 [03:37<00:20,  8.02it/s]
Generating Completions:  82%|████████▏ | 740/900 [03:37<00:19,  8.28it/s]
Generating Completions:  83%|████████▎ | 745/900 [03:38<00:18,  8.60it/s]
Generating Completions:  83%|████████▎ | 750/900 [03:38<00:17,  8.63it/s]
Generating Completions:  84%|████████▍ | 755/900 [03:39<00:15,  9.45it/s]
Generating Completions:  84%|████████▍ | 760/900 [03:39<00:15,  8.81it/s]
Generating Completions:  85%|████████▌ | 765/900 [03:40<00:14,  9.01it/s]
Generating Completions:  86%|████████▌ | 770/900 [03:41<00:15,  8.14it/s]
Generating Completions:  86%|████████▌ | 775/900 [03:41<00:14,  8.40it/s]
Generating Completions:  87%|████████▋ | 780/900 [03:42<00:17,  6.86it/s]
Generating Completions:  87%|████████▋ | 785/900 [03:43<00:17,  6.40it/s]
Generating Completions:  88%|████████▊ | 790/900 [03:44<00:16,  6.72it/s]
Generating Completions:  88%|████████▊ | 795/900 [03:44<00:13,  7.54it/s]
Generating Completions:  89%|████████▉ | 800/900 [03:44<00:11,  8.75it/s]
Generating Completions:  89%|████████▉ | 805/900 [03:48<00:30,  3.13it/s]
Generating Completions:  90%|█████████ | 810/900 [03:52<00:41,  2.19it/s]
Generating Completions:  91%|█████████ | 815/900 [03:56<00:47,  1.79it/s]
Generating Completions:  91%|█████████ | 820/900 [04:00<00:49,  1.61it/s]
Generating Completions:  92%|█████████▏| 825/900 [04:04<00:50,  1.49it/s]
Generating Completions:  92%|█████████▏| 830/900 [04:07<00:43,  1.61it/s]
Generating Completions:  93%|█████████▎| 835/900 [04:11<00:43,  1.50it/s]
Generating Completions:  93%|█████████▎| 840/900 [04:13<00:37,  1.60it/s]
Generating Completions:  94%|█████████▍| 845/900 [04:17<00:36,  1.49it/s]
Generating Completions:  94%|█████████▍| 850/900 [04:21<00:35,  1.42it/s]
Generating Completions:  95%|█████████▌| 855/900 [04:25<00:32,  1.40it/s]
Generating Completions:  96%|█████████▌| 860/900 [04:29<00:29,  1.34it/s]
Generating Completions:  96%|█████████▌| 865/900 [04:32<00:24,  1.45it/s]
Generating Completions:  97%|█████████▋| 870/900 [04:35<00:21,  1.40it/s]
Generating Completions:  97%|█████████▋| 875/900 [04:40<00:18,  1.34it/s]
Generating Completions:  98%|█████████▊| 880/900 [04:44<00:15,  1.30it/s]
Generating Completions:  98%|█████████▊| 885/900 [04:47<00:11,  1.31it/s]
Generating Completions:  99%|█████████▉| 890/900 [04:50<00:06,  1.45it/s]
Generating Completions:  99%|█████████▉| 895/900 [04:54<00:03,  1.40it/s]
Generating Completions: 100%|██████████| 900/900 [04:58<00:00,  1.35it/s]
Generating Completions: 100%|██████████| 900/900 [04:58<00:00,  3.02it/s]
Calculating F1, EM ...
Scores:
{
    "english": {
        "exact_match": 58.0,
        "f1": 70.77168821516648
    },
    "arabic": {
        "exact_match": 37.0,
        "f1": 52.500809297491216
    },
    "bengali": {
        "exact_match": 32.0,
        "f1": 43.857142857142854
    },
    "finnish": {
        "exact_match": 48.0,
        "f1": 62.45294006818774
    },
    "indonesian": {
        "exact_match": 62.0,
        "f1": 73.80938764638563
    },
    "korean": {
        "exact_match": 63.0,
        "f1": 71.42954545454545
    },
    "russian": {
        "exact_match": 37.0,
        "f1": 58.16453501778578
    },
    "swahili": {
        "exact_match": 46.0,
        "f1": 57.97829272177098
    },
    "telugu": {
        "exact_match": 0.0,
        "f1": 0.9523809523809522
    },
    "average": {
        "f1": 54.65741358120634,
        "exact_match": 42.55555555555556
    }
}
Done!