Reproduce Validate MAE

[alirezamshi-zz]

Hi,

Thanks for your interesting work. I have a problem regarding the evaluation. I downloaded your checkpoints from here, then I run the following command as mentioned in the Readme (for all_fold_seed0 checkpoint):

conda activate graphormer-lsc
export arch="--ffn_dim 768 --hidden_dim 768 --attention_dropout_rate 0.1 --dropout_rate 0.1 --n_layers 12 --peak_lr 2e-4 --edge_type multi_hop --multi_hop_max_dist 20 --weight_decay 0.0 --intput_dropout_rate 0.0"
export ckpt_path="checkpoints"
export ckpt_name="all_fold_seed0.ckpt"
bash inference.sh

The output log is:

Global seed set to 1
 > PCQM4M-LSC loaded!
{'num_class': 1, 'loss_fn': <function l1_loss at 0x7fc2381b3950>, 'metric': 'mae', 'metric_mode': 'min', 'evaluator': <ogb.lsc.pcqm4m.PCQM4MEvaluator object at 0x7fc1995d2110>, 'dataset': MyPygPCQM4MDataset2(3803453), 'max_node': 128}
 > dataset info ends
total params: 47167841
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
Global seed set to 1
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/1
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
len(val_dataloader) 1487
Validating: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1487/1487 [03:04<00:00,  7.42it/s]
0.027769196778535843
--------------------------------------------------------------------------------                                                                                                                           
DATALOADER:0 VALIDATE RESULTS
{'valid_mae': 0.027769196778535843}
--------------------------------------------------------------------------------
[{'valid_mae': 0.027769196778535843}]

I assumed that I should get results near Table 1 "validate MAE column", but it's different from that. Do I miss something?

Thanks for your help.

[chengxuanying]

Hi @alirezamshi ,

As mentioned in the report, we trained 8 Graphormer models using 8-fold cross-validation, and additional 2 Graphormer models using whole data.

The checkpoint you are using now is all_fold_seed0.ckpt, please try to use the cross-validation split here here and use the correct checkpoint instead of these begin with all_fold.

Best,
Chengxuan

[alirezamshi-zz]

Hi @chengxuanying

Thanks for your reply.

So, If I remove this line , and modify this line to just include train split ids, then it should be fine, right?

Best,
Alireza

[chengxuanying]

You can alternatively uncomment L71-86, and comment L89-96.
And remember to load the correct checkpoint.

By the way, in the code, I simply set fold index the same as seed.

[alirezamshi-zz]

Got it. Thanks.

[PierreHao]

@chengxuanying , great work, i have a problem, if we want to reproduce validate MAE 0.1234 (single model) with training from scratch, which branch (main or ogb-lsc) we should use?