Different accuracy results after save_test_submission for MAG240M-LSC dataset?

[tadpole]

Hi,
When I test my results on MAG240M-LSC dataset with the provided rgat baseline by "python rgnn.py --device=0 --model=rgat --evaluate", the accuracy is 0.6843 in the validation data. But after I save the predicted label with save_test_submission , load the file and eval the results by MAG240MEvaluator().eval(), the result is 0.6804. There was about 0.5% difference. Why is that?

Thanks!

[weihua916]

Hi! How were you able to call save_test_submission on the validation data? The number of data should not match in the first place (see here).

Otherwise, please give us more context on how you save your data, how you call evaluation etc.

[weihua916]

Can you share a broader context, your eval part? My guess is the non-deterministic nature of our inference here. Also wondering why your validation accuracy is worse than reported. Did you run the code as it is?

[tadpole]

    if args.evaluate:
        dirs = glob.glob(f'logs/{args.model}/lightning_logs/*')
        if args.version == '-1':
            version = max([int(x.split(os.sep)[-1].split('_')[-1]) for x in dirs])
        else:
            version = args.version
        logdir = f'logs/{args.model}/lightning_logs/version_{version}'
        print(f'Evaluating saved model in {logdir}...')
        ckpt = glob.glob(f'{logdir}/checkpoints/*')[0]

        trainer = Trainer(gpus=args.device, resume_from_checkpoint=ckpt)
        model = MODEL.load_from_checkpoint(
            checkpoint_path=ckpt, hparams_file=f'{logdir}/hparams.yaml')

        datamodule.batch_size = 16
        datamodule.sizes = [160] * len(args.sizes)  # (Almost) no sampling...

        datamodule.setup()
        """
        print('Starting validation....')
        trainer.test(model=model,test_dataloaders=datamodule.val_dataloader())
        print('Starting test....')
        trainer.test(model = model, test_dataloaders=datamodule.test_dataloader())
        """

        if args.mode == 'offline':
            evaluator = MAG240MEvaluator()
            loader = datamodule.test_dataloader()

            model.eval()
            y_preds = []
            for batch in tqdm(loader):
                batch = batch.to(int(args.device))
                model = model.to(int(args.device))
                with torch.no_grad():
                    out = model(batch.x, batch.adjs_t).argmax(dim=-1).cpu()
                    y_preds.append(out)
            res = {'y_pred': torch.cat(y_preds, dim=0)}
            save_test_submission(res, f'results/{args.model}')

I re-split the dataset with papers before 2018 as train data, the paper in 2018 as valid, and the paper in 2019 as test, so the validation accuracy is worse than reported.

[weihua916]

Thanks for sharing. I think the cause is the non-deterministic nature of the neighbor-sampling-based inference. Different neighbors are sampled for your eval log part and save test submission part, making the model's predictions different even for the same set of target nodes. Making the neighbor sampling sizes (datamodule.sizes = [160] * len(args.sizes)) larger should close the gap, but increase the inference cost.

[tadpole]

Sorry to trouble you again. I increase the sample number from 160 to 1000 and run many times, the accuracy of trainer.test is always larger than that of MAG240MEvaluator().eval(). It is very strange. And the accuracy of trainer.test gives 0.6900462508201599. So the number of correctly predicted labels which equals 0.6900462508201599*number of test data 138949 = 95881.2365052104 is not an integer. I am very confuse about how the trainer.test calculate the accuracy?

[rusty1s]

There seems to be an issue in PyTorch lightning in which the accuracy of the last batch (with drop_last=False) is not correctly re-weighted and contributes more to the overall performance. I created an issue here: Lightning-AI/pytorch-lightning#6889.