Are you allowed to use validation labels for label propagation?

[Chillee]

I plan on submitting several label propagation baselines. However, when using label propagation (which has 0 parameters), it seems quite weird (and unfair to it) to not use the validation labels for the purposes of propagation.

There's only one hyperparameter to tune (alpha) and no parameters learned. In addition, regardless of what hyperparameters you choose (and how you choose them), it's hard for me to imagine an instance where using train + validation labels will worsen your performance at inference.

For GNNs a validation set is often needed, as you need it for stuff like early stopping as well as tuning your (many) hyperparameters.

Essentially, if say, ogbn-arxiv was a realistic setting, I can't imagine a case where you wouldn't want to use validation labels for your label propagation.

However, I can understand if OGB would like to prohibit that, given that OGB is meant as a benchmark, which by their very nature, are somewhat synthetic.

[weihua916]

We do not allow this.

[Chillee]

@weihua916 can you provide more explanation of the rationale?

[weihua916]

Sure. If there is absolutely no hyper-parameter, using validation data during training is fine. Otherwise, we would like to keep things simple and standardized, so we only allow validation set to be used for hyper-parameter search, not for training models. I see your point but this is to keep things simple and standardized. We have told the same thing to other people who have contacted us the same thing.

[Chillee]

I see your point but this is to keep things simple and standardized.

I think this is understandable. It probably disadvantages simpler methods with fewer hyperparameters, but all benchmarks have some kinds of biases :)

If there is absolutely no hyper-parameter, using validation data during training is fine.

Considering the guidelines, I think that even in this case, validation data should not be allowed. What is and isn't a hyperparameter is quite ambiguous - for example, say I define a variant of label propagation that always sets alpha=0.9.

[weihua916]

That's a good point. It's probably better to always disallow people to train on validation sets. I am aware that it is disadvantageous to simpler models, but it's hard to draw a clear line of what a simple model is. So we should probably just stick with the clearest rule--- no training on the validation set.

[Chillee]

Well, if you have no parameters, it's hard to call it "training on the validation set" haha. I suppose the rule is: "no touching the validation set for anything except for tuning hyperparameters".

Actually, I'm not sure this is a reasonable guideline either. For example, imagine I simply had one hyperparameter for each node, and I tuned it until my hyperparameters encoded the validation set's labels. Then, I run label propagation using my "hyperparameters" (ie: the validation labels). For example, you could imagine doing this: https://arxiv.org/abs/1911.02590

Perhaps another reasonable guideline is that you cannot do gradient based optimization based off of your validation set. In that case, that would allow methods like label propagation with 0 learnable parameters, but wouldn't allow linear layers to use the validation set.

[arbenson]

I would like a better explanation of the guidelines, @weihua916. If the labels help, why not use them? Isn't the goal of the benchmark to find new and good methods for challenging problems? You are placing a severe limitation that hurts the creativity of the algorithms that people can apply.

Consider this simple example. Suppose I have a time series dataset, where the output (labels) are given by a sequence
0, 1, 1, 2, 3, 5, 8, 13, 21
(the next label is the sum of the previous two labels). Now suppose that the first five are for training, the next two are for validation, and the last two are for test. I learn a model on the training data, using the validation labels and can correctly predict that the output is the sum of the previous two numbers. Now I want to make predictions on test data. Not using the validation labels (5 and 8) in the predictions would be absurd. Datasets like the arxiv one are time series! We should be trying to model what happens over time.

Also, saying "disallow people to train on validation sets" is ambiguous --- does that mean no hyperparameter tuning either? The validation sets are used in training the models. Also, unsupervised embedding methods are using the entire graph to form the embedding. Are those disallowed? It doesn't look like it, as I see deepwalk and node2vec submissions.

Is the benchmark only for GNN models and only for training GNN models in a particular way? If so, please make that clear.

Finally, I'll just add that there is precedent in image classification benchmarks for using some out-of-the box methods to boost performance substantially (http://i.stanford.edu/~julian/pdfs/eccv2012.pdf).

[weihua916]

Thanks for your opinions. We will discuss this internally and get back to you.

A quick clarification: What is definitely allowed in the current node classification datasets is the use of validation and test nodes (their features and connectivity information, not their labels) during training and inference.

[weihua916]

Hi, thanks for your patience. Below is the general guideline of how we expect people to use train/validation/test labels.

• Train labels: Training model parameters (can be used in whatever ways, e.g., for gradient-descent, architecture search, model input)
• Validation labels: Meant for standard hyper-parameters tuning (not allowed: gradient-based search, super exhaustive architecture search, use as model input)
• Test labels: Final model evaluation

The rationale behind the above choice is as follows:
1. Standardize and simplify the experimental setting.
While the most practically-relevant setting is to merge train and validation sets and let users decide how they utilize the merged set, in OGB, we have decided to provide the train/validation splits on our side to make the model comparison across papers easy and standardized.
2. Any ML model can make use of validation labels to improve the performance.
While label propagation can certainly improve the model performance by incorporating labels from the validation set, deep learning models can also take advantage of the validation labels in similar ways. Although there are more hyper-parameters in DL models (especially early stopping), all those hyper-parameters can be chosen and fixed based on validation labels. After those hyper-parameters are fixed, DL models can be directly trained on the validation labels. If carefully tuned, the performance of DL models may improve with validation labels. That being said, it is totally valid for you to demonstrate and claim in your work that label propagation still benefits more from validation labels than carefully-tuned DL models; however, drawing a clear distinction between models that can and cannot benefit from the direct use of validation labels is hard, so for the sake of OGB leaderboards, we would like to stick with the simple rule: validation labels can be only used for standard hyper-parameter search.
3. Comparison between models that do and do not utilize validation labels could be highly unfair when validation label size is large.
In OGB, some datasets contain lots of validation labels (perhaps more than necessary to get an accurate estimate of the test performance). For instance, in ogbn-arxiv, the number of validation data is more than 30% of the training data. Explicitly using these validation labels would certainly boost the test performance a lot, but it would be unfair to the existing models that do not explicitly use validation labels during training (as mentioned above, those models could have benefited more from validation labels). While we thought about reducing the validation label size to mitigate the issue, this would create a lot of additional work (re-running all the experiments as well as all the existing submissions).

Regarding the time-series nature of the dataset, we completely agree that the most recent validation labels should be utilized. For the current node classification datasets, however, we regard them as larger and more challenging variants of Cora and CiteSeer, rather than time-series datasets; thus, we prioritize the conventional evaluation protocol (based on the above three rationales) over the evaluation protocol tailored to time-series prediction. That being said, our graphs are still realistic, large, and contain temporal information. They can be used in any ways you want to demonstrate the effectiveness of your time-series model, i.e., you can create your own experimental settings and claim that your model utilizes validation labels much more effectively than conventional models. If time-series models start to show promise, the community can create a more specialized time-series dataset, and we can help standardized the dataset and the task.

On the other hand, for ogbl-collab (predicting near-future academic collaboration based on the past collaboration), we realized that the dataset is inherently about time-series, and incorporating the most recent validation edges in the model is crucial and necessary; otherwise, the model would trivially predict the same edge score for the collaboration between authors regardless of validation (corresponding to 2018 collaboration) or test (corresponding to 2019 collaboration), which is conceptually problematic for the particular application. Therefore, we would allow the use of the validation labels (i.e., edges) for both model input and model training only for this dataset. In our experiments, we find that using validation edges at GraphSAGE’s inference time greatly increases the test performance (48% -> 54%) despite the fact that the number of validation edges is quite small (around 4% of training edges).

In summary, we still hope OGB to provide a useful set of datasets to benchmark models and play its role in distinguishing useful models from others, while maintaining the simplicity and standardization in experimental settings. Thank you again for bringing this up. Please let us know if you need further clarification.

Best,
OGB Team

[arbenson]

Points 1 + 2. There is a much better solution that keeps a clean standard. Instead, let any model use all of the training and validation data and enforce that hyperparameters should be tuned using the given training / validation splits. If this helps DL models, then great! The whole point of graph learning for transductive node prediction is that the data are not i.i.d. and it's a semi-supervised problem. Your arguments would make sense for standard i.i.d. supervised problems.
Edit: A potential downside of this proposed approach is that, in practice, it might be more difficult to enforce. However, I still think it is a better setup.

Point 3. Having large amounts of validation labels is an artifact of designing the benchmark only for models that need lots of hyperparameter tuning. If you use the fair setup described above (use all of the training and validation data in your model but tune hyperparameters on the given splits), then this issue is mitigated.
Edit: the issue here is the fact that splits are done by time, which determines the size of the validation set.

[arbenson]

For the current node classification datasets, however, we regard them as larger and more challenging variants of Cora and CiteSeer, rather than time-series datasets

If time-series models start to show promise, the community can create a more specialized time-series dataset

This seems to contradict the paper on the benchmark:
https://arxiv.org/abs/2005.00687
In several locations, the paper explicitly explains how the benchmark intentionally uses time as a split in node classification problems (e.g., in Table 2).

To quote the paper...

...suggesting that the realistic time split is indeed more challenging than the random split, providing an opportunity to improve the out-of-distribution generalization performance

... so OGB has already established the time series benchmarks!

The train / validation / test splits emphasize distributional shifts that make the problems difficult. By disallowing the use of validation labels, I think it is very difficult to evaluate whether or not new models are actually making progress on this challenge.

[weihua916]

The train / validation / test splits emphasize distributional shifts that make the problems difficult. By disallowing the use of validation labels, I think it is very difficult to evaluate whether or not new models are actually making progress on this challenge.

This is a great point, and definitely worth investigating. We will think about how to encourage this direction of research.

[weihua916]

Edit: A potential downside of this proposed approach is that, in practice, it might be more difficult to enforce. However, I still think it is a better setup.

As we discussed, there are pros and cons to different approaches. I completely agree that your proposed approach is practically sound and closer to the realistic use case. The cons are that it could result in many different experimental protocols (e.g., how validation labels are used, how they are combined with training sets), making it hard to compare different models directly. Combining train+validation could also encourage cheating on the public test set (not exactly following the proposed protocol).