GAT model

[jermainewang]

Graph attention networks. Also fixed some bugs.

[jermainewang]

Benchmark results: https://docs.google.com/spreadsheets/d/1qhKvNfPqYLrnSbBH4CP6KLVpvm5TOsPPtrFLTI7H_NU/edit?usp=sharing

Summary:
We compare the DGL implementation with the official implementation by the author: https://github.com/PetarV-/GAT
The author has two implementation:

• Dense: It treats the adj matrix as a tense matrix and use dense MatMult and dense softmax.
• Sparse: It uses sparse operations like SPMV and sparse_softmax.
  Note that the author uses Tensorflow while DGL uses Pytorch backend.

Naive implementation (looping over every node and do message passing) is extremely bad. It is 50-100x slower than the batched implementation. The DGL-GAT-batch-GPU is not much faster than CPU, which shows the significant overhead in the python side. The TF-GAT is much faster than current DGL-GAT for two reasons:

• Degree-bucketing is avoided in both dense and sparse implementation. For example, in the dense version, they fill the empty 0 slot with -Inf so softmax can be directly applied.
• DGL cannot use SPMV due to the custom reducer. TF-GAT-sparse is able to use SPMV. They use a sparse matrix similar to the adj matrix but with the non-zero values equal to the attention value. They can then multiple this sparse matrix with the node features to get the attended features from neighbors.

These two optimizations cannot be easily adopted by DGL right now as it requires us to understand what softmax and the user-defined reduce function is doing. We plan this to be future work.