GPSConv Graph Transformer Layer

[rusty1s]

No description provided.

[codecov]

Codecov Report

Merging #6326 (975dfe1) into master (abc3ad2) will decrease coverage by 1.85%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #6326      +/-   ##
==========================================
- Coverage   86.26%   84.41%   -1.86%     
==========================================
  Files         385      386       +1     
  Lines       21445    21478      +33     
==========================================
- Hits        18499    18130     -369     
- Misses       2946     3348     +402     

Impacted Files	Coverage Δ
torch_geometric/nn/conv/__init__.py	100.00% <100.00%> (ø)
torch_geometric/nn/conv/gps_conv.py	100.00% <100.00%> (ø)
torch_geometric/nn/models/dimenet_utils.py	0.00% <0.00%> (-75.52%)	⬇️
torch_geometric/nn/models/dimenet.py	14.90% <0.00%> (-52.76%)	⬇️
torch_geometric/visualization/graph.py	68.25% <0.00%> (-26.99%)	⬇️
torch_geometric/profile/profile.py	33.33% <0.00%> (-25.39%)	⬇️
torch_geometric/nn/conv/utils/typing.py	83.75% <0.00%> (-15.00%)	⬇️
torch_geometric/nn/pool/asap.py	92.10% <0.00%> (-7.90%)	⬇️
torch_geometric/nn/inits.py	67.85% <0.00%> (-7.15%)	⬇️
torch_geometric/resolver.py	82.75% <0.00%> (-6.90%)	⬇️
... and 32 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[Jerome-Cong]

Hi, thanks for adding this layer! The paper of GraphGPS mentioned that the use of linear transformers (e.g. BigBird and Performer) dramatically improves the computation efficiency for big graphs (hundreds of nodes). It seems that GPSConv currently uses a vanilla transformer (i.e. the global attention). Will linear transformer options be one of the TODOs in the future?

[rusty1s]

Yes, we are happy to extend GraphGPS to support other transformers. Also appreciate a PR if you would like to contribute :)

[alexlyzhov]

pytorch_geometric/torch_geometric/nn/conv/gps_conv.py

Line 155 in 1bb0306

h = self.norm3(h, batch=batch)

Seems like norm3 is never used

[rusty1s]

Can you clarify what you mean?

[alexlyzhov]

It changes h, but h is not saved.

[rusty1s]

You are absolutely, right, thanks! Fixed this :(