Implement PEG from ICLR 2022 #4571

zoom-wang112358 · 2022-04-30T04:19:39Z

PEG uses a GCN layer with edge weights according to the distance between the end nodes of the edge and keep the positional encodings unchanged. The positional encodings could to be calculated by some graph embedding techniques, such as DeepWalk and Laplacian eigenmap.

Examples and descriptions could be found here.

for more information, see https://pre-commit.ci

codecov · 2022-04-30T04:22:43Z

Codecov Report

Merging #4571 (ed0ce31) into master (e8b6def) will increase coverage by 0.13%.
The diff coverage is 100.00%.

❗ Current head ed0ce31 differs from pull request most recent head dc8c640. Consider uploading reports for the commit dc8c640 to get more accurate results

@@            Coverage Diff             @@
##           master    #4571      +/-   ##
==========================================
+ Coverage   82.70%   82.83%   +0.13%     
==========================================
  Files         329      330       +1     
  Lines       17844    17744     -100     
==========================================
- Hits        14758    14699      -59     
+ Misses       3086     3045      -41

Impacted Files	Coverage Δ
torch_geometric/nn/conv/__init__.py	`100.00% <100.00%> (ø)`
torch_geometric/nn/conv/peg_conv.py	`100.00% <100.00%> (ø)`
torch_geometric/data/in_memory_dataset.py	`77.27% <0.00%> (-3.89%)`	⬇️
torch_geometric/data/separate.py	`97.77% <0.00%> (-2.23%)`	⬇️
torch_geometric/data/storage.py	`81.01% <0.00%> (-0.64%)`	⬇️
torch_geometric/data/data.py	`91.43% <0.00%> (-0.14%)`	⬇️
torch_geometric/data/hetero_data.py	`94.11% <0.00%> (-0.10%)`	⬇️
torch_geometric/data/__init__.py	`100.00% <0.00%> (ø)`
torch_geometric/nn/dense/mincut_pool.py	`100.00% <0.00%> (ø)`
torch_geometric/nn/models/autoencoder.py	`100.00% <0.00%> (ø)`
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e8b6def...dc8c640. Read the comment docs.

for more information, see https://pre-commit.ci

…tric

for more information, see https://pre-commit.ci

Padarn · 2022-05-02T02:38:18Z

torch_geometric/nn/conv/peg_conv.py

+        **kwargs (optional): Additional arguments of
+            :class:`torch_geometric.nn.conv.MessagePassing`.
+    """
+    def __init__(self, in_feats_dim: int, pos_dim: int, out_feats_dim: int,


Suggested change

def __init__(self, in_feats_dim: int, pos_dim: int, out_feats_dim: int,

def __init__(self, in_channels: int, pos_dim: int, out_channels: int,

Padarn · 2022-05-02T02:41:31Z

torch_geometric/nn/conv/peg_conv.py

+            torch.Tensor(in_feats_dim + in_feats_dim, out_feats_dim))
+        self.weight_noformer = Parameter(
+            torch.Tensor(in_feats_dim, out_feats_dim))
+        self.edge_mlp = nn.Sequential(nn.Linear(1, edge_mlp_dim),


[newbie question] somewhat confused by this - If you go from 1 dim back to 1 dim are you learning much?

Maybe I can use 'information bottleneck' to explain it. Though I go from 1 dim back to 1 dim, the network rids noisy input data of extraneous details as if by squeezing the information through a bottleneck, retaining only the features most relevant to the edge weight.

Padarn · 2022-05-02T02:43:38Z

torch_geometric/nn/conv/peg_conv.py

+        # return tuple
+        return self.update((hidden_out, coors_out), **update_kwargs)
+
+    def glorot(self, tensor):


available in torch_geometric.nn.inits

Padarn · 2022-05-02T02:43:50Z

torch_geometric/nn/conv/peg_conv.py

+            stdv = math.sqrt(6.0 / (tensor.size(-2) + tensor.size(-1)))
+            tensor.data.uniform_(-stdv, stdv)
+
+    def zeros(self, tensor):


same as above

Padarn · 2022-05-02T02:44:46Z

torch_geometric/nn/conv/peg_conv.py

+        r"""The initial call to start propagating messages.
+
+        Args:
+            `edge_index` holds the indices of a general (sparse)


Suggested change

`edge_index` holds the indices of a general (sparse)

edge_index (Tensor): holds the indices of a general (sparse)

Padarn · 2022-05-02T02:44:58Z

torch_geometric/nn/conv/peg_conv.py

+        Args:
+            `edge_index` holds the indices of a general (sparse)
+                assignment matrix of shape :obj:`[N, M]`.
+            size (tuple, optional) if none, the size will be inferred


Suggested change

size (tuple, optional) if none, the size will be inferred

size (tuple, optional): if none, the size will be inferred

Padarn · 2022-05-02T02:48:08Z

torch_geometric/nn/conv/peg_conv.py

+        return x_j if edge_weight is None else (PE_edge_weight *
+                                                edge_weight.view(-1, 1) * x_j)
+
+    def propagate(self, edge_index: Adj, size: Size = None, **kwargs):


Sorry I didn't fully follow the implementation yet - but I think you should be able to do this without needing to reimplement propagate - I.e you can put most of this logic inside your forward function.

+1 . We definitely do not want to override propagate here. Any reason you did this in the first place?

Thanks for your advice. I put the operational logic inside my forward function.

for more information, see https://pre-commit.ci

zoom-wang112358 · 2022-06-04T07:58:40Z

@rusty1s Have just modified the implementation. Please review all the files and let me know if anything more is required. Thanks so much.

rusty1s

Thanks for the updates. This looks good. Please add your layer to the README.md as well.

rusty1s · 2022-06-06T08:45:07Z

CHANGELOG.md

@@ -5,6 +5,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).

 ## [2.0.5] - 2022-MM-DD
 ### Added
+- Added the `PEG` model ([#4571](https://github.com/pyg-team/pytorch_geometric/pull/4571))


Suggested change

- Added the `PEG` model ([#4571](https://github.com/pyg-team/pytorch_geometric/pull/4571))

- Added the `PEGConv` layer from the ["Equivariant and Stable Positional Encoding for More Powerful Graph Neural Networks"](https://arxiv.org/abs/2203.00199>) paper ([#4571](https://github.com/pyg-team/pytorch_geometric/pull/4571))

rusty1s · 2022-06-06T08:46:53Z

torch_geometric/nn/conv/peg_conv.py

+    for More Powerful Graph Neural Networks"
+    <https://arxiv.org/abs/2203.00199>`_ paper.
+
+    $$X^{'},Z^{'}=(\sigma [(\hat{A} \odot M)XW],Z)$$


Can we follow the .. math:: syntax here as in other layers? The formula also looks wrong: https://pytorch-geometric--4571.org.readthedocs.build/en/4571/modules/nn.html#torch_geometric.nn.conv.PEGConv

rusty1s · 2022-06-06T08:47:38Z

torch_geometric/nn/conv/peg_conv.py

+    <https://arxiv.org/abs/2203.00199>`_ paper.
+
+    $$X^{'},Z^{'}=(\sigma [(\hat{A} \odot M)XW],Z)$$
+    where\ $M_{uv}=MLP(||Z_u-Z_v||),\forall u,v \in V$.


Here again. Use

:math:`M_{u,v} ...`

rusty1s · 2022-06-06T08:49:03Z

torch_geometric/nn/conv/peg_conv.py

+        return x_j if edge_weight is None else (PE_edge_weight *
+                                                edge_weight.view(-1, 1) * x_j)
+
+    def __repr__(self):


Can be dropped.

rusty1s · 2022-06-06T08:49:36Z

torch_geometric/nn/conv/peg_conv.py

+    def message(self, x_i: Tensor, x_j: Tensor, edge_weight: OptTensor,
+                pos) -> Tensor:
+        PE_edge_weight = self.edge_mlp(pos)
+        return x_j if edge_weight is None else (PE_edge_weight *


Shouldn't we always use edge_mlp here?

out = x_j * self.edge_mlp(pos) return out if edge_weight is None else edge_weight.view(-1, 1) * out

rusty1s · 2022-06-21T05:34:48Z

Hi @ZoomWang666, thanks again for this PR. I would like to merge it once the open conversations are resolved. Can you take a look if you have time and let me know? Best

for more information, see https://pre-commit.ci

…tric

for more information, see https://pre-commit.ci

…tric

for more information, see https://pre-commit.ci

zoom-wang112358 and others added 2 commits April 30, 2022 11:59

Implement PEG

2f19c21

[pre-commit.ci] auto fixes from pre-commit.com hooks

60ada00

for more information, see https://pre-commit.ci

zoom-wang112358 changed the title ~~Implement PEG~~ Implement PEG from ICLR 2022 Apr 30, 2022

zoom-wang112358 and others added 16 commits April 30, 2022 12:30

Merge branch 'master' into master

ff204e5

fix some bugs

1ab9097

fix some bugs

335e599

[pre-commit.ci] auto fixes from pre-commit.com hooks

ae29dfe

for more information, see https://pre-commit.ci

Update peg_conv.py

944842c

Merge branch 'master' of https://github.com/ZoomWang666/pytorch_geome…

4fd6424

…tric

[pre-commit.ci] auto fixes from pre-commit.com hooks

44b6220

for more information, see https://pre-commit.ci

Update peg_conv.py

1246010

[pre-commit.ci] auto fixes from pre-commit.com hooks

f6d5aa9

for more information, see https://pre-commit.ci

Update peg_conv.py

975567d

Update peg_conv.py

bbb6133

[pre-commit.ci] auto fixes from pre-commit.com hooks

ce88d7a

for more information, see https://pre-commit.ci

fix unexpected indentation

d64a7a8

[pre-commit.ci] auto fixes from pre-commit.com hooks

223b628

for more information, see https://pre-commit.ci

Unexpected indentation?

e423f3b

[pre-commit.ci] auto fixes from pre-commit.com hooks

8c38189

for more information, see https://pre-commit.ci

Padarn reviewed May 2, 2022

View reviewed changes

rusty1s assigned zoom-wang112358 May 2, 2022

rusty1s added feature 0 - Priority P0 labels May 2, 2022

zoom-wang112358 and others added 4 commits June 3, 2022 21:50

Update CHANGELOG.md

f846cf7

Merge branch 'master' into master

17fccca

[pre-commit.ci] auto fixes from pre-commit.com hooks

9ba2c5b

for more information, see https://pre-commit.ci

Update CHANGELOG.md

5957635

rusty1s reviewed Jun 6, 2022

View reviewed changes

rusty1s and others added 2 commits June 6, 2022 10:50

Merge branch 'master' into master

01ceaf9

update readme

6728676

rusty1s assigned zoom-wang112358 Jun 26, 2022

zoom-wang112358 and others added 20 commits June 28, 2022 03:26

Update CHANGELOG.md

2a9fff4

Update CHANGELOG.md

a3d6a4b

update log

a8c6c14

Merge branch 'master' into master

188e957

update log

ee17232

Update peg_conv.py

bbe2698

[pre-commit.ci] auto fixes from pre-commit.com hooks

e22a2ac

for more information, see https://pre-commit.ci

Update peg_conv.py

dc9be9b

Merge branch 'master' of https://github.com/ZoomWang666/pytorch_geome…

5ca1e59

…tric

[pre-commit.ci] auto fixes from pre-commit.com hooks

6ed1bfb

for more information, see https://pre-commit.ci

Update peg_conv.py

4cbe801

Merge branch 'master' of https://github.com/ZoomWang666/pytorch_geome…

09999fe

…tric

[pre-commit.ci] auto fixes from pre-commit.com hooks

43a0fed

for more information, see https://pre-commit.ci

Update peg_conv.py

d8fb443

Update peg_conv.py

ed0ce31

[pre-commit.ci] auto fixes from pre-commit.com hooks

b815b93

for more information, see https://pre-commit.ci

Update CHANGELOG.md

cb93362

Update CHANGELOG.md

ab072c5

Merge branch 'master' into master

53ba5aa

Update CHANGELOG.md

dc8c640

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement PEG from ICLR 2022 #4571

Implement PEG from ICLR 2022 #4571

zoom-wang112358 commented Apr 30, 2022

codecov bot commented Apr 30, 2022 •

edited

Padarn May 2, 2022

Padarn May 2, 2022 •

edited

zoom-wang112358 May 19, 2022

Padarn May 2, 2022

Padarn May 2, 2022

Padarn May 2, 2022

Padarn May 2, 2022

Padarn May 2, 2022

rusty1s May 2, 2022

zoom-wang112358 May 19, 2022

zoom-wang112358 commented Jun 4, 2022

rusty1s left a comment •

edited

rusty1s Jun 6, 2022

rusty1s Jun 6, 2022

rusty1s Jun 6, 2022

rusty1s Jun 6, 2022

rusty1s Jun 6, 2022

rusty1s commented Jun 21, 2022

	def __init__(self, in_feats_dim: int, pos_dim: int, out_feats_dim: int,
	def __init__(self, in_channels: int, pos_dim: int, out_channels: int,

	`edge_index` holds the indices of a general (sparse)
	edge_index (Tensor): holds the indices of a general (sparse)

	size (tuple, optional) if none, the size will be inferred
	size (tuple, optional): if none, the size will be inferred

	- Added the `PEG` model ([#4571](https://github.com/pyg-team/pytorch_geometric/pull/4571))
	- Added the `PEGConv` layer from the ["Equivariant and Stable Positional Encoding for More Powerful Graph Neural Networks"](https://arxiv.org/abs/2203.00199>) paper ([#4571](https://github.com/pyg-team/pytorch_geometric/pull/4571))

Implement PEG from ICLR 2022 #4571

Are you sure you want to change the base?

Implement PEG from ICLR 2022 #4571

Conversation

zoom-wang112358 commented Apr 30, 2022

codecov bot commented Apr 30, 2022 • edited

Codecov Report

Choose a reason for hiding this comment

Padarn May 2, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zoom-wang112358 commented Jun 4, 2022

rusty1s left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rusty1s commented Jun 21, 2022

codecov bot commented Apr 30, 2022 •

edited

Padarn May 2, 2022 •

edited

rusty1s left a comment •

edited