Unable to reproduce the results in the paper

[hengruizhang98]

Dear authors,

Thanks for sharing the codes! While reproducing your method, I find it difficult to obtain the results presented in the table. Also, the performance is very unstable.

For example, I use the following commands to condense the Cora dataset into a small graph of 140 nodes:

python train_SGDD.py --dataset cora --nlayers=2 --beta 0.1 --r=1.0 --gpu_id=0

The best test accuracy obtained is only about 0.75 at about 400th epoch:

Similarly, I also met the "NAN" loss issue after training a large number of epochs (sometimes it is 400-600th, some times it is after 1000-th).

Another question is about "ogbn-arxiv" dataset. I noticed that the inner_loop is set 0 for ogbn-arxiv dataset. Doesn't it mean that the GNN classifier will not be trained? Also, I failed to obtain a decent performance on ogbn-arxiv dataset using the codes.

My environment is as follows:

absl-py==2.0.0
aiohttp==3.8.6
aiosignal==1.3.1
anndata==0.10.3
anyio @ file:///tmp/build/80754af9/anyio_1644463572971/work/dist
appdirs==1.4.4
argon2-cffi @ file:///opt/conda/conda-bld/argon2-cffi_1645000214183/work
argon2-cffi-bindings @ file:///tmp/build/80754af9/argon2-cffi-bindings_1644569679365/work
array-api-compat==1.4
asttokens @ file:///opt/conda/conda-bld/asttokens_1646925590279/work
async-timeout==4.0.3
attrs @ file:///croot/attrs_1668696182826/work
backcall @ file:///home/ktietz/src/ci/backcall_1611930011877/work
beautifulsoup4 @ file:///opt/conda/conda-bld/beautifulsoup4_1650462163268/work
bleach @ file:///opt/conda/conda-bld/bleach_1641577558959/work
brotlipy==0.7.0
cachetools==5.3.1
cdsapi==0.6.1
certifi @ file:///croot/certifi_1671487769961/work/certifi
cffi @ file:///croot/cffi_1670423208954/work
charset-normalizer @ file:///tmp/build/80754af9/charset-normalizer_1630003229654/work
clean-fid==0.1.35
click==8.1.3
cloudpickle==2.2.1
cmake==3.26.3
comm @ file:///croot/comm_1671231121260/work
contourpy==1.0.7
coolgpus==0.23
cryptography @ file:///croot/cryptography_1677533068310/work
cycler==0.11.0
daal4py==2023.0.2
dask==2023.3.2
debugpy @ file:///tmp/build/80754af9/debugpy_1637091799509/work
decorator @ file:///opt/conda/conda-bld/decorator_1643638310831/work
deeprobust==0.2.9
defusedxml @ file:///tmp/build/80754af9/defusedxml_1615228127516/work
dgl==1.0.1+cu117
entrypoints @ file:///tmp/build/80754af9/entrypoints_1649926439650/work
exceptiongroup==1.1.3
executing @ file:///opt/conda/conda-bld/executing_1646925071911/work
fastjsonschema @ file:///opt/conda/conda-bld/python-fastjsonschema_1661371079312/work
filelock==3.11.0
flit_core @ file:///opt/conda/conda-bld/flit-core_1644941570762/work/source/flit_core
fonttools==4.39.3
frozenlist==1.4.0
fsspec==2023.4.0
gdown==4.7.1
gensim==3.8.3
geomloss==0.2.6
get-annotations==0.1.2
google-auth==2.23.0
google-auth-oauthlib==1.0.0
grpcio==1.58.0
h5py==3.10.0
huggingface-hub==0.13.4
idna @ file:///croot/idna_1666125576474/work
imageio==2.33.0
importlib-metadata==6.3.0
importlib-resources==5.12.0
install==1.3.5
ipykernel @ file:///croot/ipykernel_1671488378391/work
ipython @ file:///croot/ipython_1676582224036/work
ipython-genutils @ file:///tmp/build/80754af9/ipython_genutils_1606773439826/work
ipywidgets==8.0.6
jedi @ file:///tmp/build/80754af9/jedi_1644297102865/work
Jinja2 @ file:///croot/jinja2_1666908132255/work
joblib @ file:///croot/joblib_1666298844297/work
jsonschema @ file:///croot/jsonschema_1676558650973/work
jupyter-server @ file:///croot/jupyter_server_1671707632269/work
jupyter_client @ file:///croot/jupyter_client_1676329080601/work
jupyter_core @ file:///croot/jupyter_core_1676538566912/work
jupyterlab-pygments @ file:///tmp/build/80754af9/jupyterlab_pygments_1601490720602/work
jupyterlab-widgets==3.0.7
keopscore==2.1.2
kiwisolver==1.4.4
lazy_loader==0.3
lightning-bolts==0.7.0
lightning-utilities==0.9.0
littleutils==0.2.2
llvmlite==0.41.1
locket==1.0.0
lxml @ file:///opt/conda/conda-bld/lxml_1657545139709/work
Markdown==3.4.4
MarkupSafe @ file:///opt/conda/conda-bld/markupsafe_1654597864307/work
matplotlib==3.7.1
matplotlib-inline @ file:///opt/conda/conda-bld/matplotlib-inline_1662014470464/work
mistune @ file:///tmp/build/80754af9/mistune_1607364877025/work
mkl-fft==1.3.1
mkl-random @ file:///tmp/build/80754af9/mkl_random_1626186066731/work
mkl-service==2.4.0
multidict==6.0.4
natsort==8.4.0
nb-conda==2.2.1
nb-conda-kernels @ file:///tmp/build/80754af9/nb_conda_kernels_1636035787744/work
nbclassic @ file:///croot/nbclassic_1676902904845/work
nbclient @ file:///tmp/build/80754af9/nbclient_1650290509967/work
nbconvert @ file:///croot/nbconvert_1668450669124/work
nbformat @ file:///croot/nbformat_1670352325207/work
nest-asyncio @ file:///croot/nest-asyncio_1672387112409/work
networkx @ file:///opt/conda/conda-bld/networkx_1657784097507/work
notebook @ file:///croot/notebook_1668179881751/work
notebook_shim @ file:///croot/notebook-shim_1668160579331/work
numba==0.58.1
numpy==1.23.0
oauthlib==3.2.2
ogb==1.3.6
outdated==0.2.2
packaging @ file:///croot/packaging_1671697413597/work
pandas==1.5.3
pandocfilters @ file:///opt/conda/conda-bld/pandocfilters_1643405455980/work
parso @ file:///opt/conda/conda-bld/parso_1641458642106/work
partd==1.4.0
pastel==0.2.1
patsy==0.5.3
pexpect @ file:///tmp/build/80754af9/pexpect_1605563209008/work
pickleshare @ file:///tmp/build/80754af9/pickleshare_1606932040724/work
Pillow==9.4.0
platformdirs @ file:///opt/conda/conda-bld/platformdirs_1662711380096/work
poethepoet==0.10.0
pooch @ file:///tmp/build/80754af9/pooch_1623324770023/work
POT==0.9.1
prometheus-client @ file:///tmp/abs_d3zeliano1/croots/recipe/prometheus_client_1659455100375/work
prompt-toolkit @ file:///croot/prompt-toolkit_1672387306916/work
protobuf==4.24.3
psutil @ file:///opt/conda/conda-bld/psutil_1656431268089/work
ptyprocess @ file:///tmp/build/80754af9/ptyprocess_1609355006118/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
pure-eval @ file:///opt/conda/conda-bld/pure_eval_1646925070566/work
pyasn1==0.5.0
pyasn1-modules==0.3.0
pybind11==2.10.4
pycparser @ file:///tmp/build/80754af9/pycparser_1636541352034/work
Pygments @ file:///opt/conda/conda-bld/pygments_1644249106324/work
pykeops==2.1.2
pynndescent==0.5.10
pyOpenSSL @ file:///croot/pyopenssl_1677607685877/work
pyparsing @ file:///opt/conda/conda-bld/pyparsing_1661452539315/work
pyrsistent @ file:///tmp/build/80754af9/pyrsistent_1636110951836/work
PySocks @ file:///tmp/build/80754af9/pysocks_1605305812635/work
python-dateutil @ file:///tmp/build/80754af9/python-dateutil_1626374649649/work
pytorch-lightning==1.9.5
pytz==2023.3
PyYAML==6.0
pyzmq @ file:///opt/conda/conda-bld/pyzmq_1657724186960/work
requests @ file:///opt/conda/conda-bld/requests_1657734628632/work
requests-oauthlib==1.3.1
rsa==4.9
scanpy==1.9.6
scikit-image==0.22.0
scikit-learn @ file:///croot/scikit-learn_1676911643119/work
scikit-learn-intelex==20230228.214108
scipy==1.10.0
scprep==1.2.3
seaborn==0.12.2
Send2Trash @ file:///tmp/build/80754af9/send2trash_1632406701022/work
session-info==1.0.0
six @ file:///tmp/build/80754af9/six_1644875935023/work
smart-open==6.4.0
sniffio @ file:///tmp/build/80754af9/sniffio_1614030464178/work
sortedcontainers==2.1.0
soupsieve @ file:///croot/soupsieve_1666296392845/work
stack-data @ file:///opt/conda/conda-bld/stack_data_1646927590127/work
statsmodels==0.14.0
stdlib-list==0.9.0
tbb==2021.10.0
tensorboard==2.14.0
tensorboard-data-server==0.7.1
tensorboardX==2.6.2.2
terminado @ file:///croot/terminado_1671751832461/work
texttable==1.7.0
threadpoolctl @ file:///Users/ktietz/demo/mc3/conda-bld/threadpoolctl_1629802263681/work
tifffile==2023.9.26
timm==0.6.13
tinycss2 @ file:///croot/tinycss2_1668168815555/work
tomlkit==0.12.2
toolz==0.12.0
torch==1.12.1
torch-cluster @ file:///usr/share/miniconda/envs/test/conda-bld/pytorch-cluster_1656604971222/work
torch-geometric @ file:///usr/share/miniconda/envs/test/conda-bld/pyg_1669879312936/work
torch-scatter @ file:///usr/share/miniconda/envs/test/conda-bld/pytorch-scatter_1669298921716/work
torch-sparse @ file:///usr/share/miniconda/envs/test/conda-bld/pytorch-sparse_1671715073705/work
torchaudio==0.12.1
torchcde==0.2.5
torchcfm==1.0.4
torchdiffeq==0.2.3
torchdyn==1.0.6
torchmetrics==1.2.0
torchsde==0.2.6
torchvision==0.13.1
tornado @ file:///opt/conda/conda-bld/tornado_1662061693373/work
tqdm @ file:///opt/conda/conda-bld/tqdm_1664392687731/work
traitlets @ file:///croot/traitlets_1671143879854/work
trampoline==0.1.2
typing_extensions @ file:///croot/typing_extensions_1669924550328/work
tzdata==2023.3
umap-learn==0.5.4
urllib3 @ file:///croot/urllib3_1673575502006/work
urwid==2.0.1
wcwidth @ file:///Users/ktietz/demo/mc3/conda-bld/wcwidth_1629357192024/work
webencodings==0.5.1
websocket-client @ file:///tmp/build/80754af9/websocket-client_1614803975924/work
Werkzeug==2.3.7
widgetsnbextension==4.0.7
xarray==2023.3.0
yarl==1.9.2
zipp==3.15.0

[Suchun-sv]

Thanks for your attention of our work and kindly sharing the environment, maybe there are some inconsistent environment settings. I'll check the code based on your environment as soon as possible, sorry for the inconvenience.

[Suchun-sv]

For the ogbn-arxiv question, since we followed the great GCond work [1]. The default inner 0 is actually used to mimic the one-step algorithm [2]. The recommended setting for ogbn-arxiv is in the range of 3-5, but I cannot ensure if this work in your environment. Thanks for your attention again.

[1] https://github.com/ChandlerBang/GCond
[2] Wei Jin, et.al. 2022. Condensing Graphs via One-Step Gradient Matching. KDD '22.

[hengruizhang98]

Thanks for your quick response!

[Suchun-sv]

Thanks for you attention to our work, we find the bug might be the incompatible problem, some function that used to calculating the eigenvalue are removed in the new version of PyTorch. However, we do not catch this exception in this version code.

We fixed these bugs and will soon release the new version code. Hope that will help you.