Add V2 code. #10

pn51 · 2023-09-29T23:53:38Z

Summary/Motivation:

V2 of the MGN codebase.

Legal Acknowledgement

By contributing to this software project, I agree to the following terms and conditions for my contribution:

I agree my contributions are submitted under the copyright and license terms described in the LICENSE.md file at the top level of this directory.
I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

pn51 · 2023-09-30T00:02:30Z

Also addresses #9.

ksbeattie

LGTM

pn51 · 2023-10-06T19:18:34Z

@ksbeattie Can you merge?

lbianchi-lbl · 2023-10-06T20:52:26Z

environment.yml

@@ -0,0 +1,168 @@
+name: ccsi_release


Suggested change

name: ccsi_release

name: ccsi-mgn-v2

lbianchi-lbl

I've tried creating the environment using the environment.yml file in this PR, and I encountered an error related to the versions tags used for some of the torch subpackages. However, even after removing the +... portion of the tags, the installation is still failing:

Installing pip dependencies: ...working... Ran pip subprocess with arguments:
['/opt/conda/envs/ccsi-mgn-v2/bin/python', '-m', 'pip', 'install', '-U', '-r', '/home/ludo/lbl/ccsi/MGN/condaenv.tjpx6ooi.requirements.txt']
Pip subprocess output:
Collecting charset-normalizer==3.2.0 (from -r /home/ludo/lbl/ccsi/MGN/condaenv.tjpx6ooi.requirements.txt (line 1))
  Obtaining dependency information for charset-normalizer==3.2.0 from https://files.pythonhosted.org/packages/a4/65/057bf29660aae6ade0816457f8db4e749e5c0bfa2366eb5f67db9912fa4c/charset_normalizer-3.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Using cached charset_normalizer-3.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (31 kB)
Collecting cmake==3.27.4.1 (from -r /home/ludo/lbl/ccsi/MGN/condaenv.tjpx6ooi.requirements.txt (line 2))
  Obtaining dependency information for cmake==3.27.4.1 from https://files.pythonhosted.org/packages/94/87/68536d2dde5acec492742c63bb71f43534eb7d3d83122cce3067c4abca2b/cmake-3.27.4.1-py2.py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata
  Using cached cmake-3.27.4.1-py2.py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (6.7 kB)
Collecting filelock==3.12.3 (from -r /home/ludo/lbl/ccsi/MGN/condaenv.tjpx6ooi.requirements.txt (line 3))
  Obtaining dependency information for filelock==3.12.3 from https://files.pythonhosted.org/packages/52/90/45223db4e1df30ff14e8aebf9a1bf0222da2e7b49e53692c968f36817812/filelock-3.12.3-py3-none-any.whl.metadata
  Using cached filelock-3.12.3-py3-none-any.whl.metadata (2.7 kB)
Collecting jinja2==3.1.2 (from -r /home/ludo/lbl/ccsi/MGN/condaenv.tjpx6ooi.requirements.txt (line 4))
  Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)
Collecting joblib==1.3.2 (from -r /home/ludo/lbl/ccsi/MGN/condaenv.tjpx6ooi.requirements.txt (line 5))
  Obtaining dependency information for joblib==1.3.2 from https://files.pythonhosted.org/packages/10/40/d551139c85db202f1f384ba8bcf96aca2f329440a844f924c8a0040b6d02/joblib-1.3.2-py3-none-any.whl.metadata
  Using cached joblib-1.3.2-py3-none-any.whl.metadata (5.4 kB)
Collecting lit==16.0.6 (from -r /home/ludo/lbl/ccsi/MGN/condaenv.tjpx6ooi.requirements.txt (line 6))
  Using cached lit-16.0.6.tar.gz (153 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting markupsafe==2.1.3 (from -r /home/ludo/lbl/ccsi/MGN/condaenv.tjpx6ooi.requirements.txt (line 7))
  Obtaining dependency information for markupsafe==2.1.3 from https://files.pythonhosted.org/packages/12/b3/d9ed2c0971e1435b8a62354b18d3060b66c8cb1d368399ec0b9baa7c0ee5/MarkupSafe-2.1.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Using cached MarkupSafe-2.1.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB)
Collecting mpmath==1.3.0 (from -r /home/ludo/lbl/ccsi/MGN/condaenv.tjpx6ooi.requirements.txt (line 8))
  Using cached mpmath-1.3.0-py3-none-any.whl (536 kB)
Collecting psutil==5.9.5 (from -r /home/ludo/lbl/ccsi/MGN/condaenv.tjpx6ooi.requirements.txt (line 9))
  Using cached psutil-5.9.5-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (282 kB)
Collecting pyparsing==3.1.1 (from -r /home/ludo/lbl/ccsi/MGN/condaenv.tjpx6ooi.requirements.txt (line 10))
  Obtaining dependency information for pyparsing==3.1.1 from https://files.pythonhosted.org/packages/39/92/8486ede85fcc088f1b3dba4ce92dd29d126fd96b0008ea213167940a2475/pyparsing-3.1.1-py3-none-any.whl.metadata
  Using cached pyparsing-3.1.1-py3-none-any.whl.metadata (5.1 kB)
Requirement already satisfied: pyqt5-sip==12.11.0 in /opt/conda/envs/ccsi-mgn-v2/lib/python3.10/site-packages/PyQt5_sip-12.11.0-py3.10-linux-x86_64.egg (from -r /home/ludo/lbl/ccsi/MGN/condaenv.tjpx6ooi.requirements.txt (line 11)) (12.11.0)
Collecting python-louvain==0.16 (from -r /home/ludo/lbl/ccsi/MGN/condaenv.tjpx6ooi.requirements.txt (line 12))
  Using cached python-louvain-0.16.tar.gz (204 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting scikit-learn==1.3.0 (from -r /home/ludo/lbl/ccsi/MGN/condaenv.tjpx6ooi.requirements.txt (line 13))
  Obtaining dependency information for scikit-learn==1.3.0 from https://files.pythonhosted.org/packages/5c/e9/ee572691a3fb05555bcde41826faad29ae4bc1fb07982e7f53d54a176879/scikit_learn-1.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Using cached scikit_learn-1.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (11 kB)
Collecting scipy==1.11.2 (from -r /home/ludo/lbl/ccsi/MGN/condaenv.tjpx6ooi.requirements.txt (line 14))
  Obtaining dependency information for scipy==1.11.2 from https://files.pythonhosted.org/packages/a8/cc/c36f3439f5d47c3b13833ce6687b43a040cc7638c502ac46b41e2d4f3d6f/scipy-1.11.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Using cached scipy-1.11.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (59 kB)
Collecting sympy==1.12 (from -r /home/ludo/lbl/ccsi/MGN/condaenv.tjpx6ooi.requirements.txt (line 15))
  Using cached sympy-1.12-py3-none-any.whl (5.7 MB)
Collecting threadpoolctl==3.2.0 (from -r /home/ludo/lbl/ccsi/MGN/condaenv.tjpx6ooi.requirements.txt (line 16))
  Obtaining dependency information for threadpoolctl==3.2.0 from https://files.pythonhosted.org/packages/81/12/fd4dea011af9d69e1cad05c75f3f7202cdcbeac9b712eea58ca779a72865/threadpoolctl-3.2.0-py3-none-any.whl.metadata
  Using cached threadpoolctl-3.2.0-py3-none-any.whl.metadata (10.0 kB)
Collecting torch==2.0.1 (from -r /home/ludo/lbl/ccsi/MGN/condaenv.tjpx6ooi.requirements.txt (line 17))
  Downloading torch-2.0.1-cp310-cp310-manylinux1_x86_64.whl (619.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 619.9/619.9 MB 1.6 MB/s eta 0:00:00
Collecting torch-cluster==1.6.1 (from -r /home/ludo/lbl/ccsi/MGN/condaenv.tjpx6ooi.requirements.txt (line 18))
  Downloading torch_cluster-1.6.1.tar.gz (53 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.8/53.8 kB 5.0 MB/s eta 0:00:00
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'error'

Pip subprocess error:
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-od6784yj/torch-cluster_b03cfecd7ea540e983026db333218c8a/setup.py", line 8, in <module>
          import torch
      ModuleNotFoundError: No module named 'torch'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

failed
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/conda/exceptions.py", line 1079, in __call__
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/conda_env/cli/main.py", line 80, in do_call
    exit_code = getattr(module, func_name)(args, parser)
  File "/opt/conda/lib/python3.7/site-packages/conda_env/cli/main_create.py", line 117, in execute
    result[installer_type] = installer.install(prefix, pkg_specs, args, env)
  File "/opt/conda/lib/python3.7/site-packages/conda_env/installers/pip.py", line 70, in install
    return _pip_install_via_requirements(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/conda_env/installers/pip.py", line 53, in _pip_install_via_requirements
    stdout, stderr = pip_subprocess(pip_cmd, prefix, cwd=pip_workdir)
  File "/opt/conda/lib/python3.7/site-packages/conda_env/pip_util.py", line 41, in pip_subprocess
    raise CondaEnvException("Pip failed")
conda_env.exceptions.CondaEnvException: Pip failed

Do you have more information on the Conda environment from which environment.yml was created, such as:

Which operating system?
How Pytorch was installed?

lbianchi-lbl · 2023-10-06T21:02:45Z

environment.yml

+      - torch==2.0.1+cu117
+      - torch-cluster==1.6.1+pt20cu117
+      - torch-geometric==2.3.1
+      - torch-scatter==2.1.1+pt20cu117


Suggested change

- torch==2.0.1+cu117

- torch-cluster==1.6.1+pt20cu117

- torch-geometric==2.3.1

- torch-scatter==2.1.1+pt20cu117

- torch==2.0.1

- torch-cluster==1.6.1

- torch-geometric==2.3.1

- torch-scatter==2.1.1

It looks like this issue has to do with torch-cluster expecting an existing installation of torch. This means we won't be able to use a environment .yml file for installing torch-cluster.

I'll be adding a subsection in the repo's root README.md that will provide new installation instructions. (manual pip/conda installs, post-installation of torch)

…ls.data_utils add_edges_to_nodes

fixed bug in CF where incorrect parameter was being passed to GNN.uti…

pn51 · 2023-10-18T23:47:17Z

@lbianchi-lbl The README has been updated with env setup instructions. We'll keep environment.yml in the repo as a reference for version numbers, but it shouldn't be used with conda create due to the dependency issue @AmarSaini mentioned.

Add V2 code.

4b1f6e3

pn51 requested a review from ksbeattie September 29, 2023 23:53

pn51 added this to the EY23.4.1.G: New release of the MGN repo (integration with Subtask 4.3) milestone Sep 30, 2023

pn51 added 2 commits September 29, 2023 17:12

Updated publications list.

f8bbe86

Update release number format.

0986ffd

ksbeattie added the Priority:High High Priority Issue or PR label Oct 3, 2023

ksbeattie previously approved these changes Oct 3, 2023

View reviewed changes

lbianchi-lbl reviewed Oct 6, 2023

View reviewed changes

environment.yml

@@ -0,0 +1,168 @@

name: ccsi_release

Copy link

lbianchi-lbl Oct 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

name: ccsi_release

name: ccsi-mgn-v2

lbianchi-lbl requested changes Oct 6, 2023

View reviewed changes

Add environment setup instructions.

5dc9041

pn51 dismissed ksbeattie’s stale review via 5dc9041 October 11, 2023 14:43

pn51 and others added 4 commits October 11, 2023 11:14

Change "dpp" -> "ddp".

752bc76

fixed bug in CF where incorrect parameter was being passed to GNN.uti…

5f31a49

…ls.data_utils add_edges_to_nodes

non_fluid_nodes -> non_fluid_indices

ee54d58

Merge pull request #1 from jcadena001/v2_release

eab7f45

fixed bug in CF where incorrect parameter was being passed to GNN.uti…

pn51 requested review from lbianchi-lbl and ksbeattie October 25, 2023 16:42

lbianchi-lbl approved these changes Oct 31, 2023

View reviewed changes

ksbeattie merged commit a2a32f3 into CCSI-Toolset:main Oct 31, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add V2 code. #10

Add V2 code. #10

pn51 commented Sep 29, 2023

pn51 commented Sep 30, 2023

ksbeattie left a comment

pn51 commented Oct 6, 2023

lbianchi-lbl Oct 6, 2023

lbianchi-lbl left a comment

lbianchi-lbl Oct 6, 2023

AmarSaini Oct 7, 2023

pn51 commented Oct 18, 2023

Add V2 code. #10

Add V2 code. #10

Conversation

pn51 commented Sep 29, 2023

Summary/Motivation:

Legal Acknowledgement

pn51 commented Sep 30, 2023

ksbeattie left a comment

Choose a reason for hiding this comment

pn51 commented Oct 6, 2023

lbianchi-lbl Oct 6, 2023

Choose a reason for hiding this comment

lbianchi-lbl left a comment

Choose a reason for hiding this comment

lbianchi-lbl Oct 6, 2023

Choose a reason for hiding this comment

AmarSaini Oct 7, 2023

Choose a reason for hiding this comment

pn51 commented Oct 18, 2023