PropertyGraph set index to vertex and edge ids #2523

eriknw · 2022-08-09T23:16:41Z

~~Currently, this only does SG version for #2401. MG is still TODO.~~

This also doesn't change anything user-facing (yet).

Currently, this only does SG version for rapidsai#2401. MG is still TODO. This also doesn't change anything user-facing (yet).

codecov-commenter · 2022-08-10T01:17:11Z

Codecov Report

❗ No coverage uploaded for pull request base (branch-22.10@238f909). Click here to learn what that means.
Patch has no changes to coverable lines.

❗ Current head 6a365ec differs from pull request most recent head 6a7ff4e. Consider uploading reports for the commit 6a7ff4e to get more accurate results

Additional details and impacted files

@@               Coverage Diff               @@
##             branch-22.10    #2523   +/-   ##
===============================================
  Coverage                ?   60.13%           
===============================================
  Files                   ?      111           
  Lines                   ?     6184           
  Branches                ?        0           
===============================================
  Hits                    ?     3719           
  Misses                  ?     2465           
  Partials                ?        0

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

alexbarghi-nv · 2022-08-18T18:41:47Z

@eriknw is this ready for review?

eriknw · 2022-08-18T18:45:44Z

Yeah, for SG. I'm going to put MG into a different WIP branch, because of issues I'm having with dask_cudf.

I still want to add one thing: the ability to add vertex data that already has the vertex id as the index.

alexbarghi-nv · 2022-08-18T18:46:36Z

ok, I'll wait until you've made that addition, I wasn't sure if that was going into this branch or not

rlratzel

LGTM, I had just one optional minor question/suggestion which need not hold up approval.

rlratzel · 2022-08-19T18:45:59Z

python/cugraph/cugraph/structure/property_graph.py

+        if self.__series_type is cudf.Series:
+            latest = {n: self.__vertex_prop_dataframe[n]
+                      for n in self.__vertex_prop_dataframe.columns}
+        else:
+            latest = self.__vertex_prop_dataframe.to_dict('series')


I'm curious how much faster/more efficient it is to create latest by separately using .to_dict() only for pandas vs. just having one way that works with both. Granted it's not many more lines of code to support both, but still might be worth only doing it one way if the speedup is negligible.

I doubt that it's any faster, but I aspire to write canonical pandas, and, who knows, maybe pandas can do something smarter and/or maybe cudf will add this API in the future.

BradReesWork · 2022-09-14T15:19:33Z

rerun tests

This includes a slow workaround for rapidsai/cudf#11550

BradReesWork · 2022-09-19T13:17:41Z

rerun tests

alexbarghi-nv · 2022-09-21T02:59:41Z

rerun tests

… branch-22.10

Some updates to "renumber by type"

BradReesWork · 2022-09-23T17:32:21Z

rerun tests

VibhuJawa

Can we add a check that we fetch ids in correct order and behave correctly with repeated ids ? (for both PropertyGraph and MGProperty Graph and for both get_edge_data and get_node_data).

See example:

import cudf
import cugraph
from cugraph.experimental import PropertyGraph
import numpy as np

src_ser = cudf.Series([1, 1, 1, 2])
dst_ser = cudf.Series([2, 3, 4, 1])
df = cudf.DataFrame(
    {"src": src_ser, "dst": dst_ser, "edge_feat": np.arange(len(src_ser))}
)
pg = PropertyGraph()
pg.add_edge_data(df, vertex_col_names=['src', 'dst'])

Current Main Behavior:

pg.get_edge_data(edge_ids = [2,1,3,1], columns=['edge_feat'])

_SRC_	_DST_	_EDGE_ID_	_TYPE_	edge_feat
1	1	3	1		1
2	1	4	2		2
3	2	1	3		3

Expected behavior on this PR.

_SRC_	_DST_	_EDGE_ID_	_TYPE_	edge_feat
0	1	4	2		2
1	1	3	1		1
2	2	1	3		3
3	1	3	1		1

BradReesWork · 2022-09-30T14:23:30Z

rerun tests

This PR fixes rapidsai/graph_dl#27 This PR fixes rapidsai/graph_dl#43 This PR fixes rapidsai/graph_dl#39 **Tests Added:** Single GPU: - [x] APIs like num_nodes, num_edges - [x] test_sampling_basic - [x] test_sampling_homogeneous_gs_in_dir - [x] test_sampling_homogeneous_gs_out_dir - [x] test_sampling_gs_homogeneous_neg_one_fanout - [x] test_sampling_gs_heterogeneous_in_dir - [x] test_sampling_gs_heterogeneous_out_dir - [x] test_sampling_gs_heterogeneous_neg_one_fanout Multi GPU: - [x] APIs like num_nodes, num_edges - [x] test_sampling_basic - [x] test_sampling_homogeneous_gs_in_dir - [x] test_sampling_homogeneous_gs_out_dir - [x] test_sampling_gs_homogeneous_neg_one_fanout - [x] test_sampling_gs_heterogeneous_in_dir - [x] test_sampling_gs_heterogeneous_out_dir - [x] test_sampling_gs_heterogeneous_neg_one_fanout Bugs to reproduce: - [ ] Repro heterogeneous single gpu hang outside pytest - [x] Repro hetrogenous multi gpu incorrect results for with_replace=False #2760 - [x] Repro hetrogenous incorrect results for different amount of GPUs #2761 Tests that depend upon #2523 - [x] Add minimal example to PR to ensure it gets fixed Added comment here: #2523 (review) - [x] test_get_node_storage_gs (Failing cause of a PG bug) - [x] test_get_edge_storage_gs (Failing cause of a PG bug) - [x] test_get_node_storage_gs (Failing cause of a PG bug) - [x] test_get_edge_storage_gs (Failing cause of a PG bug) Authors: - Vibhu Jawa (https://github.com/VibhuJawa) Approvers: - Rick Ratzel (https://github.com/rlratzel) - Brad Rees (https://github.com/BradReesWork) URL: #2592

python/cugraph/cugraph/tests/mg/test_mg_property_graph.py

BradReesWork · 2022-09-30T19:58:03Z

@gpucibot merge

This PR fixes the below errors that have popped up in MNMG testing. - [x] fix_out of index keys on MNMG graphs - [x] fix loc/get_node_storage error on MNMG graphs (Work around rapidsai/cudf#11877) - [x] Clear Cached Properties when they become invalid - [x] Remove 6 pytest skipping as both these PRs have landed - #2751 - #2523 - [x] Change `vertex_col_names` to `node_col_names` to match DGL - [x] Ensure MNMG tests pass - [x] Work around the PG bug and also prevent redundant conversion to lists Authors: - Vibhu Jawa (https://github.com/VibhuJawa) Approvers: - Rick Ratzel (https://github.com/rlratzel) - Erik Welch (https://github.com/eriknw) URL: #2786

PropertyGraph set index to vertex and edge ids

2a6c9cf

Currently, this only does SG version for rapidsai#2401. MG is still TODO. This also doesn't change anything user-facing (yet).

eriknw requested a review from a team as a code owner August 9, 2022 23:16

eriknw added 2 commits August 10, 2022 10:55

Update graph_store

4c93f77

flake8

2631715

eriknw mentioned this pull request Aug 17, 2022

[BUG] cudf incorrect when merging on both index level and column when specifying left_on and right_on rapidsai/cudf#11550

Closed

alexbarghi-nv added python non-breaking Non-breaking change labels Aug 17, 2022

alexbarghi-nv added this to the 22.10 milestone Aug 17, 2022

alexbarghi-nv assigned eriknw Aug 17, 2022

alexbarghi-nv added the improvement Improvement / enhancement to an existing function label Aug 18, 2022

rlratzel approved these changes Aug 19, 2022

View reviewed changes

This was referenced Aug 31, 2022

Reduce cuGraph Sampling Overhead for PyG #2653

Merged

Allow setting the edge weights to edge ids as currently required by uniform_neighbor_sample #2614

Closed

Merge branch 'branch-22.10' into pg_set_index

ff0f80c

eriknw mentioned this pull request Sep 8, 2022

[BUG] df.loc with list of length 1 does not propagate index name rapidsai/cudf#11664

Closed

BradReesWork removed python labels Sep 14, 2022

Set index to vertex or edge IDs in PG for MG

99c2e0e

This includes a slow workaround for rapidsai/cudf#11550

alexbarghi-nv and others added 3 commits September 21, 2022 14:03

Merge branch 'pg_set_index' of https://github.com/eriknw/cugraph into…

1a1e039

… branch-22.10

fixes

9bbf048

Merge branch 'branch-22.10' into pg_set_index

6a365ec

Some updates to "renumber by type"

eriknw added 4 commits September 23, 2022 11:07

Merge branch 'pg_set_index_fixes' into pg_set_index

674f5ff

Add workaround when index dtypes are different in dask_cudf.concat

1feb5e8

Merge branch 'branch-22.10' into pg_set_index

6a7ff4e

flake8

21a3b89

VibhuJawa mentioned this pull request Sep 27, 2022

[REVIEW]Hetrograph and dask_cudf support #2592

Merged

24 tasks

VibhuJawa suggested changes Sep 29, 2022

View reviewed changes

eriknw added 2 commits September 30, 2022 10:18

Add test to ensure ids are fetch in correct order (and when repeated)

aa591b8

Merge branch 'branch-22.10' into pg_set_index

5c8a33f

VibhuJawa suggested changes Sep 30, 2022

View reviewed changes

python/cugraph/cugraph/tests/mg/test_mg_property_graph.py Show resolved Hide resolved

rapids-bot bot merged commit d51b1f9 into rapidsai:branch-22.10 Sep 30, 2022

VibhuJawa mentioned this pull request Oct 8, 2022

[REVIEW] Fix MNMG failures in mg_dgl_extensions #2786

Merged

7 tasks

VibhuJawa mentioned this pull request Nov 7, 2022

[REVIEW]Add remote storage support #2859

Merged

18 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PropertyGraph set index to vertex and edge ids #2523

PropertyGraph set index to vertex and edge ids #2523

eriknw commented Aug 9, 2022 •

edited

Loading

codecov-commenter commented Aug 10, 2022 •

edited

Loading

alexbarghi-nv commented Aug 18, 2022

eriknw commented Aug 18, 2022

alexbarghi-nv commented Aug 18, 2022 •

edited

Loading

rlratzel left a comment

rlratzel Aug 19, 2022

eriknw Sep 7, 2022

BradReesWork commented Sep 14, 2022

BradReesWork commented Sep 19, 2022

alexbarghi-nv commented Sep 21, 2022

BradReesWork commented Sep 23, 2022

VibhuJawa left a comment •

edited

Loading

BradReesWork commented Sep 30, 2022

BradReesWork commented Sep 30, 2022

PropertyGraph set index to vertex and edge ids #2523

PropertyGraph set index to vertex and edge ids #2523

Conversation

eriknw commented Aug 9, 2022 • edited Loading

codecov-commenter commented Aug 10, 2022 • edited Loading

Codecov Report

alexbarghi-nv commented Aug 18, 2022

eriknw commented Aug 18, 2022

alexbarghi-nv commented Aug 18, 2022 • edited Loading

rlratzel left a comment

Choose a reason for hiding this comment

rlratzel Aug 19, 2022

Choose a reason for hiding this comment

eriknw Sep 7, 2022

Choose a reason for hiding this comment

BradReesWork commented Sep 14, 2022

BradReesWork commented Sep 19, 2022

alexbarghi-nv commented Sep 21, 2022

BradReesWork commented Sep 23, 2022

VibhuJawa left a comment • edited Loading

Choose a reason for hiding this comment

BradReesWork commented Sep 30, 2022

BradReesWork commented Sep 30, 2022

eriknw commented Aug 9, 2022 •

edited

Loading

codecov-commenter commented Aug 10, 2022 •

edited

Loading

alexbarghi-nv commented Aug 18, 2022 •

edited

Loading

VibhuJawa left a comment •

edited

Loading