Graph Profiler: save() and load() Functionality #628

micdavis · 2022-09-14T17:21:50Z

Added functionality for save() and load() for the GraphProfiler
Wrote a test to cover this new functionality

taylorfturner · 2022-09-14T17:30:26Z

dataprofiler/profilers/graph_profiler.py

+        with open(filepath, "rb") as infile:
+            data = pickle.load(infile)
+
+        profiler = cls(None, options=None)


just any class? should there not be some logic around loading the correct class for the profiler?

cls is the class it is loading within, this is same logic in BaseProfiler which will allow us to merge all 3 profilers in the long term.

JGSweets · 2022-09-14T17:55:48Z

dataprofiler/profilers/graph_profiler.py


 import networkx as nx
 import numpy as np
 import pandas as pd
+import pickle


pickle is built-in so I think should be within the top imports

JGSweets · 2022-09-14T17:57:32Z

dataprofiler/profilers/graph_profiler.py

+            "_continuous_distribution": self._continuous_distribution,
+            "_categorical_distribution": self._categorical_distribution,
+            "metadata": self.metadata,
+            "__calculations": self.__calculations,


Do we need __calculations bc we aren't saving options, is that the reason?

don't see options used anywhere...

good call, we don't

JGSweets · 2022-09-14T18:01:06Z

dataprofiler/tests/profilers/test_graph_profiler.py

+        save_profile.profile["continuous_distribution"]["weight"].pop("properties")
+        load_profile.profile["continuous_distribution"]["weight"].pop("properties")


We should also assert these props are the same.
e.g. above:

save_continuous_distribution_props = profile.profile["continuous_distribution"]["weight"].pop("properties") # Then can (unsure if it is a dict or list or what format. self.assertEqual(save_continuous_distribution_props, load_continuous_distribution_props)

… in actual graph data to ensure load and save work correctly

micdavis · 2022-09-14T18:20:14Z

examples/great_expecations/README.md

@@ -2,17 +2,17 @@
 [Great Expectations](https://greatexpectations.io/) is an open source tool built around the idea that should always know what to expect from your data. It helps data teams eliminate pipeline debt, through data testing, documentation, and profiling.


pre-commit formatting

micdavis · 2022-09-14T18:20:21Z

examples/great_expecations/ge_expect_column_value_confidence_for_data_label.ipynb

@@ -183,4 +183,4 @@
 },


pre-commit formatting

micdavis · 2022-09-14T18:20:26Z

examples/great_expecations/ge_expect_column_values_vs_profile_min.ipynb

@@ -291,4 +291,4 @@
 },


pre-commit formatting

micdavis · 2022-09-14T18:20:32Z

examples/great_expecations/ge_expect_profile_numeric_columns_diff_in_range.ipynb

@@ -300,4 +300,4 @@
 },


pre-commit formatting

micdavis · 2022-09-14T18:20:38Z

examples/great_expecations/ge_expect_profile_numeric_columns_percent_diff_in_range.ipynb

@@ -300,4 +300,4 @@
 },


pre-commit formatting

taylorfturner · 2022-09-14T18:31:54Z

dataprofiler/profilers/graph_profiler.py

@@ -23,7 +25,6 @@ def __init__(self, data, options=None):
        Initialize Graph Profiler.

        :param data: data


Suggested change

:param data: data

:param data: data

:type name: String

I think this should be: one of these two types:
data_readers.graph_data.GraphData or nx.Graph

JGSweets · 2022-09-14T18:53:36Z

dataprofiler/profilers/graph_profiler.py

@@ -23,7 +25,6 @@ def __init__(self, data, options=None):
        Initialize Graph Profiler.

        :param data: data


I think this should be: one of these two types:
data_readers.graph_data.GraphData or nx.Graph

micdavis requested review from JGSweets, ksneab7, taylorfturner and tyfarnan as code owners September 14, 2022 17:21

taylorfturner assigned micdavis Sep 14, 2022

taylorfturner enabled auto-merge (squash) September 14, 2022 17:24

taylorfturner reviewed Sep 14, 2022

View reviewed changes

JGSweets reviewed Sep 14, 2022

View reviewed changes

taylorfturner added the High Priority Dramatic improvement, inaccurate calculation(s) or bug / feature making the library unusable label Sep 14, 2022

micdavis added 3 commits September 14, 2022 14:19

added functionality for load and save. Also added tests. Need to pass…

2b1c826

… in actual graph data to ensure load and save work correctly

updated save and load test

2860c27

pre-commit hooks

821cf3b

micdavis force-pushed the graph_profiler_save branch from 74e039a to 821cf3b Compare September 14, 2022 18:19

micdavis commented Sep 14, 2022

View reviewed changes

examples/great_expecations/ge_expect_column_values_vs_profile_min.ipynb

@@ -291,4 +291,4 @@

},

Copy link

Contributor Author

micdavis Sep 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pre-commit formatting

micdavis commented Sep 14, 2022

View reviewed changes

taylorfturner requested changes Sep 14, 2022

View reviewed changes

added in doc string

6f8727b

taylorfturner previously approved these changes Sep 14, 2022

View reviewed changes

JGSweets suggested changes Sep 14, 2022

View reviewed changes

updated doc string

55f07b2

micdavis dismissed taylorfturner’s stale review via 55f07b2 September 14, 2022 19:02

docstring comment

6c236f6

taylorfturner approved these changes Sep 14, 2022

View reviewed changes

taylorfturner merged commit 9998919 into capitalone:main Sep 14, 2022

JGSweets approved these changes Sep 14, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Graph Profiler: save() and load() Functionality #628

Graph Profiler: save() and load() Functionality #628

micdavis commented Sep 14, 2022

taylorfturner Sep 14, 2022

JGSweets Sep 14, 2022

JGSweets Sep 14, 2022

JGSweets Sep 14, 2022

taylorfturner Sep 14, 2022

micdavis Sep 14, 2022

JGSweets Sep 14, 2022

micdavis Sep 14, 2022

micdavis Sep 14, 2022

micdavis Sep 14, 2022

micdavis Sep 14, 2022

micdavis Sep 14, 2022

taylorfturner Sep 14, 2022

JGSweets Sep 14, 2022

JGSweets Sep 14, 2022

		save_profile.profile["continuous_distribution"]["weight"].pop("properties")
		load_profile.profile["continuous_distribution"]["weight"].pop("properties")

		@@ -2,17 +2,17 @@
		[Great Expectations](https://greatexpectations.io/) is an open source tool built around the idea that should always know what to expect from your data. It helps data teams eliminate pipeline debt, through data testing, documentation, and profiling.

		@@ -23,7 +25,6 @@ def __init__(self, data, options=None):
		Initialize Graph Profiler.

		:param data: data

Graph Profiler: save() and load() Functionality #628

Graph Profiler: save() and load() Functionality #628

Conversation

micdavis commented Sep 14, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment