Return JSON from Pipeline DAG structure #2812

ParthivNaresh · 2021-09-20T16:32:11Z

Fixes #1969

This allows users to return a serialized JSON representation of a pipeline's DAG structure for their own graphing purposes.
The template for the structure is

{"Edges": {"from_parent_node_to_child_node" : {"data": what_value_is_passed,
                                               "from": "parent_node",
                                               "to": "child_node"}
                  ...},
 "Nodes": {"component_name": {"class": class_name,
                              "attributes": self.parameters_attributes}
                  ...}
}

dag_json = json.loads(pipeline_.graph_json())
----------------------------------------------
{'Edges': {'from_Imputer_to_Standard Scaler': {'data': 'X modified by Imputer',
                                               'from': 'Imputer',
                                               'to': 'Standard Scaler'},
           'from_One_Hot_Encoder_Special_to_Imputer': {'data': 'X modified by One Hot '
                                                       'Encoder',
                                               'from': 'One Hot Encoder',
                                               'to': 'Imputer'},
           'from_Standard Scaler_to_Linear Regressor': {'data': 'X modified by '
                                                                'Standard '
                                                                'Scaler',
                                                        'from': 'Standard '
                                                                'Scaler',
                                                        'to': 'Linear '
                                                              'Regressor'},
           'from_X_to_One_Hot_Encoder_Special': {'data': 'Original X input',
                                         'from': 'X',
                                         'to': 'One Hot Encoder'},
           'from_y_to_Imputer': {'data': 'Original y input',
                                 'from': 'y',
                                 'to': 'Imputer'},
           'from_y_to_Linear Regressor': {'data': 'Original y input',
                                          'from': 'y',
                                          'to': 'Linear Regressor'},
           'from_y_to_One_Hot_Encoder_Special': {'data': 'Original y input',
                                         'from': 'y',
                                         'to': 'One Hot Encoder'},
           'from_y_to_Standard Scaler': {'data': 'Original y input',
                                         'from': 'y',
                                         'to': 'Standard Scaler'}},
 'Nodes': {'Imputer': {'attributes': {'categorical_fill_value': None,
                                      'categorical_impute_strategy': 'most_frequent',
                                      'numeric_fill_value': None,
                                      'numeric_impute_strategy': 'mean'},
                       'class': 'Imputer'},
           'Linear Regressor': {'attributes': {'fit_intercept': True,
                                               'n_jobs': -1,
                                               'normalize': False},
                                'class': 'Linear Regressor'},
           'One_Hot_Encoder_Special': {'attributes': {'categories': None,
                                              'drop': 'if_binary',
                                              'features_to_encode': None,
                                              'handle_missing': 'error',
                                              'handle_unknown': 'ignore',
                                              'top_n': 10},
                               'class': 'One Hot Encoder'},
           'Standard Scaler': {'attributes': None, 'class': 'Standard Scaler'},
           'X': 'X',
           'y': 'y'}}

# Conflicts: # docs/source/release_notes.rst

codecov · 2021-09-20T16:43:16Z

Codecov Report

Merging #2812 (61846a6) into main (e4dabef) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@           Coverage Diff           @@
##            main   #2812     +/-   ##
=======================================
+ Coverage   99.7%   99.7%   +0.1%     
=======================================
  Files        302     302             
  Lines      28312   28388     +76     
=======================================
+ Hits       28216   28292     +76     
  Misses        96      96

Impacted Files	Coverage Δ
evalml/automl/automl_search.py	`99.9% <ø> (ø)`
evalml/pipelines/pipeline_base.py	`98.4% <100.0%> (+0.2%)`	⬆️
evalml/tests/automl_tests/test_automl.py	`99.5% <100.0%> (+0.1%)`	⬆️
evalml/tests/conftest.py	`98.3% <100.0%> (+0.1%)`	⬆️
evalml/tests/pipeline_tests/test_graphs.py	`100.0% <100.0%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e4dabef...61846a6. Read the comment docs.

# Conflicts: # docs/source/release_notes.rst

…SON-Graph

ParthivNaresh · 2021-09-21T13:14:30Z

evalml/automl/automl_search.py

@@ -314,7 +314,7 @@ class AutoMLSearch:
            Only applicable if patience is not None. Defaults to None.

        allowed_component_graphs (dict): A dictionary of lists or ComponentGraphs indicating the component graphs allowed in the search.
-            The format should follow { "Name_0": [list_of_components], "Name_1": [ComponentGraph(...)] }
+            The format should follow { "Name_0": [list_of_components], "Name_1": ComponentGraph(...) }


I don't think we meant to add a list here when passing ComponentGraphs

ParthivNaresh · 2021-09-21T13:15:58Z

evalml/pipelines/pipeline_base.py

+        Returns:
+            dag_json (str): A serialized JSON representation of a DAG structure.
+        """
+        dag_json = {"Nodes": {"X": "X", "y": "y"}, "Edges": {}}


Added X and y as default nodes

ParthivNaresh · 2021-09-21T13:16:30Z

evalml/pipelines/pipeline_base.py

+                else component_info[0].name,
+                "attributes": self.parameters[component_name]
+                if component_name in self.parameters
+                else None,


Some classes don't have any parameters that get passed through like StandardScaler

ParthivNaresh · 2021-09-21T13:17:45Z

evalml/pipelines/pipeline_base.py

+                    dag_json["Edges"][f"from_{from_comp}_to_{component_name}"] = {
+                        "from": from_comp,
+                        "to": component_name,
+                        "data": f"X modified by {from_comp}",


data is meant to provide a little more detail as to what's being passed through that edge even though it can be inferred based on the from and to fields. OS users might find this useful in understanding this output.

ParthivNaresh · 2021-09-21T13:19:29Z

evalml/tests/automl_tests/test_automl.py

+    dag_str = automl.allowed_pipelines[0].graph_json()
+    dag_json = json.loads(dag_str)
+    for node_, params_ in automl_parameters_.items():
+        for key_, val_ in params_.items():


The specific structure is tested in test_graphs, this is primarily to check the self.parameters as they get passed through the pipeline

ParthivNaresh · 2021-09-21T13:20:16Z

evalml/tests/conftest.py

+    component_graph = {
+        "Imputer": ["Imputer", "X", "y"],
+        "Target Imputer": ["Target Imputer", "X", "y"],
+        "OneHot_RandomForest": ["One Hot Encoder", "Imputer.x", "Target Imputer.y"],


Wanted a text fixture that provided a graph with a modification of y

eccabay

This is super interesting! I'm a little worried about the structure of the nodes, where the format of the X and y nodes don't match the rest of the dictionary. I can't really think of a better way to represent the data, but I think users might be a little confused by it. Because of this, I'd argue we need to make sure the JSON structure is in the documentation, and is easy to both find and read.

docs/source/release_notes.rst

evalml/pipelines/pipeline_base.py

chukarsten

Nice work! That test_component_as_json test is mighty thorough. I think we might want to move towards removing that intermediary key from the json, though, as it doesn't really add too much and might actually clutter up the json a little bit. I think also that we owe you a much better specification for the JSON next time. I'll add this to our sprint feedback.

# Conflicts: # evalml/tests/automl_tests/test_automl.py

…SON-Graph

bchen1116

Left some comments on potential improvements that could be added! Let me know what you think. Looking good though!

bchen1116 · 2021-10-08T16:43:40Z

evalml/pipelines/pipeline_base.py

@@ -425,6 +426,58 @@ def feature_importance(self):
        df = pd.DataFrame(importance, columns=["feature", "importance"])
        return df

+    def graph_json(self):


Nit: Can we make the output formatted? Would make it a lot easier to read

bchen1116 · 2021-10-08T16:50:32Z

evalml/pipelines/pipeline_base.py

@@ -425,6 +426,58 @@ def feature_importance(self):
        df = pd.DataFrame(importance, columns=["feature", "importance"])
        return df

+    def graph_json(self):


Another nit:

In "x_edges", can the node that takes in X (the "DRop Columns Transformer") come first in the list?

evalml/pipelines/pipeline_base.py

bchen1116 · 2021-10-08T18:45:03Z

evalml/pipelines/pipeline_base.py

+                    x_edges.append(("X", component_name))
+                elif parent == "y":
+                    y_edges.append(("y", component_name))
+        nodes.update({"X": "X", "y": "y"})


Do we need this? Doesn't seem like very useful information to the user, since it's not going to change between different pipelines, right?

It's not necessary, but since X_edges and y_edges can both include X and y, I'd expect a user looking to implement this in a custom visualization tool to want any node mentioned in the edges to also be mentioned in nodes.

bchen1116 · 2021-10-08T18:55:12Z

evalml/tests/automl_tests/test_automl.py

@@ -5214,3 +5215,44 @@ def test_automl_chooses_engine(engine_choice, X_y_binary):
            automl = AutoMLSearch(
                X_train=X, y_train=y, problem_type="binary", engine=engine_choice
            )
+
+
+def test_graph_automl(X_y_multi):


I know that this isn't likely using much time or memory but can we mock this test?

evalml/tests/pipeline_tests/test_graphs.py

chukarsten

Looks good Parthiv!

ParthivNaresh added 2 commits September 20, 2021 12:30

initial commit

434adc3

Merge branch 'main' into JSON-Graph

0f728bf

# Conflicts: # docs/source/release_notes.rst

ParthivNaresh added 9 commits September 20, 2021 12:51

lint fixes and release notes

5110fb4

add target imputer for example graph

8bd19bf

lint fix

d0a0a68

Merge branch 'main' into JSON-Graph

5d83e17

fix test case

2b1a5e7

Merge branch 'main' into JSON-Graph

98ebc06

# Conflicts: # docs/source/release_notes.rst

Merge branch 'JSON-Graph' of https://github.com/alteryx/evalml into J…

7a62a98

…SON-Graph

remove uncovered conftest

82a2daf

Merge branch 'main' into JSON-Graph

1d0bf8a

ParthivNaresh commented Sep 21, 2021

View reviewed changes

ParthivNaresh marked this pull request as ready for review September 21, 2021 13:20

auto-assign bot assigned ParthivNaresh Sep 21, 2021

ParthivNaresh requested review from chukarsten, freddyaboulton, angela97lin, christopherbunn and eccabay and removed request for chukarsten September 21, 2021 13:20

eccabay reviewed Sep 21, 2021

View reviewed changes

docs/source/release_notes.rst Outdated Show resolved Hide resolved

evalml/pipelines/pipeline_base.py Outdated Show resolved Hide resolved

evalml/pipelines/pipeline_base.py Outdated Show resolved Hide resolved

ParthivNaresh added 2 commits September 21, 2021 12:18

clean up code

9ba6983

lint fix

f780427

chukarsten reviewed Sep 23, 2021

View reviewed changes

ParthivNaresh added 9 commits September 23, 2021 12:50

Merge branch 'main' into JSON-Graph

b758518

change format for json and tests

4eeefcc

Merge branch 'main' into JSON-Graph

0c1d362

# Conflicts: # evalml/tests/automl_tests/test_automl.py

lint

c1aabe9

Merge branch 'main' into JSON-Graph

91e352e

Merge branch 'JSON-Graph' of https://github.com/alteryx/evalml into J…

178dd96

…SON-Graph

docstring update

ae62d79

sphinx fixes

3ba9d8b

lint fix

11f8f7c

ParthivNaresh requested review from angela97lin, eccabay, dsherry and bchen1116 October 8, 2021 15:59

bchen1116 reviewed Oct 8, 2021

View reviewed changes

ParthivNaresh added 12 commits October 11, 2021 18:03

Merge branch 'main' into JSON-Graph

d0127d6

remove blank line in docstring

648a215

remove space between description and returns

5aaa0ba

remove indent

0e0647e

moving paranthesis

4d5fe69

no message

c916ce1

no message

fc28e21

no message

9f6b583

no message

51693b2

no message

a2db5cd

no message

8a4c8dc

no message

61846a6

chukarsten approved these changes Oct 12, 2021

View reviewed changes

ParthivNaresh merged commit 2c6bbe3 into main Oct 12, 2021

freddyaboulton deleted the JSON-Graph branch May 13, 2022 15:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Return JSON from Pipeline DAG structure #2812

Return JSON from Pipeline DAG structure #2812

ParthivNaresh commented Sep 20, 2021 •

edited

Loading

codecov bot commented Sep 20, 2021 •

edited

Loading

ParthivNaresh Sep 21, 2021

ParthivNaresh Sep 21, 2021

ParthivNaresh Sep 21, 2021

ParthivNaresh Sep 21, 2021

ParthivNaresh Sep 21, 2021

ParthivNaresh Sep 21, 2021

eccabay left a comment

chukarsten left a comment

bchen1116 left a comment

bchen1116 Oct 8, 2021

bchen1116 Oct 8, 2021

bchen1116 Oct 8, 2021

ParthivNaresh Oct 11, 2021

bchen1116 Oct 8, 2021

chukarsten left a comment

Return JSON from Pipeline DAG structure #2812

Return JSON from Pipeline DAG structure #2812

Conversation

ParthivNaresh commented Sep 20, 2021 • edited Loading

codecov bot commented Sep 20, 2021 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eccabay left a comment

Choose a reason for hiding this comment

chukarsten left a comment

Choose a reason for hiding this comment

bchen1116 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chukarsten left a comment

Choose a reason for hiding this comment

ParthivNaresh commented Sep 20, 2021 •

edited

Loading

codecov bot commented Sep 20, 2021 •

edited

Loading