Return JSON from Pipeline DAG structure by ParthivNaresh · Pull Request #2812 · alteryx/evalml

ParthivNaresh · 2021-09-20T16:32:11Z

Fixes #1969

This allows users to return a serialized JSON representation of a pipeline's DAG structure for their own graphing purposes.
The template for the structure is

{"Edges": {"from_parent_node_to_child_node" : {"data": what_value_is_passed,
                                               "from": "parent_node",
                                               "to": "child_node"}
                  ...},
 "Nodes": {"component_name": {"class": class_name,
                              "attributes": self.parameters_attributes}
                  ...}
}

dag_json = json.loads(pipeline_.graph_json())
----------------------------------------------
{'Edges': {'from_Imputer_to_Standard Scaler': {'data': 'X modified by Imputer',
                                               'from': 'Imputer',
                                               'to': 'Standard Scaler'},
           'from_One_Hot_Encoder_Special_to_Imputer': {'data': 'X modified by One Hot '
                                                       'Encoder',
                                               'from': 'One Hot Encoder',
                                               'to': 'Imputer'},
           'from_Standard Scaler_to_Linear Regressor': {'data': 'X modified by '
                                                                'Standard '
                                                                'Scaler',
                                                        'from': 'Standard '
                                                                'Scaler',
                                                        'to': 'Linear '
                                                              'Regressor'},
           'from_X_to_One_Hot_Encoder_Special': {'data': 'Original X input',
                                         'from': 'X',
                                         'to': 'One Hot Encoder'},
           'from_y_to_Imputer': {'data': 'Original y input',
                                 'from': 'y',
                                 'to': 'Imputer'},
           'from_y_to_Linear Regressor': {'data': 'Original y input',
                                          'from': 'y',
                                          'to': 'Linear Regressor'},
           'from_y_to_One_Hot_Encoder_Special': {'data': 'Original y input',
                                         'from': 'y',
                                         'to': 'One Hot Encoder'},
           'from_y_to_Standard Scaler': {'data': 'Original y input',
                                         'from': 'y',
                                         'to': 'Standard Scaler'}},
 'Nodes': {'Imputer': {'attributes': {'categorical_fill_value': None,
                                      'categorical_impute_strategy': 'most_frequent',
                                      'numeric_fill_value': None,
                                      'numeric_impute_strategy': 'mean'},
                       'class': 'Imputer'},
           'Linear Regressor': {'attributes': {'fit_intercept': True,
                                               'n_jobs': -1,
                                               'normalize': False},
                                'class': 'Linear Regressor'},
           'One_Hot_Encoder_Special': {'attributes': {'categories': None,
                                              'drop': 'if_binary',
                                              'features_to_encode': None,
                                              'handle_missing': 'error',
                                              'handle_unknown': 'ignore',
                                              'top_n': 10},
                               'class': 'One Hot Encoder'},
           'Standard Scaler': {'attributes': None, 'class': 'Standard Scaler'},
           'X': 'X',
           'y': 'y'}}

# Conflicts: # docs/source/release_notes.rst

codecov · 2021-09-20T16:43:16Z

Codecov Report

Merging #2812 (61846a6) into main (e4dabef) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@           Coverage Diff           @@
##            main   #2812     +/-   ##
=======================================
+ Coverage   99.7%   99.7%   +0.1%     
=======================================
  Files        302     302             
  Lines      28312   28388     +76     
=======================================
+ Hits       28216   28292     +76     
  Misses        96      96

Impacted Files	Coverage Δ
evalml/automl/automl_search.py	`99.9% <ø> (ø)`
evalml/pipelines/pipeline_base.py	`98.4% <100.0%> (+0.2%)`	⬆️
evalml/tests/automl_tests/test_automl.py	`99.5% <100.0%> (+0.1%)`	⬆️
evalml/tests/conftest.py	`98.3% <100.0%> (+0.1%)`	⬆️
evalml/tests/pipeline_tests/test_graphs.py	`100.0% <100.0%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e4dabef...61846a6. Read the comment docs.

# Conflicts: # docs/source/release_notes.rst

…SON-Graph

ParthivNaresh · 2021-09-21T13:14:30Z


        allowed_component_graphs (dict): A dictionary of lists or ComponentGraphs indicating the component graphs allowed in the search.
-            The format should follow { "Name_0": [list_of_components], "Name_1": [ComponentGraph(...)] }
+            The format should follow { "Name_0": [list_of_components], "Name_1": ComponentGraph(...) }


I don't think we meant to add a list here when passing ComponentGraphs

ParthivNaresh · 2021-09-21T13:15:58Z

+        Returns:
+            dag_json (str): A serialized JSON representation of a DAG structure.
+        """
+        dag_json = {"Nodes": {"X": "X", "y": "y"}, "Edges": {}}


Added X and y as default nodes

ParthivNaresh · 2021-09-21T13:16:30Z

+                else component_info[0].name,
+                "attributes": self.parameters[component_name]
+                if component_name in self.parameters
+                else None,


Some classes don't have any parameters that get passed through like StandardScaler

ParthivNaresh · 2021-09-21T13:17:45Z

+                    dag_json["Edges"][f"from_{from_comp}_to_{component_name}"] = {
+                        "from": from_comp,
+                        "to": component_name,
+                        "data": f"X modified by {from_comp}",


data is meant to provide a little more detail as to what's being passed through that edge even though it can be inferred based on the from and to fields. OS users might find this useful in understanding this output.

ParthivNaresh · 2021-09-21T13:19:29Z

+    dag_str = automl.allowed_pipelines[0].graph_json()
+    dag_json = json.loads(dag_str)
+    for node_, params_ in automl_parameters_.items():
+        for key_, val_ in params_.items():


The specific structure is tested in test_graphs, this is primarily to check the self.parameters as they get passed through the pipeline

ParthivNaresh · 2021-09-21T13:20:16Z

+    component_graph = {
+        "Imputer": ["Imputer", "X", "y"],
+        "Target Imputer": ["Target Imputer", "X", "y"],
+        "OneHot_RandomForest": ["One Hot Encoder", "Imputer.x", "Target Imputer.y"],


Wanted a text fixture that provided a graph with a modification of y

eccabay

This is super interesting! I'm a little worried about the structure of the nodes, where the format of the X and y nodes don't match the rest of the dictionary. I can't really think of a better way to represent the data, but I think users might be a little confused by it. Because of this, I'd argue we need to make sure the JSON structure is in the documentation, and is easy to both find and read.

chukarsten

Nice work! That test_component_as_json test is mighty thorough. I think we might want to move towards removing that intermediary key from the json, though, as it doesn't really add too much and might actually clutter up the json a little bit. I think also that we owe you a much better specification for the JSON next time. I'll add this to our sprint feedback.

# Conflicts: # evalml/tests/automl_tests/test_automl.py

…SON-Graph

bchen1116

Left some comments on potential improvements that could be added! Let me know what you think. Looking good though!

bchen1116 · 2021-10-08T16:43:40Z

        df = pd.DataFrame(importance, columns=["feature", "importance"])
        return df

+    def graph_json(self):


Nit: Can we make the output formatted? Would make it a lot easier to read

bchen1116 · 2021-10-08T16:50:32Z

        df = pd.DataFrame(importance, columns=["feature", "importance"])
        return df

+    def graph_json(self):


Another nit:

In "x_edges", can the node that takes in X (the "DRop Columns Transformer") come first in the list?

bchen1116 · 2021-10-08T18:45:03Z

+                    x_edges.append(("X", component_name))
+                elif parent == "y":
+                    y_edges.append(("y", component_name))
+        nodes.update({"X": "X", "y": "y"})


Do we need this? Doesn't seem like very useful information to the user, since it's not going to change between different pipelines, right?

It's not necessary, but since X_edges and y_edges can both include X and y, I'd expect a user looking to implement this in a custom visualization tool to want any node mentioned in the edges to also be mentioned in nodes.

bchen1116 · 2021-10-08T18:55:12Z

            )
+
+
+def test_graph_automl(X_y_multi):


I know that this isn't likely using much time or memory but can we mock this test?

chukarsten

Looks good Parthiv!

ParthivNaresh added 2 commits September 20, 2021 12:30

initial commit

434adc3

Merge branch 'main' into JSON-Graph

0f728bf

# Conflicts: # docs/source/release_notes.rst

ParthivNaresh added 9 commits September 20, 2021 12:51

lint fixes and release notes

5110fb4

add target imputer for example graph

8bd19bf

lint fix

d0a0a68

Merge branch 'main' into JSON-Graph

5d83e17

fix test case

2b1a5e7

Merge branch 'main' into JSON-Graph

98ebc06

# Conflicts: # docs/source/release_notes.rst

Merge branch 'JSON-Graph' of https://github.com/alteryx/evalml into J…

7a62a98

…SON-Graph

remove uncovered conftest

82a2daf

Merge branch 'main' into JSON-Graph

1d0bf8a

ParthivNaresh commented Sep 21, 2021

View reviewed changes

ParthivNaresh marked this pull request as ready for review September 21, 2021 13:20

auto-assign Bot assigned ParthivNaresh Sep 21, 2021

ParthivNaresh requested review from angela97lin, christopherbunn, chukarsten, eccabay and freddyaboulton and removed request for chukarsten September 21, 2021 13:20

eccabay reviewed Sep 21, 2021

View reviewed changes

Comment thread docs/source/release_notes.rst Outdated

Comment thread evalml/pipelines/pipeline_base.py

Comment thread evalml/pipelines/pipeline_base.py Outdated

ParthivNaresh added 2 commits September 21, 2021 12:18

clean up code

9ba6983

lint fix

f780427

chukarsten reviewed Sep 23, 2021

View reviewed changes

ParthivNaresh added 9 commits September 23, 2021 12:50

Merge branch 'main' into JSON-Graph

b758518

change format for json and tests

4eeefcc

Merge branch 'main' into JSON-Graph

0c1d362

# Conflicts: # evalml/tests/automl_tests/test_automl.py

lint

c1aabe9

Merge branch 'main' into JSON-Graph

91e352e

Merge branch 'JSON-Graph' of https://github.com/alteryx/evalml into J…

178dd96

…SON-Graph

docstring update

ae62d79

sphinx fixes

3ba9d8b

lint fix

11f8f7c

ParthivNaresh requested review from angela97lin, bchen1116, dsherry and eccabay October 8, 2021 15:59

bchen1116 reviewed Oct 8, 2021

View reviewed changes

ParthivNaresh added 12 commits October 11, 2021 18:03

Merge branch 'main' into JSON-Graph

d0127d6

remove blank line in docstring

648a215

remove space between description and returns

5aaa0ba

remove indent

0e0647e

moving paranthesis

4d5fe69

no message

c916ce1

no message

fc28e21

no message

9f6b583

no message

51693b2

no message

a2db5cd

no message

8a4c8dc

no message

61846a6

chukarsten approved these changes Oct 12, 2021

View reviewed changes

ParthivNaresh merged commit 2c6bbe3 into main Oct 12, 2021

freddyaboulton deleted the JSON-Graph branch May 13, 2022 15:03

Uh oh!

Conversation

ParthivNaresh commented Sep 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Sep 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eccabay left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chukarsten left a comment

Choose a reason for hiding this comment

Uh oh!

bchen1116 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chukarsten left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

ParthivNaresh commented Sep 20, 2021 •

edited

Loading

codecov Bot commented Sep 20, 2021 •

edited

Loading