Parameterized extract #259

elijahbenizzy · 2022-12-27T03:52:29Z

[Short description explaining the high-level reason for the pull request]

Changes

How I tested this

Notes

Checklist

PR has an informative and human-readable title (this will be pulled into the release notes)
Changes are limited to a single goal (no scope creep)
Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
Any change in functionality is tested
New functions are documented (with a description, list of inputs, and expected output)
Placeholder code is flagged / future TODOs are captured in comments
Project documentation has been updated if adding/changing functionality.

This provides a convenience API for #196. The idea is that people want to do a parameterization and extract columns operator at once -- this should be easy. The cool thing is that this just uses the parameterize and extract APIs. It also has a from_df() function to allow for passing in a dataframe to be more concise.

This adds testing for the decorator, and changes it to a simpler naming scheme. Currently the naming scheme appends __{i} to the node name. This is a reserved pattern that won't be used later.

elijahbenizzy · 2023-01-29T22:39:26Z

OK, this is ready. Calling @parameterized_frame experimental till we get a question on it -- the API is a bit clunky but I haven't ifgured out how to handle it perfectly yet...

skrawcz · 2023-01-30T01:24:36Z

@elijahbenizzy as discussed, I think making this live under experimental would be better short term.

Not quite production-ready yet...

skrawcz

seems reasonable -- just a few minor things. Fix them and I approve.

hamilton/experimental/decorators/parameterize_frame.py

skrawcz · 2023-01-30T05:57:06Z

decorators.md

+        )
+```
+
+Note that we have a double-index. Note that this is still in experimental.


Suggested change

Note that we have a double-index. Note that this is still in experimental.

Note that we have a double-index. Note that this is still experimental, and has the possibility of being changed; we'd love feedback on this API if you end up using it!

skrawcz · 2023-01-30T05:57:57Z

decorators.md

+ df = pd.DataFrame(
+        [
+            ["outseries1a", "outseries2a", "inseries1a", "inseries2a", 10],
+            ["outseries1b", "outseries2b", "inseries1b", "inseries2b", 100],
+            # ...
+        ],
+        # Have to switch as indices have to be unique
+        columns=[
+            [
+                "output1",
+                "output2",
+                "input1",
+                "input2",
+                "input3",
+            ],  # configure whether column is source or value and also whether it's input ("source", "value") or output ("out")
+            ["out", "out", "source", "source", "value"],
+        ],
+    )
+
+    @parameterize_frame(df)
+    def my_func(input1: pd.Series, input2: pd.Series, input3: float) -> pd.DataFrame:
+        return pd.DataFrame(
+            [input1 * input2 * input3, input1 + input2 + input3]
+        )


indentation off

Suggested change

df = pd.DataFrame(

[

["outseries1a", "outseries2a", "inseries1a", "inseries2a", 10],

["outseries1b", "outseries2b", "inseries1b", "inseries2b", 100],

# ...

],

# Have to switch as indices have to be unique

columns=[

[

"output1",

"output2",

"input1",

"input2",

"input3",

], # configure whether column is source or value and also whether it's input ("source", "value") or output ("out")

["out", "out", "source", "source", "value"],

],

)

@parameterize_frame(df)

def my_func(input1: pd.Series, input2: pd.Series, input3: float) -> pd.DataFrame:

return pd.DataFrame(

[input1 * input2 * input3, input1 + input2 + input3]

)

df = pd.DataFrame(

[

["outseries1a", "outseries2a", "inseries1a", "inseries2a", 10],

["outseries1b", "outseries2b", "inseries1b", "inseries2b", 100],

# ...

],

# Have to switch as indices have to be unique

columns=[

[

"output1",

"output2",

"input1",

"input2",

"input3",

], # configure whether column is source or value and also whether it's input ("source", "value") or output ("out")

["out", "out", "source", "source", "value"],

],

)

@parameterize_frame(df)

def my_func(input1: pd.Series, input2: pd.Series, input3: float) -> pd.DataFrame:

return pd.DataFrame(

[input1 * input2 * input3, input1 + input2 + input3]

)

skrawcz · 2023-01-30T05:58:31Z

hamilton/experimental/decorators/parameterize_frame.py

+    raise ValueError(f"Invalid dep type: {dep_type}")
+
+
+def _get_index_levels(index: pd.MultiIndex) -> List[list]:


doc string please

skrawcz · 2023-01-30T05:59:15Z

hamilton/experimental/decorators/parameterize_frame.py

+from hamilton.function_modifiers.expanders import ParameterizedExtract
+
+
+def _get_dep_type(dep_type: str):


short doc string -- was not clear that this function returns the right function modifier type.

or return type annotation.

elijahbenizzy added 4 commits January 29, 2023 12:43

Adds decorators to public exposition (functions_modifier.__init__.py)

b1964e6

Fixes typo coming from parameter name

9bbcebd

elijahbenizzy force-pushed the parameterized_extract branch from 91b12f2 to 9bbcebd Compare January 29, 2023 20:43

Moves parameterized_frame to extensions

e0d8596

elijahbenizzy force-pushed the parameterized_extract branch 2 times, most recently from 757aa05 to 4ad63e7 Compare January 29, 2023 22:06

elijahbenizzy changed the title ~~Parameterized extract - WIP, needs more testing~~ Parameterized extract Jan 29, 2023

Gets paraemterized_extract_columns ready for production

b3881f4

This adds testing for the decorator, and changes it to a simpler naming scheme. Currently the naming scheme appends __{i} to the node name. This is a reserved pattern that won't be used later.

elijahbenizzy force-pushed the parameterized_extract branch from 4ad63e7 to b3881f4 Compare January 29, 2023 22:35

elijahbenizzy requested a review from skrawcz January 29, 2023 22:38

Moves parameterize_frame to experimental

5c49311

Not quite production-ready yet...

elijahbenizzy force-pushed the parameterized_extract branch from 46fd862 to 5c49311 Compare January 30, 2023 02:36

skrawcz approved these changes Jan 30, 2023

View reviewed changes

Minor cleanup for parameterize_frame

3c65330

elijahbenizzy merged commit 8913643 into main Jan 30, 2023

elijahbenizzy deleted the parameterized_extract branch January 30, 2023 17:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parameterized extract #259

Parameterized extract #259

elijahbenizzy commented Dec 27, 2022

elijahbenizzy commented Jan 29, 2023

skrawcz commented Jan 30, 2023

skrawcz left a comment

skrawcz Jan 30, 2023

skrawcz Jan 30, 2023

skrawcz Jan 30, 2023

skrawcz Jan 30, 2023

skrawcz Jan 30, 2023

	Note that we have a double-index. Note that this is still in experimental.
	Note that we have a double-index. Note that this is still experimental, and has the possibility of being changed; we'd love feedback on this API if you end up using it!

		raise ValueError(f"Invalid dep type: {dep_type}")


		def _get_index_levels(index: pd.MultiIndex) -> List[list]:

		from hamilton.function_modifiers.expanders import ParameterizedExtract


		def _get_dep_type(dep_type: str):

Parameterized extract #259

Parameterized extract #259

Conversation

elijahbenizzy commented Dec 27, 2022

Changes

How I tested this

Notes

Checklist

elijahbenizzy commented Jan 29, 2023

skrawcz commented Jan 30, 2023

skrawcz left a comment

Choose a reason for hiding this comment

skrawcz Jan 30, 2023

Choose a reason for hiding this comment

skrawcz Jan 30, 2023

Choose a reason for hiding this comment

skrawcz Jan 30, 2023

Choose a reason for hiding this comment

skrawcz Jan 30, 2023

Choose a reason for hiding this comment

skrawcz Jan 30, 2023

Choose a reason for hiding this comment