Examples: Example with pandas for split apply combine

[nhuray]

Hi everyone,

This is my first Pull Request to contribute to the project 🎉

As a beginner I had issue reading the documentation about how to implement a simple split / apply / combine on a pandas DataFrame.

So after discussing it on Slack with @skrawcz and @elijahbenizzy, I decided to investigate how things works creating a simple code example.

Changes

The changes are located in examples/pandas/split-apply-combine (can be renamed).

This is the files I added:

• README.md: to explain the example
• my_functions.py: module when all the transformation functions have been implemented
• my_wrapper.py: module when I just define a class with static methods wrapping the Hamilton driver
• my_script.py: module defining the input for the example and run the wrapper to execute the dataflow

Screenshots

Here is the screenshots of the console when we run the example:

And this is the DAG generated:

How I tested this

I just run it locally in Debug to ensure the output expected is rendered and the DAG generated.

[skrawcz]

Thanks for the example. Looks good. Mostly stylistic feedback and questions :)

Oh, and can you add the image too, that way it can also be displayed in the README please :)

[skrawcz]

Otherwise I got these pandas warnings:

hamilton/examples/pandas/split-apply-combine/my_functions.py:51: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df["Tax Rate"] = output["Tax Rate"]
/hamilton/examples/pandas/split-apply-combine/my_functions.py:69: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df["Tax Credit"] = output["Tax Credit"]
/hamilton/examples/pandas/split-apply-combine/my_functions.py:51: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df["Tax Rate"] = output["Tax Rate"]

I sugget doing pd.concat([df, output], axis=1) instead

[nhuray]

Otherwise I got these pandas warnings:

hamilton/examples/pandas/split-apply-combine/my_functions.py:51: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df["Tax Rate"] = output["Tax Rate"]
/hamilton/examples/pandas/split-apply-combine/my_functions.py:69: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df["Tax Credit"] = output["Tax Credit"]
/hamilton/examples/pandas/split-apply-combine/my_functions.py:51: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df["Tax Rate"] = output["Tax Rate"]

I sugget doing pd.concat([df, output], axis=1) instead

I also got them ! It was my plan to fix it today !

[nhuray]

Otherwise I got these pandas warnings:

hamilton/examples/pandas/split-apply-combine/my_functions.py:51: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df["Tax Rate"] = output["Tax Rate"]
/hamilton/examples/pandas/split-apply-combine/my_functions.py:69: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df["Tax Credit"] = output["Tax Credit"]
/hamilton/examples/pandas/split-apply-combine/my_functions.py:51: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df["Tax Rate"] = output["Tax Rate"]

I sugget doing pd.concat([df, output], axis=1) instead

I also got them ! It was my plan to fix it today !

Fixed !

[skrawcz]

Beautiful. I think this looks good to me. CC @elijahbenizzy ?

We'll squash merge this one so will take the PR description largely.

Otherwise not sure what's up with circleci. It should have run -- only thing I'd need to check is the pre-commit check.

[skrawcz]

@nhuray okay pre-commit failed locally for me:

$ pre-commit run --all-files
black....................................................................Failed
- hook id: black
- files were modified by this hook

reformatted examples/pandas/split-apply-combine/my_wrapper.py
reformatted examples/pandas/split-apply-combine/my_functions.py

All done! ✨ 🍰 ✨
2 files reformatted, 519 files left unchanged.

trim trailing whitespace.................................................Failed
- hook id: trailing-whitespace
- exit code: 1
- files were modified by this hook

Fixing examples/pandas/split-apply-combine/README.md

fix end of files.........................................................Failed
- hook id: end-of-file-fixer
- exit code: 1
- files were modified by this hook

Fixing examples/pandas/split-apply-combine/README.md

fix requirements.txt.....................................................Passed
check python ast.........................................................Passed
isort....................................................................Failed
- hook id: isort
- files were modified by this hook

Fixing /hamilton/examples/pandas/split-apply-combine/my_functions.py
Fixing /hamilton/examples/pandas/split-apply-combine/my_script.py
Fixing /hamilton/examples/pandas/split-apply-combine/my_wrapper.py

flake8...................................................................Failed
- hook id: flake8
- exit code: 1

examples/pandas/split-apply-combine/my_functions.py:8:1: F401 'hamilton.function_modifiers.value' imported but unused

So if you run that locally it should fix/flag things to fix.

Thanks!

[nhuray]

@skrawcz I removed the my_full_dag file, add a local .gitignore and fix the lint issues !

I think the PR is ready to be merged now ! Thanks for the review and suggestions.