Skip to content
This repository has been archived by the owner on Jul 3, 2023. It is now read-only.

Create Hamilton converter for pandas code #132

Closed
skrawcz opened this issue May 13, 2022 · 3 comments
Closed

Create Hamilton converter for pandas code #132

skrawcz opened this issue May 13, 2022 · 3 comments
Assignees
Labels
enhancement New feature or request help wanted Extra attention is needed product idea

Comments

@skrawcz
Copy link
Collaborator

skrawcz commented May 13, 2022

Is your feature request related to a problem? Please describe.
With Hamilton you need need to restructure your code. This can be too much of a friction point for someone. Wouldn't it be nice if we had a way to help automate this step?

Describe the solution you'd like
We should be able to write some python code that parses the AST to covert code like:

df['a'] = df['b'] + df['c']

into

def a(b: pd.Series, c: pd.Series) -> pd.Series:
      return b + c

Core to this problem, is building code to parse python code and output/print hamilton functions. Once we have that, we can think about the places we could provide this, e.g. CLI, a website, some other means...

Describe alternatives you've considered
Not doing this.

Additional context
It would enable people to get up and running with Hamilton faster. E.g. if they provided a script, and we "walked" the script and guessed what should be output...

@skrawcz skrawcz added enhancement New feature or request help wanted Extra attention is needed product idea labels May 13, 2022
@elijahbenizzy
Copy link
Collaborator

Interesting... Another approach is to use some version of pandas that traces the actions and delays it (I bet koalas does something like this, or ibis), then use the intermediate data representation to compile to hamilton.

@elijahbenizzy
Copy link
Collaborator

Started working on this -- got a rough POC working but its fairly complicated and wasn't worth my time.

What I'd love is doing this on the "intro to hamilton" web-page -- live conversion with an example we can get to. Will take another stab with my old prototype.

@elijahbenizzy
Copy link
Collaborator

We are moving repositories! Please see the new version of this issue at DAGWorks-Inc/hamilton#28. Also, please give us a star/update any of your internal links.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request help wanted Extra attention is needed product idea
Projects
None yet
Development

No branches or pull requests

2 participants