Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[good first issue - beginner] Add Hamilton example using Vaex #457

Closed
skrawcz opened this issue Oct 12, 2023 · 4 comments · Fixed by #708
Closed

[good first issue - beginner] Add Hamilton example using Vaex #457

skrawcz opened this issue Oct 12, 2023 · 4 comments · Fixed by #708
Assignees
@tyapochkin
Labels
good first issue Good for newcomers hacktoberfest Hacktoberfest issues help wanted Extra attention is needed

Comments

@skrawcz
Copy link
Collaborator

skrawcz commented Oct 12, 2023 •
edited

Is your feature request related to a problem? Please describe.
Vaex is a dataframe libary. We should have an example showing how you can use Hamilton with it.

Describe the solution you'd like
We should have an analogous hello world to the pandas example, but with Vaex.

This will mean we'll need:

  • an example mirroring the Hamilton pandas hello world [beginner]
  • add a vaex extension mirroring the ones for pandas and polars -- the materializers are not needed but the registry of types is. [beginner]
  • add a h_vaex to contain a result builder for vaex similar to the one for polars. [beginner]

Describe alternatives you've considered
N/A

Additional context
@skrawcz skrawcz added good first issue Good for newcomers help wanted Extra attention is needed hacktoberfest Hacktoberfest issues labels Oct 12, 2023
@skrawcz skrawcz changed the title [good first issue - beginner/intermediate] Add Hamilton example using Vaex [good first issue - beginner] Add Hamilton example using Vaex Oct 12, 2023
@tyapochkin
Copy link
Contributor

@skrawcz assign this issue to me, please

@tyapochkin
Copy link
Contributor

Just to note: Vaex still doesn't support python 3.11 and 3.12: vaexio/vaex#2331. But it looks like it's under development right now.

@skrawcz
Copy link
Collaborator Author

skrawcz commented Feb 15, 2024

@tyapochkin cool -- that's fine.

Does it look like Vaex is compatible with Hamilton? I just kind of assumed.

@tyapochkin
Copy link
Contributor

I've finally put together a working example with Hamilton and Vaex, so I can confirm that they are compatible. Well, at least in basic functionality and in-memory executions.
I'll finalize the code and tests and submit a PR shortly.

tyapochkin added a commit to tyapochkin/hamilton that referenced this issue Feb 21, 2024
@tyapochkin
Adds plugin, example and tests for Vaex (DAGWorks-Inc#457)
c10903c
@tyapochkin tyapochkin mentioned this issue Feb 21, 2024
Merged
7 tasks
skrawcz pushed a commit that referenced this issue Feb 24, 2024
@tyapochkin
Adds plugin, example and tests for Vaex (#457) (#708)
e2ea1d8 
This commit provides support for decorators usage with vaex,
as well as a result builder to stitch everything back. 

Squashed commits:

* Adds plugin, example and tests for Vaex (#457)

* Small fix in description of VaexDataFrameResult class

* Adds minor fixes for Vaex

Adding to the registry so we try to load the extension if vaex is in the environment.
Then fixing a few minor typos otherwise to get things to work.

---------

Co-authored-by: Konstantin Tyapochkin <tyapochkin.ks@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tyapochkin tyapochkin

Labels
good first issue Good for newcomers hacktoberfest Hacktoberfest issues help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Adds plugin, example and tests for Vaex (#457) tyapochkin/hamilton
2 participants
@skrawcz @tyapochkin