Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide unpacked keyword arguments to component.transform method #74

Merged
merged 8 commits into from
May 5, 2023

Conversation

RobbeSneyders
Copy link
Member

@RobbeSneyders RobbeSneyders commented May 4, 2023

Fixes #67

@@ -1,3 +1,3 @@
git+https://github.com/ml6team/fondant.git@d697be951fa33faa0baa3625cf3cb04d6800834a
git+https://github.com/ml6team/fondant.git
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we squash merge, any commit hash part of a PR will actually not become part of main, so it's not ideal to use them as a reference.

I think we should always keep these examples in line with main, so I would propose to unfreeze the fondant version in their requirements.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think those are mainly used for testing the functionality of your branch code. Right now the workflow is a bit annoying to do so: you have to push your code and then reference the hash so that the component can rebuild with the latest version. Wondering if there isn't a better way to do. @NielsRogge any thoughts?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once we 'actually open source', the examples on main should always work though. For components to share, the version can still be fixed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes for now we probably need to include the commit hash when trying out new functionality, but when merging a PR it should always be updated to main.

@RobbeSneyders RobbeSneyders mentioned this pull request May 4, 2023
Copy link
Contributor

@NielsRogge NielsRogge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for improving!

@RobbeSneyders RobbeSneyders merged commit 1ac0fa0 into main May 5, 2023
@NielsRogge
Copy link
Contributor

NielsRogge commented May 5, 2023

Note that ideally also the simple pipeline gets updated and tested:

def transform(self, dataframe: dd.DataFrame, args: Dict) -> dd.DataFrame:

Cause we had updates to the Fondant code base in the past which turned out to need fixes to run the pipeline successfully on KubeFlow

RobbeSneyders added a commit that referenced this pull request May 5, 2023
Small follow-up to #74 to update the simple pipeline components.
@RobbeSneyders RobbeSneyders deleted the feature/transform-interface branch May 15, 2023 16:29
Hakimovich99 pushed a commit that referenced this pull request Oct 16, 2023
Small follow-up to #74 to update the simple pipeline components.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Redesign component.transform interface
3 participants