Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update metadata #72

Merged
merged 9 commits into from
May 5, 2023
Merged

Update metadata #72

merged 9 commits into from
May 5, 2023

Conversation

NielsRogge
Copy link
Contributor

This PR simplifies the passing of metadata, by only making loading components accept an args.metadata argument that only includes the run_id of the pipeline as well as the base_path. Transform components don't need a metadata argument anymore as they just update the component_id which we can get from the component spec (FondantComponentSpec).

Fixes #69

Copy link
Contributor

@PhilippeMoussalli PhilippeMoussalli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks! I think you should still resolve some issue with lower test coverage by merging with main

fondant/component.py Outdated Show resolved Hide resolved
@NielsRogge NielsRogge marked this pull request as draft May 4, 2023 15:31
@NielsRogge
Copy link
Contributor Author

Don't merge yet as running pytest tests creates Parquet files, will use the same folder as the one on main to store them

Copy link
Member

@RobbeSneyders RobbeSneyders left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @NielsRogge!

Looks good to me, one minor comment.

If you want to test component.run(), we should mock / monkeypatch some stuff. This will probably come in handy for other tests as well. I added some steps towards this in #74

tests/example_specs/valid_component/main.py Outdated Show resolved Hide resolved
@NielsRogge NielsRogge marked this pull request as ready for review May 5, 2023 08:28
@NielsRogge NielsRogge merged commit 31b0270 into main May 5, 2023
@RobbeSneyders RobbeSneyders deleted the fix_metadata_bis branch May 15, 2023 16:29
Hakimovich99 pushed a commit that referenced this pull request Oct 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Clean up args.metadata
3 participants