Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FeatureStore] Support SQL DBs as source and online target #2869

Merged
merged 18 commits into from Jan 19, 2023
Merged

[FeatureStore] Support SQL DBs as source and online target #2869

merged 18 commits into from Jan 19, 2023

Conversation

davesh0812
Copy link
Contributor

@davesh0812 davesh0812 commented Jan 3, 2023 •
edited by assaf758

This pr focus in implementation of source, target for SqlDB in mlrun.
SqlDB target can create or read from exists sql collection. The collection is fixed and can't change is schema in the flow.

not support spark

ML-2610

@davesh0812 davesh0812 mentioned this pull request Jan 3, 2023
Closed
@assaf758 assaf758 assigned assaf758 and unassigned assaf758 Jan 10, 2023
@assaf758 assaf758 self-requested a review January 10, 2023 08:21
assaf758
assaf758 requested changes Jan 10, 2023
View reviewed changes
Copy link
Member

@assaf758 assaf758 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any change in requirments?
trying to run tests was getting error: ModuleNotFoundError: No module named 'MySQLdb'

tests/system/env-template.yml Outdated Show resolved Hide resolved
tests/system/feature_store/test_sql_db.py Outdated Show resolved Hide resolved
tests/system/env-template.yml Outdated Show resolved Hide resolved
tests/system/feature_store/test_sql_db.py Outdated Show resolved Hide resolved
assaf758
assaf758 requested changes Jan 10, 2023
View reviewed changes
Copy link
Member

@assaf758 assaf758 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please rebase pr to latest development.

assaf758
assaf758 reviewed Jan 16, 2023
View reviewed changes
Copy link
Member

@assaf758 assaf758 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. added some comments inline
  2. what about support for runtimes? seems MLRUN_SQL_DB_PATH_STRING is not propagated there, and no testing
  3. spark engine support?

tests/system/base.py Outdated Show resolved Hide resolved
mlrun/datastore/sources.py Outdated Show resolved Hide resolved
mlrun/datastore/sources.py Outdated Show resolved Hide resolved
mlrun/feature_store/api.py Outdated Show resolved Hide resolved
mlrun/feature_store/feature_vector.py Outdated Show resolved Hide resolved
@davesh0812 davesh0812 closed this Jan 16, 2023
@davesh0812 davesh0812 reopened this Jan 16, 2023
assaf758
assaf758 requested changes Jan 18, 2023
View reviewed changes
mlrun/datastore/targets.py Outdated Show resolved Hide resolved
mlrun/datastore/targets.py Outdated Show resolved Hide resolved
@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

assaf758
assaf758 requested changes Jan 19, 2023
View reviewed changes
Copy link
Member

@assaf758 assaf758 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

almost done - few last comments.

mlrun/datastore/targets.py Outdated Show resolved Hide resolved
tests/system/feature_store/test_sql_db.py Outdated Show resolved Hide resolved
mlrun/config.py Outdated Show resolved Hide resolved
tests/system/env-template.yml Outdated Show resolved Hide resolved
@assaf758 assaf758 changed the title [Data Store] My Sql - target, source and driver [FeatureStore] Support SQL DBs as source and online target Jan 19, 2023
@assaf758 assaf758 merged commit 4311f75 into mlrun:development Jan 19, 2023
@gtopper gtopper mentioned this pull request May 31, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@assaf758 assaf758 assaf758 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@davesh0812 @assaf758