[Feature Store] Fix remote ingest parquet filters #5631

tomerm-iguazio · 2024-05-26T15:07:04Z

Json serialization changes additional filters by transforming tuples into lists.
This issue cause remote run to fail.

Fix this issue by transform lists back to tuples.
Added system and unit tests.
Implement this solution on ParquetTarget and ParquetSource.
As a following Idea -> In order to properly check our test, we should perform the same process of encoding and decoding from JSON in a local ingestion as well.

ML-6563

switch and upgrade tests

…lters

tests/datastore/test_targets.py

mlrun/datastore/targets.py

mlrun/datastore/utils.py

tests/datastore/test_sources.py

tests/datastore/test_targets.py

mlrun/datastore/utils.py

tests/datastore/test_base.py

tests/system/feature_store/test_feature_store.py

mlrun/datastore/sources.py

tests/system/feature_store/test_feature_store.py

mlrun/datastore/utils.py

tests/datastore/test_sources.py

mlrun/datastore/utils.py

mlrun/datastore/sources.py

mlrun/datastore/utils.py

tomerm-iguazio and others added 21 commits May 23, 2024 15:43

add prints

5ca320f

add prints 2

2e63ae1

add prints 2

c6fc7e8

add validate method to revert_list_filters_to_tuple

8e733d5

add unit tests for revert_list_filters_to_tuple

acea8e0

fix to step additional filters fix.

8b2eb69

change as df + prints

8a87217

change to None

2c651d5

add print to parquetsource in to step

205c3f3

switch revert_list_filters_to_tuple to be used as a property...

4bc6d03

change comment

f2cbd17

add prints for dasks reader

159f454

use revert_list_filters_to_tuple also in target.

7a002bd

switch and upgrade tests

add target test.

16115b7

remove some prints

b423c55

change test.

e7da252

delete prints

4feba5e

fix image

c427eb7

Merge branch 'upstream-development' into fix_remote_ingest_parquet_fi…

b174638

…lters

change from_dict builder

2f0cea1

Merge branch 'development' into fix_remote_ingest_parquet_filters

9d00741

assaf758 requested a review from gtopper May 28, 2024 11:38

Merge branch 'development' into fix_remote_ingest_parquet_filters

aa8d5b0

gtopper reviewed May 29, 2024

View reviewed changes

gtopper changed the title ~~[FeatureStore] Fix remote ingest parquet filters~~ [Feature Store] Fix remote ingest parquet filters May 29, 2024

tomerm-iguazio added 5 commits May 29, 2024 17:39

rename function

df08473

delete json_change function

a7da238

remove set

be82197

move test_transform_list_filters_to_tuple to test_utils.py

cd784e6

fix hints and remove target name

93105d4

tomerm-iguazio requested a review from gtopper May 29, 2024 15:43

tomerm-iguazio closed this May 29, 2024

tomerm-iguazio reopened this May 29, 2024

gtopper reviewed May 30, 2024

View reviewed changes

tests/system/feature_store/test_feature_store.py Outdated Show resolved Hide resolved

gtopper reviewed May 30, 2024

View reviewed changes

mlrun/datastore/utils.py Outdated Show resolved Hide resolved

gtopper reviewed May 30, 2024

View reviewed changes

mlrun/datastore/utils.py Outdated Show resolved Hide resolved

gtopper reviewed May 30, 2024

View reviewed changes

tests/datastore/test_sources.py Outdated Show resolved Hide resolved

gtopper reviewed May 30, 2024

View reviewed changes

mlrun/datastore/utils.py Outdated Show resolved Hide resolved

gtopper reviewed May 30, 2024

View reviewed changes

mlrun/datastore/sources.py Show resolved Hide resolved

gtopper reviewed May 30, 2024

View reviewed changes

mlrun/datastore/sources.py Show resolved Hide resolved

gtopper reviewed May 30, 2024

View reviewed changes

mlrun/datastore/sources.py Outdated Show resolved Hide resolved

gtopper reviewed May 30, 2024

View reviewed changes

mlrun/datastore/utils.py Outdated Show resolved Hide resolved

gtopper reviewed May 30, 2024

View reviewed changes

mlrun/datastore/utils.py Outdated Show resolved Hide resolved

tomerm-iguazio added 9 commits May 30, 2024 10:05

use type_error as variable

373889a

change var name

7832eb8

rename var

3341e20

change function call + import

b948a54

change validate_additional_filters

ff8f7fc

change condition in validate_additional_filters

0b24900

lint

de7f58e

add exception for nested list check

eaee97b

fix exception

045df00

tomerm-iguazio requested a review from gtopper May 30, 2024 10:41

gtopper reviewed May 30, 2024

View reviewed changes

mlrun/datastore/utils.py Outdated Show resolved Hide resolved

gtopper approved these changes May 30, 2024

View reviewed changes

change condition

ed16a91

tomerm-iguazio closed this May 30, 2024

tomerm-iguazio reopened this May 30, 2024

assaf758 merged commit e452539 into mlrun:development May 30, 2024
20 checks passed

tomerm-iguazio deleted the fix_remote_ingest_parquet_filters branch June 3, 2024 12:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature Store] Fix remote ingest parquet filters #5631

[Feature Store] Fix remote ingest parquet filters #5631

tomerm-iguazio commented May 26, 2024 •

edited by jira bot

Loading

[Feature Store] Fix remote ingest parquet filters #5631

[Feature Store] Fix remote ingest parquet filters #5631

Conversation

tomerm-iguazio commented May 26, 2024 • edited by jira bot Loading

tomerm-iguazio commented May 26, 2024 •

edited by jira bot

Loading