[Spark] Enable support for non-aggregation operations in Spark using the streaming interface #5599

alxtkr77 · 2024-05-20T12:58:58Z

https://iguazio.atlassian.net/browse/ML-6474

Merge 1.7.0-rc17
Enable support for non-aggregation operations in Spark using the streaming interface.
Replace hasattr(obj, 'rdd') with 'rdd' in dir(obj) since hasattr() evaluates the property, which can lead to unintended behavior.
Update the system to handle single Parquet and CSV files.

…'container' (mlrun#5521)

* how to register a datastore profile * Update docs/store/datastore.md Co-authored-by: Gal Topper <gal.topper@gmail.com> * Update docs/store/datastore.md Co-authored-by: Gal Topper <gal.topper@gmail.com> * fix: data store is 2 words * clarify example --------- Co-authored-by: Gal Topper <gal.topper@gmail.com>

…lrun#5526)

…iled_recent_count (mlrun#5523)

* Remove unused import * Change import so that PyCharm detects inheritance correctly

* Add step kind to pipeline notification * better system test * redundant comment * fix ut --------- Co-authored-by: quaark <a.melnick@icloud.com>

…leanups in datastores (mlrun#5305)

…nContext` (mlrun#5552) fix deepcopy bug

)

mlrun#5545)

…asures in the writer (mlrun#5554)

* [Model Monitoring] Add new TSDB target for all predictions Tracking count and latency. [ML-6349](https://iguazio.atlassian.net/browse/ML-6349) * Change TSDB predictions table path * Move predictions table * Fix * Fix parameter names * Fix

…t_scheduled_pipelines_pending_count (mlrun#5535)

…mlrun#5563) fix pipeline notification bugs Co-authored-by: quaark <a.melnick@icloud.com>

…ion (mlrun#5543)

…lrun#5572)

)

alxtkr77 and others added 30 commits May 7, 2024 10:32

[Datastore] Rename DatastoreProfileAzureBlob 'bucket' parameter to …

afd5bd4

…'container' (mlrun#5521)

[Project] Add delete artifact method (mlrun#5491)

913f02a

[API] Add delete artifact data endpoints (mlrun#5477)

127d1e8

[Alerts] Add backwards compatibility for igz < 3.6 (mlrun#5524)

b26fd43

[Coding Conventions] Fix mlconf usages (mlrun#5525)

f376b28

[API] Enhance usage of iguazio client (mlrun#5506)

7034459

[Model Monitoring] Implement TSDB abstraction (mlrun#5210)

8c0defe

[Automation] Don't rollout deployment before purging DB (mlrun#5527)

79608f2

[SQLDB] Refactor partition querying to minimize subquery table size (m…

23af50a

…lrun#5526)

[Docs] Enable automodule for mlrun.comon.schemas.artifact (mlrun#5529)

e2b0d85

[Alerts] Filter project resources by permissions for list (mlrun#5532)

2bd1a5d

[ProjectSummaries] Add pipelines_completed_recent_count, pipelines_fa…

08071fd

…iled_recent_count (mlrun#5523)

[KFP] Merge KFP2 Feature Branch (mlrun#5533)

f747bd0

[Nuclio] Delete remote function config map (mlrun#5530)

a93ee1d

[API] Add read model monitoring metrics values endpoint (mlrun#5519)

599900c

[Model Monitoring] Fix up imports (mlrun#5531)

fb3637c

* Remove unused import * Change import so that PyCharm detects inheritance correctly

[Notifications] Display Step Kind In Pipeline Notification (mlrun#5538)

e3eafff

* Add step kind to pipeline notification * better system test * redundant comment * fix ut --------- Co-authored-by: quaark <a.melnick@icloud.com>

[Alerts] Modify alert entity to have list of ids per alert (mlrun#5511)

88f07f8

[Application] Fix test function handler assertion (mlrun#5548)

a4a3e4e

[Docs] Feature-store + Datastore: consolidate Apache Kafka details, c…

8dbac8e

…leanups in datastores (mlrun#5305)

[Model Monitoring] Fix invalid current_stats in `MonitoringApplicatio…

617e572

…nContext` (mlrun#5552) fix deepcopy bug

[KFP] Bump version and create makefile for release (mlrun#5555)

23aec36

[Runs] Store artifact URIs instead of full artifacts in body (mlrun#5553

4478187

)

[Artifacts] Add producer_uri filter to list artifacts API (mlrun#5549)

61a874f

[API] Check the artifact kind while attempting to remove artifact data (

9c47b4c

mlrun#5545)

[CI] Fix pipeline adapters makefile (mlrun#5557)

1066a9f

[Model Monitoring] Update model endpoint's current stats and drift me…

db779a5

…asures in the writer (mlrun#5554)

[Functions] Update function with deletion task id (mlrun#5536)

40a74a3

TomerShor and others added 19 commits May 14, 2024 22:29

[Application] Create API Gateway by default (mlrun#5515)

6fa0bfb

[Logger] Enhance formatter creation (mlrun#5561)

4fef2e6

[KFP] Suppress reusable-components warnings (mlrun#5501)

652c12d

[ProjectSummaries] Add distinct_scheduled_jobs_pending_count, distinc…

3e18472

…t_scheduled_pipelines_pending_count (mlrun#5535)

[Notifications] Fix Pipeline Notification Log Spam + Git Notification (…

7a85be1

…mlrun#5563) fix pipeline notification bugs Co-authored-by: quaark <a.melnick@icloud.com>

[Docs] Compile notification when compiling docs (mlrun#5564)

d56334b

[Helpers] Ignore pre-release in min_iguazio_version (mlrun#5558)

c9372c3

[Runs] Enrich run with artifacts when getting a single run (mlrun#5565)

0908a90

[Application] Set api gateway name on status (mlrun#5567)

6a5d8eb

[CI] Bundling demos as release asset (mlrun#5550)

b7053f4

[Nuclio-jupyter] Bump nuclio-jupyter to 0.9.17 (mlrun#5568)

9d3152d

[FrontendSpec] Adding TTLCache to try_get_grafana_service_url funct…

a1899d4

…ion (mlrun#5543)

[KFP] Fix watch=True for remote pipeline when using set_workflow (m…

f0a1af9

…lrun#5572)

[Alerts] Add cooldown_period to notification config in alert (mlrun#5547

d437ecc

)

[Model] Replace yaml dump with safe_dump (mlrun#5571)

cb22edc

[Versioning] Fix package version for dev feature branches (mlrun#5575)

fb7d21e

Merge tag 'v1.7.0-rc17' into feature/spark-streaming

6313b4a

repace hasattr() with dir()

729b953

Support single Parquet and CSV files

3e53004

assaf758 merged commit 364839d into mlrun:feature/spark-streaming May 21, 2024
10 of 11 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Spark] Enable support for non-aggregation operations in Spark using the streaming interface #5599

[Spark] Enable support for non-aggregation operations in Spark using the streaming interface #5599

alxtkr77 commented May 20, 2024

[Spark] Enable support for non-aggregation operations in Spark using the streaming interface #5599

[Spark] Enable support for non-aggregation operations in Spark using the streaming interface #5599

Conversation

alxtkr77 commented May 20, 2024