Add `databricks_mlflow_experiment` data source #2389

840 · 2023-06-08T16:30:17Z

Changes

Add databricks_mlflow_experiment data source

Tests

Created 3 tests for this data source

make test run locally
relevant change in docs/ folder
covered with integration tests in internal/acceptance
relevant acceptance tests are passing
using Go SDK

Partially resolves #2301

840 · 2023-06-08T16:31:58Z

Currently you can only fetch the mlflow_experiment data source using the name in this PR. I believe it would be nice to provide the option to fetch using the ID, as well. I have to read a bit more into this.

codecov-commenter · 2023-06-08T16:38:48Z

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (4213468) 83.57% compared to head (9b8c8be) 83.57%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2389      +/-   ##
==========================================
- Coverage   83.57%   83.57%   -0.01%     
==========================================
  Files         168      169       +1     
  Lines       15083    15107      +24     
==========================================
+ Hits        12606    12625      +19     
- Misses       1736     1739       +3     
- Partials      741      743       +2

Files	Coverage Δ
provider/provider.go	`94.47% <100.00%> (+0.03%)`	⬆️
mlflow/data_mlflow_experiment.go	`91.30% <91.30%> (ø)`

... and 1 file with indirect coverage changes

nkvuong · 2023-06-08T17:07:09Z

@840 thank you for raising the PR. a couple of thoughts for the pending tasks

you could look at https://github.com/databricks/terraform-provider-databricks/blob/master/clusters/data_cluster.go for example of searching both by name & id

the Go SDK has GetExperiment method which takes an ExperimentId as well.

For acceptance tests, you could create an mlflow experiment and try to read it in the same block, similar to https://github.com/databricks/terraform-provider-databricks/blob/master/internal/acceptance/data_job_test.go

nkvuong · 2023-06-08T17:09:34Z

mlflow/data_mlflow_experiment.go

+			return err
+		}
+		experiment := experimentResponse.Experiment
+		data.Name = experiment.Name


if name is provided, then you don't need to overwrite it

nkvuong · 2023-06-08T17:10:19Z

docs/data-sources/mlflow_experiment.md

+This data source exports the following attributes:
+
+* `artifact_location` - Location where artifacts for the experiment are stored.
+* `name` - Human readable name that identifies the experiment.


Need to be consistent with the name argument reference

nkvuong · 2023-06-08T17:12:16Z

docs/data-sources/mlflow_experiment.md

+
+* `artifact_location` - Location where artifacts for the experiment are stored.
+* `name` - Human readable name that identifies the experiment.
+* `lifecycle_stage` - Current life cycle stage of the experiment: "active" or "deleted". Deleted experiments are not returned by APIs.


Get by Id will work on deleted experiments

Suggested change

* `lifecycle_stage` - Current life cycle stage of the experiment: "active" or "deleted". Deleted experiments are not returned by APIs.

* `lifecycle_stage` - Current life cycle stage of the experiment: `active` or `deleted`. Deleted experiments are not returned by APIs.

I guess it has to be changed in the Databricks documentation, as well.

840 · 2023-06-12T09:58:26Z

Blocked by databricks/databricks-sdk-go#428

mgyucht

Quick first pass. LMK when no longer WIP.

mlflow/data_mlflow_experiment.go

…a source

…del in the future

840 · 2024-01-04T12:17:28Z

Will continue this week or early next week.

840 · 2024-01-12T17:08:56Z

@mgyucht WIP tag removed, should be ready!

mlflow/data_mlflow_experiment.go

alexott

Thank you!

### New Features and Improvements * Add owner support to `databricks_sql_table` ([#3570](#3570)). * Add `databricks_catalog` data source ([#3573](#3573)). * Add `databricks_table` data source ([#3571](#3571)). * Add `databricks_mlflow_experiment` data source ([#2389](#2389)). ### Documentation Changes ### Exporter * Fix rare race condition with variables map ([#3568](#3568)). ### Internal Changes Dependency updates: * Bump go SDK to 0.40.1 ([#3574](#3574)), featuring improvements in tracing and debuggability of failed requests.

nkvuong reviewed Jun 8, 2023

View reviewed changes

840 force-pushed the issue-2301-1 branch from 2b81b44 to c534e0a Compare June 9, 2023 15:18

nkvuong force-pushed the issue-2301-1 branch from aa9ebb9 to 9a52a94 Compare July 4, 2023 15:00

nkvuong requested a review from a team July 4, 2023 15:01

mgyucht reviewed Jul 4, 2023

View reviewed changes

mlflow/data_mlflow_experiment.go Outdated Show resolved Hide resolved

840 added 11 commits January 4, 2024 13:13

Add data source databricks_mlflow_experiment

e8e369d

Add tests for data source databricks_mlflow_experiment

aeb91ef

Register databricks_mlflow_experiment

bfae85a

Change experiment_id to id for databricks_mlflow_experiment dat…

b759ead

…a source

Add documentation for databricks_mlflow_experiment data source

f4b6385

Add way to fetch databricks_mlflow_experiment using the Experiment ID

8286da6

Add tests for databricks_mlflow_experiment using the Experiment ID

4fe257e

Provide example to get databricks_mlflow_experiment by Experiment ID

f8bb34b

Change in doc

ac03f31

Refactor id to experiment_id to make it compatible with Go SDK mo…

d189a3c

…del in the future

Add acceptance test for databricks_mlflow_experiment data source

d3e08fd

840 force-pushed the issue-2301-1 branch from f3d62b4 to d3e08fd Compare January 4, 2024 12:14

840 requested review from a team as code owners January 4, 2024 12:14

840 requested review from mgyucht and removed request for a team January 4, 2024 12:14

840 added 2 commits January 12, 2024 16:39

Fix code & tests based on new OpenAPI specifications

1be394f

Merge branch 'databricks:master' into issue-2301-1

5f4a0ef

840 changed the title ~~[WIP] Add databricks_mlflow_experiment data source~~ Add databricks_mlflow_experiment data source Jan 12, 2024

Improve acceptance test

a59bf9d

840 force-pushed the issue-2301-1 branch from aa48a9a to a59bf9d Compare January 12, 2024 16:38

Merge branch 'master' into issue-2301-1

969010a

840 added 7 commits January 22, 2024 18:03

Merge branch 'master' into issue-2301-1

b99f8d2

Use experiment schema from SDK and map id to experiment_id

144c5a4

Merge branch 'master' into issue-2301-1

7ba0966

Update mlflow_experiment doc

6e2e56d

.

d286f43

Merge branch 'main' into issue-2301-1

c838679

Use common.Resource and modify provider.go

9b8c8be

840 marked this pull request as draft February 7, 2024 10:46

840 added 2 commits February 14, 2024 13:15

Merge branch 'main' into issue-2301-1

1c45c35

Merge branch 'databricks:main' into issue-2301-1

8b3187f

mgyucht reviewed Apr 8, 2024

View reviewed changes

mlflow/data_mlflow_experiment.go Outdated Show resolved Hide resolved

840 added 2 commits April 8, 2024 17:51

Make id optional and remove omitempty attribute

120fc21

Merge branch 'main' into issue-2301-1

7e7cf2f

840 marked this pull request as ready for review April 8, 2024 15:51

840 requested a review from mgyucht April 8, 2024 15:54

alexott approved these changes May 14, 2024

View reviewed changes

alexott added this pull request to the merge queue May 14, 2024

Merged via the queue into databricks:main with commit c1b792c May 14, 2024
5 checks passed

mgyucht mentioned this pull request May 14, 2024

Release v1.44.0 #3576

Merged

840 deleted the issue-2301-1 branch May 20, 2024 15:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `databricks_mlflow_experiment` data source #2389

Add `databricks_mlflow_experiment` data source #2389

840 commented Jun 8, 2023 •

edited

840 commented Jun 8, 2023 •

edited

codecov-commenter commented Jun 8, 2023 •

edited

nkvuong commented Jun 8, 2023

nkvuong Jun 8, 2023

nkvuong Jun 8, 2023

nkvuong Jun 8, 2023

840 Jun 9, 2023

840 commented Jun 12, 2023

mgyucht left a comment

840 commented Jan 4, 2024

840 commented Jan 12, 2024

alexott left a comment

	* `lifecycle_stage` - Current life cycle stage of the experiment: "active" or "deleted". Deleted experiments are not returned by APIs.
	* `lifecycle_stage` - Current life cycle stage of the experiment: `active` or `deleted`. Deleted experiments are not returned by APIs.

Add databricks_mlflow_experiment data source #2389

Add databricks_mlflow_experiment data source #2389

Conversation

840 commented Jun 8, 2023 • edited

Changes

Tests

840 commented Jun 8, 2023 • edited

codecov-commenter commented Jun 8, 2023 • edited

Codecov Report

nkvuong commented Jun 8, 2023

nkvuong Jun 8, 2023

Choose a reason for hiding this comment

nkvuong Jun 8, 2023

Choose a reason for hiding this comment

nkvuong Jun 8, 2023

Choose a reason for hiding this comment

840 Jun 9, 2023

Choose a reason for hiding this comment

840 commented Jun 12, 2023

mgyucht left a comment

Choose a reason for hiding this comment

840 commented Jan 4, 2024

840 commented Jan 12, 2024

alexott left a comment

Choose a reason for hiding this comment

Add `databricks_mlflow_experiment` data source #2389

Add `databricks_mlflow_experiment` data source #2389

840 commented Jun 8, 2023 •

edited

840 commented Jun 8, 2023 •

edited

codecov-commenter commented Jun 8, 2023 •

edited