Support passing job parameters to bundle run #1115
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This change adds support for job parameters. If job parameters are specified for a job that doesn't define job parameters it returns an error. Conversely, if task parameters are specified for a job that defines job parameters, it also returns an error. This changes moves the options structs and their functions to separate files and backfills test coverage for them. Job parameters can now be specified with `--param foo=bar,bar=qux`.
pietern
requested review from
lennartkats-db,
andrewnester and
shreyas-goenka
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #1115 +/- ##
==========================================
+ Coverage 49.08% 49.52% +0.44%
==========================================
Files 270 275 +5
Lines 10556 10591 +35
==========================================
+ Hits 5181 5245 +64
+ Misses 4821 4790 -31
- Partials 554 556 +2 ☔ View full report in Codecov by Sentry. |
| fs.StringToStringVar(&o.sqlParams, "sql-params", nil, "A map from keys to values for jobs with SQL tasks.") | ||
|
|
||
| // Define job parameters flag. | ||
| fs.StringToStringVar(&o.jobParams, "params", nil, "comma separated k=v pairs for job parameters") |
There was a problem hiding this comment.
What does this look like in the help output?
For e.g. notebook params, stringToString looks pretty weird:
--notebook-params stringToString A map from keys to values for jobs with notebook tasks. (default [])
There was a problem hiding this comment.
It looks the same. We should address this in a separate change (I copied most of the code from job.go).
| return fmt.Errorf("job not defined") | ||
| } | ||
|
|
||
| // Ensure mutual exclusion on job parameters and task parameters. |
lennartkats-db
approved these changes
Jan 12, 2024
andrewnester
approved these changes
Jan 14, 2024
| } | ||
|
|
||
| func (o *PipelineOptions) Define(fs *flag.FlagSet) { | ||
| fs.BoolVar(&o.RefreshAll, "refresh-all", false, "Perform a full graph update.") |
There was a problem hiding this comment.
The name of flags are very unintuitive but I guess they are just in line with API naming?
There was a problem hiding this comment.
Indeed. Aside from API, this is also in line with how these are named in the web UI.
andrewnester
added a commit
that referenced
this pull request
Jan 17, 2024
CLI: * Fix windows style file paths in fs cp command ([#1118](#1118)). * Do not require positional arguments if they should be provided in JSON ([#1125](#1125)). * Always require path parameters as positional arguments ([#1129](#1129)). Bundles: * Add debug log line for when bundle init is run from non-TTY interface ([#1117](#1117)). * Added `databricks bundle generate job` command ([#1043](#1043)). * Support passing job parameters to bundle run ([#1115](#1115)). Dependency updates: * Bump golang.org/x/oauth2 from 0.15.0 to 0.16.0 ([#1124](#1124)).
Merged
github-merge-queue bot
pushed a commit
that referenced
this pull request
Jan 17, 2024
CLI: * Fix windows style file paths in fs cp command ([#1118](#1118)). * Do not require positional arguments if they should be provided in JSON ([#1125](#1125)). * Always require path parameters as positional arguments ([#1129](#1129)). Bundles: * Add debug log line for when bundle init is run from non-TTY interface ([#1117](#1117)). * Added `databricks bundle generate job` command ([#1043](#1043)). * Support passing job parameters to bundle run ([#1115](#1115)). Dependency updates: * Bump golang.org/x/oauth2 from 0.15.0 to 0.16.0 ([#1124](#1124)).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
This change adds support for job parameters. If job parameters are specified for a job that doesn't define job parameters it returns an error. Conversely, if task parameters are specified for a job that defines job parameters, it also returns an error.
This change moves the options structs and their functions to separate files and backfills test coverage for them.
Job parameters can now be specified with
--params foo=bar,bar=qux.Tests
Unit tests and manual integration testing.