Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable environment overrides for job tasks #779

Merged
merged 4 commits into from
Sep 18, 2023
Merged

Conversation

andrewnester
Copy link
Contributor

Changes

Follow up for #658

When a job definition has multiple job tasks using the same key, it's considered invalid. Instead we should combine those definitions with the same key into one. This is consistent with environment overrides. This way, the override ends up in the original job tasks, and we've got a clear way to put them all together.

Tests

Added unit tests

Copy link
Contributor

@pietern pietern left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor changes needed, otherwise LGTM

bundle/config/resources/job.go Show resolved Hide resolved
bundle/tests/override_job_tasks/databricks.yml Outdated Show resolved Hide resolved
Copy link
Contributor

@pietern pietern left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit remaining, good to merge on update.

// The tasks field is a slice, and as such, overrides are appended to it.
// We can identify a task by its task key, however, so we can use this key
// to figure out which definitions are actually overrides and merge them.
func (j *Job) MergeJobTasks() error {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Function name also includes "JobTasks".

@andrewnester andrewnester added this pull request to the merge queue Sep 18, 2023
Merged via the queue into main with commit 43e2eef Sep 18, 2023
4 checks passed
@andrewnester andrewnester deleted the apply-task-overrides branch September 18, 2023 14:18
pietern added a commit that referenced this pull request Sep 20, 2023
Bundles:
 * Use enums for default python template ([#765](#765)).
 * Make bundle deploy work if no resources are defined ([#767](#767)).
 * Added support for experimental scripts section ([#632](#632)).
 * Error when unknown keys are encounters during template execution ([#766](#766)).
 * Fall back to full Git clone if shallow clone is not supported ([#775](#775)).
 * Enable environment overrides for job tasks ([#779](#779)).
 * Increase timeout waiting for job run to 1 day ([#786](#786)).

Internal:
 * Update Go SDK to v0.19.3 (unreleased) ([#778](#778)).
@pietern pietern mentioned this pull request Sep 20, 2023
github-merge-queue bot pushed a commit that referenced this pull request Sep 20, 2023
Bundles:
* Use enums for default python template
([#765](#765)).
* Make bundle deploy work if no resources are defined
([#767](#767)).
* Added support for experimental scripts section
([#632](#632)).
* Error when unknown keys are encounters during template execution
([#766](#766)).
* Fall back to full Git clone if shallow clone is not supported
([#775](#775)).
* Enable environment overrides for job tasks
([#779](#779)).
* Increase timeout waiting for job run to 1 day
([#786](#786)).

Internal:
* Update Go SDK to v0.19.3 (unreleased)
([#778](#778)).
pietern added a commit that referenced this pull request Sep 21, 2023
This is a follow up to #658 and #779 for jobs.

This change applies label normalization the same way the backend does.
github-merge-queue bot pushed a commit that referenced this pull request Sep 21, 2023
## Changes

This is a follow-up to #658 and #779 for jobs.

This change applies label normalization the same way the backend does.

## Tests

Unit and config loading tests.
hectorcast-db pushed a commit that referenced this pull request Oct 13, 2023
## Changes
Follow up for #658

When a job definition has multiple job tasks using the same key, it's
considered invalid. Instead we should combine those definitions with the
same key into one. This is consistent with environment overrides. This
way, the override ends up in the original job tasks, and we've got a
clear way to put them all together.

## Tests
Added unit tests
hectorcast-db pushed a commit that referenced this pull request Oct 13, 2023
Bundles:
* Use enums for default python template
([#765](#765)).
* Make bundle deploy work if no resources are defined
([#767](#767)).
* Added support for experimental scripts section
([#632](#632)).
* Error when unknown keys are encounters during template execution
([#766](#766)).
* Fall back to full Git clone if shallow clone is not supported
([#775](#775)).
* Enable environment overrides for job tasks
([#779](#779)).
* Increase timeout waiting for job run to 1 day
([#786](#786)).

Internal:
* Update Go SDK to v0.19.3 (unreleased)
([#778](#778)).
hectorcast-db pushed a commit that referenced this pull request Oct 13, 2023
This is a follow-up to #658 and #779 for jobs.

This change applies label normalization the same way the backend does.

Unit and config loading tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants