Skip to content

Conversation

@denik
Copy link
Contributor

@denik denik commented Sep 5, 2025

Changes

  • New hidden command "bundle ref-schema".
  • Acceptance test that records full output of it.

Requires #3531

Why

Track bundle schema changes. Unlike jsonschema, this includes not just input yaml but stored state and remote resource types. This will help us notice when something disappears or changes type in the state or remote, as this has potential of breaking existing deployments.

It is also in greppable format, could be useful for debugging complex $resources.jobs.foo.tasks[0].something.something else references.

Tests

I also added recording of "bundle debug" output under acceptance/bundle/debug.

@eng-dev-ecosystem-bot
Copy link
Collaborator

eng-dev-ecosystem-bot commented Sep 5, 2025

Run: 17554524923

Env ✅​pass 🔄​flaky 🙈​skip
✅​ aws linux 308 523
✅​ aws windows 309 522
✅​ aws-ucws linux 420 421
✅​ aws-ucws windows 421 420
✅​ azure linux 308 522
✅​ azure windows 309 521
✅​ azure-ucws linux 420 420
🔄​ azure-ucws windows 419 2 419
✅​ gcp linux 307 524
✅​ gcp windows 308 523
Test Name azure-ucws windows
TestSparkJarTaskDeployAndRunOnVolumes 🔄​flaky
TestSparkJarTaskDeployAndRunOnVolumes/Databricks_Runtime_15.4_LTS 🔄​flaky

@denik denik temporarily deployed to test-trigger-is September 5, 2025 12:49 — with GitHub Actions Inactive
@denik denik force-pushed the denik/remote-state branch from f99310a to f6dad89 Compare September 8, 2025 08:28
@denik denik force-pushed the denik/bundle-refschema branch from 0048704 to 3cab680 Compare September 8, 2025 08:41
@denik denik temporarily deployed to test-trigger-is September 8, 2025 08:41 — with GitHub Actions Inactive
Copy link
Contributor

@pietern pietern left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The output highlights the ambiguity between the "settings" and "spec" duplication for jobs and pipelines, respectively. Let's discuss. Not blocking for this PR.

Please check the suggestion before merging.

cmd.AddCommand(deployment.NewDeploymentCommand())
cmd.AddCommand(newOpenCommand())
cmd.AddCommand(newPlanCommand())
cmd.AddCommand(newRefSchemaCommand())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use the debug subcommand to host this, to group commands we'll keep hidden.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, we can move it there. In that case it does not need to be hidden.

Base automatically changed from denik/remote-state to main September 8, 2025 13:45
@denik denik force-pushed the denik/bundle-refschema branch from 3cab680 to 6f52ca4 Compare September 8, 2025 14:11
@denik denik temporarily deployed to test-trigger-is September 8, 2025 14:11 — with GitHub Actions Inactive
@denik denik temporarily deployed to test-trigger-is September 8, 2025 14:26 — with GitHub Actions Inactive
@denik denik changed the title Add hidden "bundle ref-schema" command Add hidden "bundle debug refschema" command Sep 8, 2025
@denik denik temporarily deployed to test-trigger-is September 8, 2025 14:34 — with GitHub Actions Inactive
@denik denik temporarily deployed to test-trigger-is September 8, 2025 14:38 — with GitHub Actions Inactive
@denik denik enabled auto-merge September 8, 2025 14:44
@denik denik added this pull request to the merge queue Sep 8, 2025
Merged via the queue into main with commit 30849a3 Sep 8, 2025
13 checks passed
@denik denik deleted the denik/bundle-refschema branch September 8, 2025 15:16
github-merge-queue bot pushed a commit that referenced this pull request Sep 9, 2025
…pe (#3573)

## Changes

Pure refactoring, no functional changes.

## Why

Follow to
#3531 (comment)

We already use STATE to refer to this schema in
#3555

Config is ambiguous, we have already bundle config (called input config
in terranova).

With this rename we have

- input config -- what's in databricks.yml
- state - what's persisted, what is sent to backend via create/update
- remote state - what we get from backend via get requests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants