Skip to content

v1.0.13 - Simplify Cluster Management

Choose a tag to compare

View all tags
@deeco deeco released this 20 Jun 11:15
· 10 commits to main since this release
v1.0.13
58a722f

What's Changed

🐛 Bug Fixes

  • Simplified cluster management: Removed complex cluster start commands that were causing errors
  • Fixed JSON parsing errors: Eliminated jq commands that were failing due to incorrect JSON structure
  • Streamlined workflow: Let bundle deploy handle cluster creation and starting automatically

📋 Technical Details

  • Removed separate cluster start commands that were causing syntax errors
  • Simplified to use Warning: unknown field: apps
    at targets.dev.resources
    in databricks.yml:20:7

Warning: unknown field: apps
at resources
in resources/app.yml:2:3

Warning: unknown field: clusters
at resources
in resources/clusters.yml:2:3

Error: cannot resolve bundle auth configuration: validate: more than one authorization method configured: oauth and pat. Config: host=https://ias-dataplatform-dev.cloud.databricks.com/, token=, client_id=your-client-id, client_secret=. Env: DATABRICKS_HOST, DATABRICKS_TOKEN, DATABRICKS_CLIENT_ID, DATABRICKS_CLIENT_SECRET which handles cluster lifecycle

  • Increased wait time to 120 seconds for better cluster initialization
  • Maintained proper job sequencing and authentication

💡 Why This Approach Works Better

The Databricks Asset Bundle system is designed to handle cluster lifecycle automatically. By removing manual cluster start commands, we let the bundle system:

  1. Create the cluster with proper configuration
  2. Start the cluster automatically
  3. Manage the cluster state properly

This release should resolve the cluster start issues and simplify the deployment process.

Assets 2
Loading