Persist deployment metadata in WSFS

[shreyas-goenka]

Changes

This PR introduces a metadata struct that stores a subset of bundle configuration that we wish to expose to other Databricks services that wish to integrate with bundles.

This metadata file is uploaded to a file ${bundle.workspace.state_path}/metadata.json in the WSFS destination of the bundle deployment.

Documentation for emitted metadata fields:

• version: Version for the metadata file schema
• config.bundle.git.branch: Name of the git branch the bundle was deployed from.
• config.bundle.git.origin_url: URL for git remote "origin"
• config.bundle.git.bundle_root_path: Relative path of the bundle root from the root of the git repository. Is set to "." if they are the same.
• config.bundle.git.commit: SHA-1 commit hash of the exact commit this bundle was deployed from. Note, the deployment might not exactly match this commit version if there are changes that have not been committed to git at deploy time,
• file_path: Path in workspace where we sync bundle files to.
• resources.jobs.[job-ref].id: Id of the job
• resources.jobs.[job-ref].relative_path: Relative path of the yaml config file from the bundle root where this job was defined.

Example metadata object when bundle root and git root are the same:

{
  "version": 1,
  "config": {
    "bundle": {
      "lock": {},
      "git": {
        "branch": "master",
        "origin_url": "www.host.com",
        "commit": "7af8e5d3f5dceffff9295d42d21606ccf056dce0",
        "bundle_root_path": "."
      }
    },
    "workspace": {
      "file_path": "/Users/shreyas.goenka@databricks.com/.bundle/pipeline-progress/default/files"
    },
    "resources": {
      "jobs": {
        "bar": {
          "id": "245921165354846",
          "relative_path": "databricks.yml"
        }
      }
    },
    "sync": {}
  }
}

Example metadata when the git root is one level above the bundle repo:

{
  "version": 1,
  "config": {
    "bundle": {
      "lock": {},
      "git": {
        "branch": "dev-branch",
        "origin_url": "www.my-repo.com",
        "commit": "3db46ef750998952b00a2b3e7991e31787e4b98b",
        "bundle_root_path": "pipeline-progress"
      }
    },
    "workspace": {
      "file_path": "/Users/shreyas.goenka@databricks.com/.bundle/pipeline-progress/default/files"
    },
    "resources": {
      "jobs": {
        "bar": {
          "id": "245921165354846",
          "relative_path": "databricks.yml"
        }
      }
    },
    "sync": {}
  }
}

This unblocks integration to the jobs break glass UI for bundles.

Tests

Unit tests and integration tests.

[pietern]

Approach LGTM, most comments are cosmetic.

[sungchiu-db]

Fields look good!
@jwm0 we should be able to use

1. config.bundle.git.bundle_root_path + resources.jobs.[job-ref].relative_path to derive relative job file path in Repo.
2. config.bundle.git.bundle_root_path + (notebook_path - file_path) to derive relative notebook path in a Repo.

[jwm0]

LGTM for metadata content

[shreyas-goenka]

Tested out the new implementation with a deploy. Here's what the metadata looks like:

{
  "version": 1,
  "config": {
    "bundle": {
      "git": {
        "branch": "my-branch-lol",
        "origin_url": "www.abc.com",
        "commit": "ed3b97a45ed7822f9963442bfb938ac36c65579b",
        "bundle_root_path": "."
      }
    },
    "workspace": {
      "file_path": "/Users/shreyas.goenka@databricks.com/.bundle/pipeline-progress/default/files"
    },
    "resources": {
      "jobs": {
        "bar": {
          "id": "1115233850898651",
          "relative_path": "databricks.yml"
        }
      }
    }
  }
}

[pietern]

Great work, espc on the integration test.

Remaining comments are tiny/style.