Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a foundation for built-in templates #685

Merged
merged 13 commits into from
Aug 25, 2023
Merged

Conversation

lennartkats-db
Copy link
Contributor

@lennartkats-db lennartkats-db commented Aug 21, 2023

Changes

This pull request extends the templating support in preparation of a new, default template (WIP, #686):

  • builtin templates that can be initialized using e.g. databricks bundle init default-python
  • builtin templates are embedded into the executable using go's embed functionality, making sure they're co-versioned with the CLI
  • new helpers to get the workspace name, current user name, etc. help craft a complete template
  • (not enabled yet) when the user types databricks bundle init they can interactively select the default-python template

And makes two tangentially related changes:

  • IsServicePrincipal now uses the "users" API rather than the "principals" API, since the latter is too slow for our purposes.
  • mode: prod no longer requires the 'target.prod.git' setting. It's hard to set that from a template. (Pieter is planning an overhaul of warnings support; this would be one of the first warnings we show.)

The actual default-python template is maintained in a separate PR: #686

Tests

Unit tests, manual testing

@lennartkats-db lennartkats-db changed the title Add foundation for a builtin templates Add foundation for a built-in templates Aug 21, 2023
@lennartkats-db lennartkats-db marked this pull request as ready for review August 22, 2023 15:45
@lennartkats-db lennartkats-db changed the title Add foundation for a built-in templates Add foundation for built-in templates Aug 22, 2023
@lennartkats-db lennartkats-db changed the title Add foundation for built-in templates Add a foundation for built-in templates Aug 22, 2023
libs/databrickscfg/profiles.go Outdated Show resolved Hide resolved
bundle/config/mutator/process_target_mode.go Outdated Show resolved Hide resolved
bundle/config/mutator/process_target_mode.go Outdated Show resolved Hide resolved
libs/template/materialize.go Outdated Show resolved Hide resolved
libs/template/materialize.go Outdated Show resolved Hide resolved
cmd/bundle/init.go Outdated Show resolved Hide resolved
@lennartkats-db lennartkats-db added this pull request to the merge queue Aug 25, 2023
Merged via the queue into main with commit a5b8609 Aug 25, 2023
4 checks passed
@lennartkats-db lennartkats-db deleted the template-foundation branch August 25, 2023 09:10
// Determines whether a given user id is a service principal.
// This function uses a heuristic: if no user exists with this id, we assume
// it's a service principal. Unfortunately, the standard service principal API is too
// slow for our purposes.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should do something else. This can fail for other reasons and if it does will incorrectly assume the user is a service principal. If I look at the scim/me output (that we include in workspace.current_user), there are other heuristics that should be more reliable. Example of workspace.current_user for a service principal:

{
  "active":true,
  "displayName":"sdk-1693297374999987000",
  "emails": [
    {
      "primary":true,
      "type":"work",
      "value":"a1e691b7-a6df-4a6b-881e-d93a03a7ce52"
    }
  ],
  "id":"6079497891687986",
  "name": {
    "givenName":"sdk-1693297374999987000"
  },
  "userName":"a1e691b7-a6df-4a6b-881e-d93a03a7ce52"
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is that always the case though, that the username is an id like that? And will that be the case in the future?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually — this should have error handling code. Not sure what happened to it. Let me add it back in a cleanup PR: #707.

libs/template/helpers.go Show resolved Hide resolved
libs/template/materialize.go Show resolved Hide resolved
libs/template/materialize.go Show resolved Hide resolved
libs/template/materialize.go Show resolved Hide resolved
cmd/root/auth.go Show resolved Hide resolved
cmd/root/bundle.go Show resolved Hide resolved
andrewnester added a commit that referenced this pull request Aug 30, 2023
CLI:
 * Fixed --environment flag ([#705](#705)).

Bundles:
 * Support cluster overrides with cluster_key and compute_key ([#696](#696)).
 * Allow referencing local Python wheels without artifacts section defined ([#703](#703)).
 * Correctly identify local paths in libraries section ([#702](#702)).
 * Fixed path joining in FindFilesWithSuffixInPath ([#704](#704)).
 * Added transformation mutator for Python wheel task for them to work on DBR <13.1 ([#635](#635)).

Internal:
 * Add a foundation for built-in templates ([#685](#685)).
@andrewnester andrewnester mentioned this pull request Aug 30, 2023
github-merge-queue bot pushed a commit that referenced this pull request Aug 30, 2023
## Changes
Add some cleanup based on @pietern's comments on
#685
andrewnester added a commit that referenced this pull request Aug 30, 2023
Bundles:
 * Support cluster overrides with cluster_key and compute_key ([#696](#696)).
 * Allow referencing local Python wheels without artifacts section defined ([#703](#703)).
 * Fixed --environment flag ([#705](#705)).
 * Correctly identify local paths in libraries section ([#702](#702)).
 * Fixed path joining in FindFilesWithSuffixInPath ([#704](#704)).
 *  Added transformation mutator for Python wheel task for them to work on DBR <13.1 ([#635](#635)).

Internal:
 * Add a foundation for built-in templates ([#685](#685)).
 * Test transform when no Python wheel tasks defined ([#714](#714)).
 * Pin Terraform binary version to 1.5.5 ([#715](#715)).
 * Cleanup after "Add a foundation for built-in templates" ([#707](#707)).
 * Filter down to Python wheel tasks only for trampoline ([#712](#712)).
 * Update Terraform provider schema structs from 1.23.0 ([#713](#713)).
@andrewnester andrewnester mentioned this pull request Aug 30, 2023
github-merge-queue bot pushed a commit that referenced this pull request Aug 30, 2023
Bundles:
* Support cluster overrides with cluster_key and compute_key
([#696](#696)).
* Allow referencing local Python wheels without artifacts section
defined ([#703](#703)).
* Fixed --environment flag
([#705](#705)).
* Correctly identify local paths in libraries section
([#702](#702)).
* Fixed path joining in FindFilesWithSuffixInPath
([#704](#704)).
* Added transformation mutator for Python wheel task for them to work on
DBR <13.1 ([#635](#635)).

Internal:
* Add a foundation for built-in templates
([#685](#685)).
* Test transform when no Python wheel tasks defined
([#714](#714)).
* Pin Terraform binary version to 1.5.5
([#715](#715)).
* Cleanup after "Add a foundation for built-in templates"
([#707](#707)).
* Filter down to Python wheel tasks only for trampoline
([#712](#712)).
* Update Terraform provider schema structs from 1.23.0
([#713](#713)).
lennartkats-db added a commit that referenced this pull request Sep 1, 2023
lennartkats-db pushed a commit that referenced this pull request Sep 1, 2023
Bundles:
* Support cluster overrides with cluster_key and compute_key
([#696](#696)).
* Allow referencing local Python wheels without artifacts section
defined ([#703](#703)).
* Fixed --environment flag
([#705](#705)).
* Correctly identify local paths in libraries section
([#702](#702)).
* Fixed path joining in FindFilesWithSuffixInPath
([#704](#704)).
* Added transformation mutator for Python wheel task for them to work on
DBR <13.1 ([#635](#635)).

Internal:
* Add a foundation for built-in templates
([#685](#685)).
* Test transform when no Python wheel tasks defined
([#714](#714)).
* Pin Terraform binary version to 1.5.5
([#715](#715)).
* Cleanup after "Add a foundation for built-in templates"
([#707](#707)).
* Filter down to Python wheel tasks only for trampoline
([#712](#712)).
* Update Terraform provider schema structs from 1.23.0
([#713](#713)).
github-merge-queue bot pushed a commit that referenced this pull request Sep 5, 2023
## Changes

This adds a built-in "default-python" template to the CLI. This is based
on the new default-template support of
#685.

The goal here is to offer an experience where customers can simply type
`databricks bundle init` to get a default template:

```
$ databricks bundle init
Template to use [default-python]: default-python
Unique name for this project [my_project]: my_project
✨ Successfully initialized template
```

The present template:
- [x] Works well with VS Code
- [x] Works well with the workspace
- [x] Works well with DB Connect
- [x] Uses minimal stubs rather than boiler-plate-heavy examples

I'll have a followup with tests + DLT support.

---------

Co-authored-by: Andrew Nester <andrew.nester@databricks.com>
Co-authored-by: PaulCornellDB <paul.cornell@databricks.com>
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
arpitjasa-db pushed a commit to arpitjasa-db/cli that referenced this pull request Sep 7, 2023
## Changes

This pull request extends the templating support in preparation of a
new, default template (WIP, databricks#686):
* builtin templates that can be initialized using e.g. `databricks
bundle init default-python`
* builtin templates are embedded into the executable using go's `embed`
functionality, making sure they're co-versioned with the CLI
* new helpers to get the workspace name, current user name, etc. help
craft a complete template
* (not enabled yet) when the user types `databricks bundle init` they
can interactively select the `default-python` template

And makes two tangentially related changes:
* IsServicePrincipal now uses the "users" API rather than the
"principals" API, since the latter is too slow for our purposes.
* mode: prod no longer requires the 'target.prod.git' setting. It's hard
to set that from a template. (Pieter is planning an overhaul of warnings
support; this would be one of the first warnings we show.)

The actual `default-python` template is maintained in a separate PR:
databricks#686

## Tests
Unit tests, manual testing

Signed-off-by: Arpit Jasapara <arpit.jasapara@databricks.com>
arpitjasa-db pushed a commit to arpitjasa-db/cli that referenced this pull request Sep 7, 2023
## Changes
Add some cleanup based on @pietern's comments on
databricks#685

Signed-off-by: Arpit Jasapara <arpit.jasapara@databricks.com>
arpitjasa-db pushed a commit to arpitjasa-db/cli that referenced this pull request Sep 7, 2023
Bundles:
* Support cluster overrides with cluster_key and compute_key
([databricks#696](databricks#696)).
* Allow referencing local Python wheels without artifacts section
defined ([databricks#703](databricks#703)).
* Fixed --environment flag
([databricks#705](databricks#705)).
* Correctly identify local paths in libraries section
([databricks#702](databricks#702)).
* Fixed path joining in FindFilesWithSuffixInPath
([databricks#704](databricks#704)).
* Added transformation mutator for Python wheel task for them to work on
DBR <13.1 ([databricks#635](databricks#635)).

Internal:
* Add a foundation for built-in templates
([databricks#685](databricks#685)).
* Test transform when no Python wheel tasks defined
([databricks#714](databricks#714)).
* Pin Terraform binary version to 1.5.5
([databricks#715](databricks#715)).
* Cleanup after "Add a foundation for built-in templates"
([databricks#707](databricks#707)).
* Filter down to Python wheel tasks only for trampoline
([databricks#712](databricks#712)).
* Update Terraform provider schema structs from 1.23.0
([databricks#713](databricks#713)).

Signed-off-by: Arpit Jasapara <arpit.jasapara@databricks.com>
arpitjasa-db pushed a commit to arpitjasa-db/cli that referenced this pull request Sep 7, 2023
## Changes

This adds a built-in "default-python" template to the CLI. This is based
on the new default-template support of
databricks#685.

The goal here is to offer an experience where customers can simply type
`databricks bundle init` to get a default template:

```
$ databricks bundle init
Template to use [default-python]: default-python
Unique name for this project [my_project]: my_project
✨ Successfully initialized template
```

The present template:
- [x] Works well with VS Code
- [x] Works well with the workspace
- [x] Works well with DB Connect
- [x] Uses minimal stubs rather than boiler-plate-heavy examples

I'll have a followup with tests + DLT support.

---------

Co-authored-by: Andrew Nester <andrew.nester@databricks.com>
Co-authored-by: PaulCornellDB <paul.cornell@databricks.com>
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
Signed-off-by: Arpit Jasapara <arpit.jasapara@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants