Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make resource and artifact paths in bundle config relative to config folder #708

Merged
merged 13 commits into from
Sep 4, 2023

Conversation

andrewnester
Copy link
Contributor

Warning: breaking change

Changes

Instead of having paths in bundle config files be relative to bundle root even if the config file is nested, this PR makes such paths relative to the folder where the config is located.

When bundle is initialised, these paths will be transformed to relative paths based on bundle root. For example,
we have file structure like this

- mybundle
| - bundle.yml
| - subfolder
| -- resource.yml
| -- my.whl

Previously, we had to reference my.whl in resource.yml like this, which was confusing because resource.yml is in the same subfolder

sync:
  include:
    - ./subfolder/*.whl
...
tasks:
  - task_key: name
    libraries:
      - whl: ./subfolder/my.whl
...

After the change we can reference it like this (which is in line with the current behaviour for notebooks)

sync:
  include:
    - ./*.whl
...
tasks:
  - task_key: name
    libraries:
      - whl: ./my.whl
...

Tests

Existing translate_path_tests successfully passed after refactoring.

Added a couple of uses cases for Libraries paths.

Added a bundle config tests with include config and sync section

Copy link
Contributor

@pietern pietern left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two thoughts on the selector/transformer approach:

  • They are the same except for the directory relative to which the rewrite should happen. I think this doesn't warrant two separate structures. Perhaps the dir can be passed through to the functions that produce the selectors (and be made to return instances of a transformer)?
  • The rewriteFunc already does a filepath.Rel to determine how the path is relative to the given directory. The same is happening in translateToBundleRootRelativePath. Can these be consolidated? Right now we're mixing up what seems to be 1) making paths relative to the bundle root, and 2) rewriting them into remote paths. If we break this up into doing 1 first and 2 second, then I suspect things to be clearer.

bundle/config/artifact.go Outdated Show resolved Hide resolved
bundle/config/artifact.go Outdated Show resolved Hide resolved
bundle/config/mutator/translate_paths.go Outdated Show resolved Hide resolved
bundle/config/mutator/translate_paths.go Outdated Show resolved Hide resolved
bundle/config/mutator/translate_paths.go Outdated Show resolved Hide resolved
bundle/config/mutator/translate_paths.go Outdated Show resolved Hide resolved
@andrewnester
Copy link
Contributor Author

@pietern

  1. Agree on the selectors / transformers logic, simplified it and removed the selectors
  2. Slightly refactored rewriteFunc and now we have transformNoOp which won't do anything and just return local relative path, it should make the intentions clearer.

bundle/config/mutator/translate_paths.go Outdated Show resolved Hide resolved
bundle/config/mutator/translate_paths.go Outdated Show resolved Hide resolved
bundle/config/mutator/translate_paths.go Outdated Show resolved Hide resolved
bundle/config/mutator/translate_paths.go Outdated Show resolved Hide resolved
@pietern pietern changed the title [Breaking Change] Make resource and artifact paths in bundle config relative to config folder Make resource and artifact paths in bundle config relative to config folder Sep 4, 2023
@pietern pietern added this pull request to the merge queue Sep 4, 2023
Merged via the queue into main with commit 83443ba Sep 4, 2023
4 checks passed
@pietern pietern deleted the consistent-paths branch September 4, 2023 10:06
github-merge-queue bot pushed a commit that referenced this pull request Sep 5, 2023
~(this should be changed to target `main`)~

This reveals the template from
#686 in CLI prompts for once #686
and #708 are merged.

---------

Co-authored-by: Andrew Nester <andrew.nester@databricks.com>
Co-authored-by: PaulCornellDB <paul.cornell@databricks.com>
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
@pietern pietern mentioned this pull request Sep 6, 2023
pietern added a commit that referenced this pull request Sep 6, 2023
This release includes permission related commands for a subset of workspace
services where they apply. These complement the `permissions` command and
do not require specification of the object type to work with, as that is
implied by the command they are nested under.

CLI:
 * Group permission related commands ([#730](#730)).

Bundles:
 * Fixed artifact file uploading on Windows and wheel execution on DBR 13.3 ([#722](#722)).
 * Make resource and artifact paths in bundle config relative to config folder ([#708](#708)).
 * Add support for ordering of input prompts ([#662](#662)).
 * Fix IsServicePrincipal() only working for workspace admins ([#732](#732)).
 * databricks bundle init template v1 ([#686](#686)).
 * databricks bundle init template v2: optional stubs, DLT support ([#700](#700)).
 * Show 'databricks bundle init' template in CLI prompt ([#725](#725)).
 * Include $PATH in set of environment variables to pass along. ([#736](#736)).

Internal:
 * Update Go SDK to v0.19.0 ([#729](#729)).
 * Replace API call to test configuration with dummy authenticate call ([#728](#728)).

API Changes:
 * Changed `databricks account storage-credentials create` command to return .
 * Changed `databricks account storage-credentials get` command to return .
 * Changed `databricks account storage-credentials list` command to return .
 * Changed `databricks account storage-credentials update` command to return .
 * Changed `databricks connections create` command with new required argument order.
 * Changed `databricks connections update` command with new required argument order.
 * Changed `databricks volumes create` command with new required argument order.
 * Added `databricks artifact-allowlists` command group.
 * Added `databricks model-versions` command group.
 * Added `databricks registered-models` command group.
 * Added `databricks cluster-policies get-permission-levels` command.
 * Added `databricks cluster-policies get-permissions` command.
 * Added `databricks cluster-policies set-permissions` command.
 * Added `databricks cluster-policies update-permissions` command.
 * Added `databricks clusters get-permission-levels` command.
 * Added `databricks clusters get-permissions` command.
 * Added `databricks clusters set-permissions` command.
 * Added `databricks clusters update-permissions` command.
 * Added `databricks instance-pools get-permission-levels` command.
 * Added `databricks instance-pools get-permissions` command.
 * Added `databricks instance-pools set-permissions` command.
 * Added `databricks instance-pools update-permissions` command.
 * Added `databricks files` command group.
 * Changed `databricks permissions set` command to start returning .
 * Changed `databricks permissions update` command to start returning .
 * Added `databricks users get-permission-levels` command.
 * Added `databricks users get-permissions` command.
 * Added `databricks users set-permissions` command.
 * Added `databricks users update-permissions` command.
 * Added `databricks jobs get-permission-levels` command.
 * Added `databricks jobs get-permissions` command.
 * Added `databricks jobs set-permissions` command.
 * Added `databricks jobs update-permissions` command.
 * Changed `databricks experiments get-by-name` command to return .
 * Changed `databricks experiments get-experiment` command to return .
 * Added `databricks experiments delete-runs` command.
 * Added `databricks experiments get-permission-levels` command.
 * Added `databricks experiments get-permissions` command.
 * Added `databricks experiments restore-runs` command.
 * Added `databricks experiments set-permissions` command.
 * Added `databricks experiments update-permissions` command.
 * Added `databricks model-registry get-permission-levels` command.
 * Added `databricks model-registry get-permissions` command.
 * Added `databricks model-registry set-permissions` command.
 * Added `databricks model-registry update-permissions` command.
 * Added `databricks pipelines get-permission-levels` command.
 * Added `databricks pipelines get-permissions` command.
 * Added `databricks pipelines set-permissions` command.
 * Added `databricks pipelines update-permissions` command.
 * Added `databricks serving-endpoints get-permission-levels` command.
 * Added `databricks serving-endpoints get-permissions` command.
 * Added `databricks serving-endpoints set-permissions` command.
 * Added `databricks serving-endpoints update-permissions` command.
 * Added `databricks token-management get-permission-levels` command.
 * Added `databricks token-management get-permissions` command.
 * Added `databricks token-management set-permissions` command.
 * Added `databricks token-management update-permissions` command.
 * Changed `databricks dashboards create` command with new required argument order.
 * Added `databricks warehouses get-permission-levels` command.
 * Added `databricks warehouses get-permissions` command.
 * Added `databricks warehouses set-permissions` command.
 * Added `databricks warehouses update-permissions` command.
 * Added `databricks dashboard-widgets` command group.
 * Added `databricks query-visualizations` command group.
 * Added `databricks repos get-permission-levels` command.
 * Added `databricks repos get-permissions` command.
 * Added `databricks repos set-permissions` command.
 * Added `databricks repos update-permissions` command.
 * Added `databricks secrets get-secret` command.
 * Added `databricks workspace get-permission-levels` command.
 * Added `databricks workspace get-permissions` command.
 * Added `databricks workspace set-permissions` command.
 * Added `databricks workspace update-permissions` command.

OpenAPI commit 09a7fa63d9ae243e5407941f200960ca14d48b07 (2023-09-04)
github-merge-queue bot pushed a commit that referenced this pull request Sep 6, 2023
This release includes permission related commands for a subset of
workspace
services where they apply. These complement the `permissions` command
and
do not require specification of the object type to work with, as that is
implied by the command they are nested under.

CLI:
* Group permission related commands
([#730](#730)).

Bundles:
* Fixed artifact file uploading on Windows and wheel execution on DBR
13.3 ([#722](#722)).
* Make resource and artifact paths in bundle config relative to config
folder ([#708](#708)).
* Add support for ordering of input prompts
([#662](#662)).
* Fix IsServicePrincipal() only working for workspace admins
([#732](#732)).
* databricks bundle init template v1
([#686](#686)).
* databricks bundle init template v2: optional stubs, DLT support
([#700](#700)).
* Show 'databricks bundle init' template in CLI prompt
([#725](#725)).
* Include $PATH in set of environment variables to pass along.
([#736](#736)).

Internal:
* Update Go SDK to v0.19.0
([#729](#729)).
* Replace API call to test configuration with dummy authenticate call
([#728](#728)).

API Changes:
* Changed `databricks account storage-credentials create` command to
return .
* Changed `databricks account storage-credentials get` command to return
.
* Changed `databricks account storage-credentials list` command to
return .
* Changed `databricks account storage-credentials update` command to
return .
* Changed `databricks connections create` command with new required
argument order.
* Changed `databricks connections update` command with new required
argument order.
* Changed `databricks volumes create` command with new required argument
order.
 * Added `databricks artifact-allowlists` command group.
 * Added `databricks model-versions` command group.
 * Added `databricks registered-models` command group.
 * Added `databricks cluster-policies get-permission-levels` command.
 * Added `databricks cluster-policies get-permissions` command.
 * Added `databricks cluster-policies set-permissions` command.
 * Added `databricks cluster-policies update-permissions` command.
 * Added `databricks clusters get-permission-levels` command.
 * Added `databricks clusters get-permissions` command.
 * Added `databricks clusters set-permissions` command.
 * Added `databricks clusters update-permissions` command.
 * Added `databricks instance-pools get-permission-levels` command.
 * Added `databricks instance-pools get-permissions` command.
 * Added `databricks instance-pools set-permissions` command.
 * Added `databricks instance-pools update-permissions` command.
 * Added `databricks files` command group.
 * Changed `databricks permissions set` command to start returning .
 * Changed `databricks permissions update` command to start returning .
 * Added `databricks users get-permission-levels` command.
 * Added `databricks users get-permissions` command.
 * Added `databricks users set-permissions` command.
 * Added `databricks users update-permissions` command.
 * Added `databricks jobs get-permission-levels` command.
 * Added `databricks jobs get-permissions` command.
 * Added `databricks jobs set-permissions` command.
 * Added `databricks jobs update-permissions` command.
 * Changed `databricks experiments get-by-name` command to return .
 * Changed `databricks experiments get-experiment` command to return .
 * Added `databricks experiments delete-runs` command.
 * Added `databricks experiments get-permission-levels` command.
 * Added `databricks experiments get-permissions` command.
 * Added `databricks experiments restore-runs` command.
 * Added `databricks experiments set-permissions` command.
 * Added `databricks experiments update-permissions` command.
 * Added `databricks model-registry get-permission-levels` command.
 * Added `databricks model-registry get-permissions` command.
 * Added `databricks model-registry set-permissions` command.
 * Added `databricks model-registry update-permissions` command.
 * Added `databricks pipelines get-permission-levels` command.
 * Added `databricks pipelines get-permissions` command.
 * Added `databricks pipelines set-permissions` command.
 * Added `databricks pipelines update-permissions` command.
 * Added `databricks serving-endpoints get-permission-levels` command.
 * Added `databricks serving-endpoints get-permissions` command.
 * Added `databricks serving-endpoints set-permissions` command.
 * Added `databricks serving-endpoints update-permissions` command.
 * Added `databricks token-management get-permission-levels` command.
 * Added `databricks token-management get-permissions` command.
 * Added `databricks token-management set-permissions` command.
 * Added `databricks token-management update-permissions` command.
* Changed `databricks dashboards create` command with new required
argument order.
 * Added `databricks warehouses get-permission-levels` command.
 * Added `databricks warehouses get-permissions` command.
 * Added `databricks warehouses set-permissions` command.
 * Added `databricks warehouses update-permissions` command.
 * Added `databricks dashboard-widgets` command group.
 * Added `databricks query-visualizations` command group.
 * Added `databricks repos get-permission-levels` command.
 * Added `databricks repos get-permissions` command.
 * Added `databricks repos set-permissions` command.
 * Added `databricks repos update-permissions` command.
 * Added `databricks secrets get-secret` command.
 * Added `databricks workspace get-permission-levels` command.
 * Added `databricks workspace get-permissions` command.
 * Added `databricks workspace set-permissions` command.
 * Added `databricks workspace update-permissions` command.

OpenAPI commit 09a7fa63d9ae243e5407941f200960ca14d48b07 (2023-09-04)
arpitjasa-db pushed a commit to arpitjasa-db/cli that referenced this pull request Sep 7, 2023
…folder (databricks#708)

# Warning: breaking change

## Changes
Instead of having paths in bundle config files be relative to bundle
root even if the config file is nested, this PR makes such paths
relative to the folder where the config is located.

When bundle is initialised, these paths will be transformed to relative
paths based on bundle root. For example,
we have file structure like this
```
- mybundle
| - bundle.yml
| - subfolder
| -- resource.yml
| -- my.whl
```

Previously, we had to reference `my.whl` in resource.yml like this,
which was confusing because resource.yml is in the same subfolder
```
sync:
  include:
    - ./subfolder/*.whl
...
tasks:
  - task_key: name
    libraries:
      - whl: ./subfolder/my.whl
...
```

After the change we can reference it like this (which is in line with
the current behaviour for notebooks)

```
sync:
  include:
    - ./*.whl
...
tasks:
  - task_key: name
    libraries:
      - whl: ./my.whl
...
```

## Tests
Existing `translate_path_tests` successfully passed after refactoring.

Added a couple of uses cases for `Libraries` paths.

Added a bundle config tests with include config and sync section

---------

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
Signed-off-by: Arpit Jasapara <arpit.jasapara@databricks.com>
arpitjasa-db pushed a commit to arpitjasa-db/cli that referenced this pull request Sep 7, 2023
~(this should be changed to target `main`)~

This reveals the template from
databricks#686 in CLI prompts for once databricks#686
and databricks#708 are merged.

---------

Co-authored-by: Andrew Nester <andrew.nester@databricks.com>
Co-authored-by: PaulCornellDB <paul.cornell@databricks.com>
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
Signed-off-by: Arpit Jasapara <arpit.jasapara@databricks.com>
arpitjasa-db pushed a commit to arpitjasa-db/cli that referenced this pull request Sep 7, 2023
This release includes permission related commands for a subset of
workspace
services where they apply. These complement the `permissions` command
and
do not require specification of the object type to work with, as that is
implied by the command they are nested under.

CLI:
* Group permission related commands
([databricks#730](databricks#730)).

Bundles:
* Fixed artifact file uploading on Windows and wheel execution on DBR
13.3 ([databricks#722](databricks#722)).
* Make resource and artifact paths in bundle config relative to config
folder ([databricks#708](databricks#708)).
* Add support for ordering of input prompts
([databricks#662](databricks#662)).
* Fix IsServicePrincipal() only working for workspace admins
([databricks#732](databricks#732)).
* databricks bundle init template v1
([databricks#686](databricks#686)).
* databricks bundle init template v2: optional stubs, DLT support
([databricks#700](databricks#700)).
* Show 'databricks bundle init' template in CLI prompt
([databricks#725](databricks#725)).
* Include $PATH in set of environment variables to pass along.
([databricks#736](databricks#736)).

Internal:
* Update Go SDK to v0.19.0
([databricks#729](databricks#729)).
* Replace API call to test configuration with dummy authenticate call
([databricks#728](databricks#728)).

API Changes:
* Changed `databricks account storage-credentials create` command to
return .
* Changed `databricks account storage-credentials get` command to return
.
* Changed `databricks account storage-credentials list` command to
return .
* Changed `databricks account storage-credentials update` command to
return .
* Changed `databricks connections create` command with new required
argument order.
* Changed `databricks connections update` command with new required
argument order.
* Changed `databricks volumes create` command with new required argument
order.
 * Added `databricks artifact-allowlists` command group.
 * Added `databricks model-versions` command group.
 * Added `databricks registered-models` command group.
 * Added `databricks cluster-policies get-permission-levels` command.
 * Added `databricks cluster-policies get-permissions` command.
 * Added `databricks cluster-policies set-permissions` command.
 * Added `databricks cluster-policies update-permissions` command.
 * Added `databricks clusters get-permission-levels` command.
 * Added `databricks clusters get-permissions` command.
 * Added `databricks clusters set-permissions` command.
 * Added `databricks clusters update-permissions` command.
 * Added `databricks instance-pools get-permission-levels` command.
 * Added `databricks instance-pools get-permissions` command.
 * Added `databricks instance-pools set-permissions` command.
 * Added `databricks instance-pools update-permissions` command.
 * Added `databricks files` command group.
 * Changed `databricks permissions set` command to start returning .
 * Changed `databricks permissions update` command to start returning .
 * Added `databricks users get-permission-levels` command.
 * Added `databricks users get-permissions` command.
 * Added `databricks users set-permissions` command.
 * Added `databricks users update-permissions` command.
 * Added `databricks jobs get-permission-levels` command.
 * Added `databricks jobs get-permissions` command.
 * Added `databricks jobs set-permissions` command.
 * Added `databricks jobs update-permissions` command.
 * Changed `databricks experiments get-by-name` command to return .
 * Changed `databricks experiments get-experiment` command to return .
 * Added `databricks experiments delete-runs` command.
 * Added `databricks experiments get-permission-levels` command.
 * Added `databricks experiments get-permissions` command.
 * Added `databricks experiments restore-runs` command.
 * Added `databricks experiments set-permissions` command.
 * Added `databricks experiments update-permissions` command.
 * Added `databricks model-registry get-permission-levels` command.
 * Added `databricks model-registry get-permissions` command.
 * Added `databricks model-registry set-permissions` command.
 * Added `databricks model-registry update-permissions` command.
 * Added `databricks pipelines get-permission-levels` command.
 * Added `databricks pipelines get-permissions` command.
 * Added `databricks pipelines set-permissions` command.
 * Added `databricks pipelines update-permissions` command.
 * Added `databricks serving-endpoints get-permission-levels` command.
 * Added `databricks serving-endpoints get-permissions` command.
 * Added `databricks serving-endpoints set-permissions` command.
 * Added `databricks serving-endpoints update-permissions` command.
 * Added `databricks token-management get-permission-levels` command.
 * Added `databricks token-management get-permissions` command.
 * Added `databricks token-management set-permissions` command.
 * Added `databricks token-management update-permissions` command.
* Changed `databricks dashboards create` command with new required
argument order.
 * Added `databricks warehouses get-permission-levels` command.
 * Added `databricks warehouses get-permissions` command.
 * Added `databricks warehouses set-permissions` command.
 * Added `databricks warehouses update-permissions` command.
 * Added `databricks dashboard-widgets` command group.
 * Added `databricks query-visualizations` command group.
 * Added `databricks repos get-permission-levels` command.
 * Added `databricks repos get-permissions` command.
 * Added `databricks repos set-permissions` command.
 * Added `databricks repos update-permissions` command.
 * Added `databricks secrets get-secret` command.
 * Added `databricks workspace get-permission-levels` command.
 * Added `databricks workspace get-permissions` command.
 * Added `databricks workspace set-permissions` command.
 * Added `databricks workspace update-permissions` command.

OpenAPI commit 09a7fa63d9ae243e5407941f200960ca14d48b07 (2023-09-04)

Signed-off-by: Arpit Jasapara <arpit.jasapara@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants