Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added support for glob patterns in pipeline libraries section #833

Merged
merged 5 commits into from
Oct 4, 2023
Merged

Added support for glob patterns in pipeline libraries section #833

merged 5 commits into from
Oct 4, 2023

Conversation

andrewnester
Copy link
Contributor

Changes

Now it's possible to specify glob pattern in pipeline libraries section and DAB will add all matched files as libraries

  pipelines:
    dummy:
      name: " DLT with Python files"
      target: "dlt_python_files"
      libraries:
        - file:
            path: ./*.py

Tests

Added unit test

pietern
pietern reviewed Oct 3, 2023
View reviewed changes
bundle/config/mutator/expand_glob_paths.go Outdated Show resolved Hide resolved
bundle/config/mutator/expand_glob_paths.go Outdated Show resolved Hide resolved
bundle/config/mutator/expand_glob_paths.go Outdated Show resolved Hide resolved
bundle/config/mutator/expand_glob_paths.go Outdated Show resolved Hide resolved
bundle/config/mutator/expand_glob_paths_test.go Outdated Show resolved Hide resolved
pietern
pietern reviewed Oct 4, 2023
View reviewed changes
bundle/config/mutator/expand_glob_paths.go Outdated Show resolved Hide resolved
bundle/config/mutator/expand_glob_paths.go Outdated Show resolved Hide resolved
bundle/config/mutator/expand_glob_paths_test.go Outdated Show resolved Hide resolved
pietern
pietern approved these changes Oct 4, 2023
View reviewed changes
Copy link
Contributor

@pietern pietern left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One remaining minor issue.

bundle/config/mutator/expand_pipeline_glob_paths.go Outdated Show resolved Hide resolved
@andrewnester andrewnester added this pull request to the merge queue Oct 4, 2023
Merged via the queue into main with commit aa54a86 Oct 4, 2023
4 checks passed
@andrewnester andrewnester deleted the glob-pipeline-main branch October 4, 2023 13:28
@andrewnester andrewnester mentioned this pull request Oct 4, 2023
Merged
andrewnester added a commit that referenced this pull request Oct 4, 2023
@andrewnester
Release v0.207.0
bfda2fa 
CLI:
 * Refactor change computation for sync ([#785](#785)).

Bundles:
 * Allow digits in the generated short name ([#820](#820)).
 * Emit an error when incompatible all purpose cluster used with Python wheel tasks ([#823](#823)).
 * Use normalized short name for tag value in development mode ([#821](#821)).
 * Added `python.DetectInterpreters` and other utils ([#805](#805)).
 * Mark artifacts properties as optional ([#834](#834)).
 * Added support for glob patterns in pipeline libraries section ([#833](#833)).

Internal:
 * Run tests to verify backend tag validation behavior ([#814](#814)).
 * Library to validate and normalize cloud specific tags ([#819](#819)).
 * Added test to submit and run various Python tasks on multiple DBR versions ([#806](#806)).
 * Create a release PR in setup-cli repo on tag push ([#827](#827)).

API Changes:
 * Changed `databricks account metastore-assignments list` command to return .
 * Changed `databricks jobs cancel-all-runs` command with new required argument order.
 * Added `databricks account o-auth-published-apps` command group.
 * Changed `databricks serving-endpoints query` command . New request type is .
 * Added `databricks serving-endpoints patch` command.
 * Added `databricks credentials-manager` command group.
 * Added `databricks settings` command group.
 * Changed `databricks clean-rooms list` command to require request of .
 * Changed `databricks statement-execution execute-statement` command with new required argument order.

OpenAPI commit bcbf6e851e3d82fd910940910dd31c10c059746c (2023-10-02)
Dependency updates:
 * Bump github.com/google/uuid from 1.3.0 to 1.3.1 ([#825](#825)).
 * Updated Go SDK to 0.22.0 ([#831](#831)).
github-merge-queue bot pushed a commit that referenced this pull request Oct 4, 2023
@andrewnester
Release v0.207.0 (#839)
2ce4f74 
CLI:
* Refactor change computation for sync
([#785](#785)).

Bundles:
* Allow digits in the generated short name
([#820](#820)).
* Emit an error when incompatible all purpose cluster used with Python
wheel tasks ([#823](#823)).
* Use normalized short name for tag value in development mode
([#821](#821)).
* Added `python.DetectInterpreters` and other utils
([#805](#805)).
* Mark artifacts properties as optional
([#834](#834)).
* Added support for glob patterns in pipeline libraries section
([#833](#833)).

Internal:
* Run tests to verify backend tag validation behavior
([#814](#814)).
* Library to validate and normalize cloud specific tags
([#819](#819)).
* Added test to submit and run various Python tasks on multiple DBR
versions ([#806](#806)).
* Create a release PR in setup-cli repo on tag push
([#827](#827)).

API Changes:
* Changed `databricks account metastore-assignments list` command to
return .
* Changed `databricks jobs cancel-all-runs` command with new required
argument order.
 * Added `databricks account o-auth-published-apps` command group.
* Changed `databricks serving-endpoints query` command . New request
type is .
 * Added `databricks serving-endpoints patch` command.
 * Added `databricks credentials-manager` command group.
 * Added `databricks settings` command group.
 * Changed `databricks clean-rooms list` command to require request of .
* Changed `databricks statement-execution execute-statement` command
with new required argument order.

OpenAPI commit bcbf6e851e3d82fd910940910dd31c10c059746c (2023-10-02)
Dependency updates:
* Bump github.com/google/uuid from 1.3.0 to 1.3.1
([#825](#825)).
* Updated Go SDK to 0.22.0
([#831](#831)).
hectorcast-db pushed a commit that referenced this pull request Oct 13, 2023
@andrewnester @hectorcast-db
Added support for glob patterns in pipeline libraries section (#833)
c4f4717 
## Changes
Now it's possible to specify glob pattern in pipeline libraries section
and DAB will add all matched files as libraries

```
  pipelines:
    dummy:
      name: " DLT with Python files"
      target: "dlt_python_files"
      libraries:
        - file:
            path: ./*.py
```

## Tests
Added unit test
hectorcast-db pushed a commit that referenced this pull request Oct 13, 2023
@andrewnester @hectorcast-db
Release v0.207.0 (#839)
641a8db 
CLI:
* Refactor change computation for sync
([#785](#785)).

Bundles:
* Allow digits in the generated short name
([#820](#820)).
* Emit an error when incompatible all purpose cluster used with Python
wheel tasks ([#823](#823)).
* Use normalized short name for tag value in development mode
([#821](#821)).
* Added `python.DetectInterpreters` and other utils
([#805](#805)).
* Mark artifacts properties as optional
([#834](#834)).
* Added support for glob patterns in pipeline libraries section
([#833](#833)).

Internal:
* Run tests to verify backend tag validation behavior
([#814](#814)).
* Library to validate and normalize cloud specific tags
([#819](#819)).
* Added test to submit and run various Python tasks on multiple DBR
versions ([#806](#806)).
* Create a release PR in setup-cli repo on tag push
([#827](#827)).

API Changes:
* Changed `databricks account metastore-assignments list` command to
return .
* Changed `databricks jobs cancel-all-runs` command with new required
argument order.
 * Added `databricks account o-auth-published-apps` command group.
* Changed `databricks serving-endpoints query` command . New request
type is .
 * Added `databricks serving-endpoints patch` command.
 * Added `databricks credentials-manager` command group.
 * Added `databricks settings` command group.
 * Changed `databricks clean-rooms list` command to require request of .
* Changed `databricks statement-execution execute-statement` command
with new required argument order.

OpenAPI commit bcbf6e851e3d82fd910940910dd31c10c059746c (2023-10-02)
Dependency updates:
* Bump github.com/google/uuid from 1.3.0 to 1.3.1
([#825](#825)).
* Updated Go SDK to 0.22.0
([#831](#831)).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pietern pietern pietern approved these changes

@shreyas-goenka shreyas-goenka Awaiting requested review from shreyas-goenka

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@andrewnester @pietern