Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Auto detect Python wheel packages and infer build command #603

Merged
merged 14 commits into from
Jul 26, 2023
Merged

Auto detect Python wheel packages and infer build command #603

merged 14 commits into from
Jul 26, 2023

Conversation

andrewnester
Copy link
Contributor

Changes

Auto detect Python wheel packages and infer build command

Tests

bundle.yml

bundle:
  name: wheel-task

workspace:
  host: ****

resources:
  jobs:
    test_job:
      name: "[${bundle.environment}] My Wheel Job"
      tasks:
        - task_key: TestTask
          existing_cluster_id: "0123-aaaaaa-bbbbbbbbb"
          python_wheel_task:
            package_name: "my_test_code"
            entry_point: "run"
          libraries:
          - whl: ./dist/my_test_code-*.whl


andrew.nester@HFW9Y94129 wheel % cli bundle deploy
artifacts.whl.AutoDetect: Detecting Python wheel project...
artifacts.whl.AutoDetect: Found Python wheel project at /Users/andrew.nester/dabs/wheel
artifacts.whl.Build(my_test_code): Building...
artifacts.whl.Build(my_test_code): Build succeeded
Starting upload of bundle files
Uploaded bundle files at /Users/andrew.nester@databricks.com/.bundle/wheel-task/default/files!

artifacts.Upload(my_test_code-0.0.0a0-py3-none-any.whl): Uploading...
artifacts.Upload(my_test_code-0.0.0a0-py3-none-any.whl): Upload succeeded
Starting resource deployment
Resource deployment completed!

pietern
pietern reviewed Jul 25, 2023
View reviewed changes
Copy link
Contributor

@pietern pietern left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Do you want to hold until the other PR is merged?

bundle/artifacts/autodetect.go Show resolved Hide resolved
bundle/artifacts/whl/autodetect.go Outdated Show resolved Hide resolved
@andrewnester
Copy link
Contributor Author

@pietern yeah, let's wait for the other PR to be merged first

@andrewnester andrewnester changed the base branch from artifact-block to main July 25, 2023 11:35
pietern
pietern approved these changes Jul 25, 2023
View reviewed changes
Copy link
Contributor

@pietern pietern left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a few unit tests before merging.

bundle/artifacts/infer.go

if len(artifact.Files) > 0 || artifact.BuildCommand != "" {
return nil
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a comment here -- I had to scroll back and forth a couple times before figuring out that we never build an artifact if the files are explicitly listed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually, we should not build only if there is no BuildCommand

bundle/artifacts/whl/autodetect.go Show resolved Hide resolved
@andrewnester andrewnester enabled auto-merge (squash) July 26, 2023 10:03
@andrewnester andrewnester merged commit cfff140 into main Jul 26, 2023
4 checks passed
@mgyucht mgyucht mentioned this pull request Jul 27, 2023
Merged
github-merge-queue bot pushed a commit that referenced this pull request Jul 27, 2023
@mgyucht
Release v0.202.0 (#619)
3697dfc 
Breaking Change:
* Require include glob patterns to be explicitly defined
([#602](#602)).

Bundles:
* Add support for more SDK config options
([#587](#587)).
* Add template renderer for Databricks templates
([#589](#589)).
* Fix formatting in renderer.go
([#593](#593)).
* Fixed python wheel test
([#608](#608)).
* Auto detect Python wheel packages and infer build command
([#603](#603)).
* Added support for artifacts building for bundles
([#583](#583)).
* Add support for cloning repositories
([#544](#544)).
* Add regexp compile helper function for templates
([#601](#601)).
* Add unit test that raw strings are printed as is
([#599](#599)).

Internal:
* Fix tests under ./cmd/configure if DATABRICKS_TOKEN is set
([#605](#605)).
* Remove dependency on global state in generated commands
([#595](#595)).
* Remove dependency on global state for the root command
([#606](#606)).
* Add merge_group trigger for build
([#612](#612)).
* Added support for build command chaining and error on missing wheel
([#607](#607)).
* Add TestAcc prefix to filer test and fix any failing tests
([#611](#611)).
* Add url parse helper function for templates
([#600](#600)).
* Remove dependency on global state for remaining commands
([#613](#613)).
* Update CHANGELOG template
([#588](#588)).
@PaulMaksud
Copy link

PaulMaksud commented Aug 16, 2023 •
edited
Loading

Hello on version Databricks CLI v0.203.1 on windows intel 64 i face this error :
$ databricks bundle deploy -e dev artifacts.whl.AutoDetect: Detecting Python wheel project... artifacts.whl.AutoDetect: Found Python wheel project at C:\Users\Sesa653627\Desktop\ml integ\pilot-databricks artifacts.whl.Build(artifact1692195769): Building... Error: artifacts.whl.Build(artifact1692195769): Failed exit status 9009, output: Python est introuvable. Ex�cutez sans argument pour proc�der � l
even though i have no python wheel task declared in my bundle.yml, however i do have a setup.py file at root level and a pyproject.toml file

@pietern
Copy link
Contributor

pietern commented Aug 16, 2023

@PaulMaksud Thanks for reporting.

Could you create a new issue for this, and include the (redacted) contents of your setup.py file?

@pietern pietern deleted the auto-detect branch August 16, 2023 14:32
@jusshen jusshen mentioned this pull request Aug 23, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pietern pietern pietern approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@andrewnester @PaulMaksud @pietern