Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Do not add wheel content hash in uploaded Python wheel path #1015

Merged
merged 2 commits into from
Nov 29, 2023
Merged

Do not add wheel content hash in uploaded Python wheel path #1015

merged 2 commits into from
Nov 29, 2023

Conversation

andrewnester
Copy link
Contributor

@andrewnester andrewnester commented Nov 28, 2023 •
edited
Loading

Changes

Removed hash from the upload path since it's not useful anyway.

The main reason for that change was to make it work on all-purpose clusters. But in order to make it work, wheel version needs to be increased anyway. So having only hash in path is useless.

Note: using --build-number (build tag) flag does not help with re-installing libraries on all-purpose clusters. The reason is that pip ignoring build tag when upgrading the library and only look at wheel version.
Build tag is only used for sorting the versions and the one with higher build tag takes priority when installed. It only works if no library is installed.
See https://github.com/pypa/pip/blob/a15dd75d98884c94a77d349b800c7c755d8c34e4/src/pip/_internal/index/package_finder.py#L522-L556
pypa/pip#4781

Thus, the only way to reinstall the library on all-purpose cluster is to increase wheel version manually or use automatic version generation, f.e.

setup(
  version=datetime.datetime.utcnow().strftime("%Y%m%d.%H%M%S"),
  ...
)

Tests

Integration tests passed.

@pietern
Copy link
Contributor

pietern commented Nov 28, 2023

@andrewnester Do you have a pointer to docs about this flag? Is it widely supported?

@andrewnester
Copy link
Contributor Author

@pietern the flag itself is specific for bdist_wheel command https://wheel.readthedocs.io/en/stable/reference/wheel_pack.html#cmdoption-build-number

But build tags are part of this PEP https://peps.python.org/pep-0427/#file-name-convention, so should be commonly supported

@andrewnester andrewnester changed the title Use --build-number when building Python wheels Do not add has in uploaded Python wheel path Nov 28, 2023
@andrewnester andrewnester changed the title Do not add has in uploaded Python wheel path Do not add wheel content hash in uploaded Python wheel path Nov 28, 2023
@andrewnester andrewnester added this pull request to the merge queue Nov 29, 2023
Merged via the queue into main with commit 5431174 Nov 29, 2023
4 checks passed
@andrewnester andrewnester deleted the py-build-tag branch November 29, 2023 10:46
@pietern pietern mentioned this pull request Nov 29, 2023
Merged
pietern added a commit that referenced this pull request Nov 29, 2023
@pietern
Release v0.210.0
e710c52 
This release includes the new `databricks labs` command to install, manage, and run Databricks Labs projects.

CLI:
 * Add `--debug` as shortcut for `--log-level debug` ([#964](#964)).
 * Improved usability of `databricks auth login ... --configure-cluster` ([#956](#956)).
 * Make `databricks configure` save only explicit fields ([#973](#973)).
 * Add `databricks labs` command group ([#914](#914)).
 * Tolerate missing .databrickscfg file during `databricks auth login` ([#1003](#1003)).
 * Add `--configure-cluster` flag to configure command ([#1005](#1005)).
 * Fix bug where the account or workspace client could be `nil` ([#1020](#1020)).

Bundles:
 * Do not allow empty descriptions for bundle template inputs ([#967](#967)).
 * Added support for top-level permissions ([#928](#928)).
 * Allow jobs to be manually unpaused in development mode ([#885](#885)).
 * Fix template initialization from current working directory ([#976](#976)).
 * Add `--tag` and `--branch` options to bundle init command ([#975](#975)).
 * Work around DLT issue with `$PYTHONPATH` not being set correctly ([#999](#999)).
 * Enable `spark_jar_task` with local JAR libraries ([#993](#993)).
 * Pass `USERPROFILE` environment variable to Terraform ([#1001](#1001)).
 * Improve error message when path is not a bundle template ([#985](#985)).
 * Correctly overwrite local state if remote state is newer ([#1008](#1008)).
 * Add mlops-stacks to the default `databricks bundle init` prompt ([#988](#988)).
 * Do not add wheel content hash in uploaded Python wheel path ([#1015](#1015)).
 * Do not replace pipeline libraries if there are no matches for pattern ([#1021](#1021)).

Internal:
 * Update CLI version in the VS Code extension during release ([#1014](#1014)).

API Changes:
 * Changed `databricks functions create` command . New request type is .
 * Changed `databricks metastores create` command with new required argument order.
 * Removed `databricks metastores enable-optimization` command.
 * Removed `databricks account o-auth-enrollment` command group.
 * Removed `databricks apps delete` command.
 * Removed `databricks apps get` command.
 * Added `databricks apps delete-app` command.
 * Added `databricks apps get-app` command.
 * Added `databricks apps get-app-deployment-status` command.
 * Added `databricks apps get-apps` command.
 * Added `databricks apps get-events` command.
 * Added `databricks account network-connectivity` command group.

OpenAPI commit 22f09783eb8a84d52026f856be3b2068f9498db3 (2023-11-23)

Dependency updates:
 * Bump golang.org/x/term from 0.13.0 to 0.14.0 ([#981](#981)).
 * Bump github.com/hashicorp/terraform-json from 0.17.1 to 0.18.0 ([#979](#979)).
 * Bump golang.org/x/oauth2 from 0.13.0 to 0.14.0 ([#982](#982)).
 * Bump github.com/databricks/databricks-sdk-go from 0.24.0 to 0.25.0 ([#980](#980)).
 * Bump github.com/databricks/databricks-sdk-go from 0.25.0 to 0.26.0 ([#1019](#1019)).
github-merge-queue bot pushed a commit that referenced this pull request Nov 29, 2023
@pietern
Release v0.210.0 (#1023)
09991da 
This release includes the new `databricks labs` command to install,
manage, and run Databricks Labs projects.

CLI:
* Add `--debug` as shortcut for `--log-level debug`
([#964](#964)).
* Improved usability of `databricks auth login ... --configure-cluster`
([#956](#956)).
* Make `databricks configure` save only explicit fields
([#973](#973)).
* Add `databricks labs` command group
([#914](#914)).
* Tolerate missing .databrickscfg file during `databricks auth login`
([#1003](#1003)).
* Add `--configure-cluster` flag to configure command
([#1005](#1005)).
* Fix bug where the account or workspace client could be `nil`
([#1020](#1020)).

Bundles:
* Do not allow empty descriptions for bundle template inputs
([#967](#967)).
* Added support for top-level permissions
([#928](#928)).
* Allow jobs to be manually unpaused in development mode
([#885](#885)).
* Fix template initialization from current working directory
([#976](#976)).
* Add `--tag` and `--branch` options to bundle init command
([#975](#975)).
* Work around DLT issue with `$PYTHONPATH` not being set correctly
([#999](#999)).
* Enable `spark_jar_task` with local JAR libraries
([#993](#993)).
* Pass `USERPROFILE` environment variable to Terraform
([#1001](#1001)).
* Improve error message when path is not a bundle template
([#985](#985)).
* Correctly overwrite local state if remote state is newer
([#1008](#1008)).
* Add mlops-stacks to the default `databricks bundle init` prompt
([#988](#988)).
* Do not add wheel content hash in uploaded Python wheel path
([#1015](#1015)).
* Do not replace pipeline libraries if there are no matches for pattern
([#1021](#1021)).

Internal:
* Update CLI version in the VS Code extension during release
([#1014](#1014)).

API Changes:
 * Changed `databricks functions create` command.
* Changed `databricks metastores create` command with new required
argument order.
 * Removed `databricks metastores enable-optimization` command.
 * Removed `databricks account o-auth-enrollment` command group.
 * Removed `databricks apps delete` command.
 * Removed `databricks apps get` command.
 * Added `databricks apps delete-app` command.
 * Added `databricks apps get-app` command.
 * Added `databricks apps get-app-deployment-status` command.
 * Added `databricks apps get-apps` command.
 * Added `databricks apps get-events` command.
 * Added `databricks account network-connectivity` command group.

OpenAPI commit 22f09783eb8a84d52026f856be3b2068f9498db3 (2023-11-23)

Dependency updates:
* Bump golang.org/x/term from 0.13.0 to 0.14.0
([#981](#981)).
* Bump github.com/hashicorp/terraform-json from 0.17.1 to 0.18.0
([#979](#979)).
* Bump golang.org/x/oauth2 from 0.13.0 to 0.14.0
([#982](#982)).
* Bump github.com/databricks/databricks-sdk-go from 0.24.0 to 0.25.0
([#980](#980)).
* Bump github.com/databricks/databricks-sdk-go from 0.25.0 to 0.26.0
([#1019](#1019)).
@andrewnester andrewnester mentioned this pull request Jul 26, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pietern pietern pietern approved these changes

@lennartkats-db lennartkats-db Awaiting requested review from lennartkats-db

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@andrewnester @pietern