[SPARK-51597][PYTHON][3.5] Fix deprecated/invalid config in setup.cfg
#50369
Conversation
Dash-separated keys for setuptools config have been deprecated in [v54.1.10 (2021)](https://setuptools.pypa.io/en/latest/history.html#id855)and removed in [v78](https://setuptools.pypa.io/en/latest/history.html#v78-0-0).
|
+1 our CI complained just now |
|
Same. |
|
+1 |
|
Please, any ETA on this? We have some pipelines that started to fail because this issue, more or less a couple of hours before this PR was open. Thanks a lot for taking care of this! |
|
There's precedent for this already in the 4.x branch: |
|
We are facing the same issue |
|
+1 |
|
I have open #50371 for this.. |
|
To be honest @bjornjorgensen, now that I am thinking about it, that field is not even the correct one... The right config according to the docs is https://setuptools.pypa.io/en/latest/userguide/declarative_config.html |
|
This was the warning that I get back in 2023 Usage of dash-separated 'description-file' will not be supported in future So I think 'description_file' will work. But you have to enable github actions on ours repo to make it run and test.. |
|
I you try to build it now it will fail because you have added a extra empty line -- linting :) 
|
|
Thanks @bjornjorgensen, I just enabled it. |
This may not give you a failure, but possibly will not do what is intended? (it might be the case |
There was a problem hiding this comment.
We're seeing the same in PyIceberg apache/iceberg-python#1838
|
oh wow |
|
Thanks for the fix. Would you mind creating a JIRA? https://issues.apache.org/jira/projects/SPARK/issues. Let's also fill the PR description |
|
@bjornjorgensen does it look good to you? |
I updated the PR description. Please feel free to close this PR if you have a different solution in the make, and/or mix and match the individual commits in a different PR. |
|
I will create one no worries |
HyukjinKwon
changed the title
setup.cfg (3.5)setup.cfg (3.5)
|
Looks fine but I will leave it to @bjornjorgensen to take a look / approve |
HyukjinKwon
changed the title
setup.cfg (3.5)setup.cfg
|
they are not the same https://stackoverflow.com/questions/60084128/does-description-file-in-setup-cfg-section-metadata-have-any-effect we have a jira for this. SPARK-45994 |
| @@ -19,4 +19,4 @@ | |||
| universal = 1 | |||
|
|
|||
| [metadata] | |||
| description-file = README.md | |||
| long_description = file: README.md | |||
There was a problem hiding this comment.
Change it to description_file
…file`
### What changes were proposed in this pull request?
Change `description-file` to `description_file`
### Why are the changes needed?
`./dev/make-distribution.sh --name custom-spark --pip -Pkubernetes > output.txt 2>&1`
in the file there is this
```
+ echo 'Building python distribution package'
Building python distribution package
+ pushd /home/bjorn/spark/python
+ rm -rf pyspark.egg-info
+ python3 setup.py sdist
/usr/lib/python3.11/site-packages/setuptools/dist.py:745: SetuptoolsDeprecationWarning: Invalid dash-separated options
!!
********************************************************************************
Usage of dash-separated 'description-file' will not be supported in future
versions. Please use the underscore name 'description_file' instead.
This deprecation is overdue, please update your project and remove deprecated
calls to avoid build errors in the future.
See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
********************************************************************************
!!
opt = self.warn_dash_deprecation(opt, section)
running sdist
running egg_info
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass GA
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #50369
Closes #50372
Closes #50371 from bjornjorgensen/bjornjorgensen-description_file3.5].
Authored-by: Bjørn Jørgensen <bjornjorgensen@gmail.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
|
thanks a lot for fixing this @HyukjinKwon @bjornjorgensen @abravalheri , our pipelines are getting green again :) |
Dash-separated keys for setuptools config have been deprecated in v54.1.10 (2021) and removed in v78.
Moreover this particular config is not supported by setuptools: #50369 (comment)
What changes were proposed in this pull request?
Replace invalid configuration field in
setup.cfgwith a valid oneWhy are the changes needed?
The previous configuration fails to build on
setuptools>=78.0.0,<78.0.2when support for dash-separated fields was removed after a long deprecation period. 78.0.2 postpones the deprecation for another year.Does this PR introduce any user-facing change?
No as far as I am aware.
How was this patch tested?
Relying on the CI for successful builds. (I am not a user of apache spark).
Was this patch authored or co-authored using generative AI tooling?
No.