Migration to `pyproject.toml` for packaging #4164

himkt · 2022-11-12T07:41:05Z

Motivation

This PR migrates from setup.py to pyproject.toml for the packaging configuration file. With this migration, we can easily retrieve the package metadata using TOML parser (it's tricky with setup.py since we need to parse a Python script). The ecosystem of packaging Python project with pyproject.toml is getting practical so I propose to start defining the package metadata with pyproject.toml.

Description of the changes

References

PEP: https://peps.python.org/pep-0621/#specification (+ https://peps.python.org/pep-0631)
reuires-python: https://stackoverflow.com/questions/72373093
package-data: https://stackoverflow.com/questions/69647590/
setuptools: https://setuptools.pypa.io/en/latest/setuptools.html#metadata + https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
setuptools 61.1.0: https://setuptools.pypa.io/en/stable/history.html
practice: https://qiita.com/ieiringoo/items/4bef4fc9975803b08671 (in Japanese)
- dynamic versioning

codecov-commenter · 2022-11-12T08:22:17Z

Codecov Report

Merging #4164 (fbe44e3) into master (b84022a) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #4164   +/-   ##
=======================================
  Coverage   90.43%   90.43%           
=======================================
  Files         172      172           
  Lines       13660    13660           
=======================================
  Hits        12354    12354           
  Misses       1306     1306

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

himkt · 2022-11-14T01:30:31Z

I got the feedback (from core-dev team) that we can go ahead for this transition, so let me remove RFC. I will revisit after some dependent PRs are merged.

c-bata

I understand it's WIP but left a single request change since I noticed while reviewing another PRs.

It's required to fix all actions that uses actions/cache to cache ~/.cache/pip since hashFiles('**/setup.py') is included in the cache key.
https://github.com/optuna/optuna/blob/master/.github/workflows/tests-storage.yml#L69-L77

himkt · 2022-11-18T06:27:48Z

I updated pyproject.toml based on the changed in setup.py.

It's required to fix all actions that uses actions/cache to cache ~/.cache/pip since hashFiles('**/setup.py') is included in the cache key.

Thank you, I just updated to use pyproject.toml to generate a cache key. 👀

c-bata · 2022-11-18T06:29:32Z

@HideakiImamura Could you review this PR?

c-bata

Left some comments while taking a look quickly. Also Could you remove setup.py?

pyproject.toml

c-bata · 2022-11-18T09:34:09Z

pyproject.toml

+    "Programming Language :: Python :: 3.7",
+    "Programming Language :: Python :: 3.8",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",


In setup.py, "Programming Language :: Python :: 3.11" is also listed.

https://github.com/optuna/optuna/blob/master/setup.py#L175

Co-authored-by: Masashi Shibata <c-bata@users.noreply.github.com>

himkt · 2022-11-18T14:31:01Z

Also Could you remove setup.py?

Oh, sorry. It restored in syncing the master branch... Removed.

himkt · 2022-11-24T10:01:36Z

@HideakiImamura @c-bata I merged the master branch and tests passed.
Could you please review the PR again? 🙏

himkt · 2023-01-06T12:32:07Z

@HideakiImamura @gen740
Sorry for the very late response. 💦 I updated the PR to make asv/pypi-publish work.
PTAL again, especially the package downloaded from (test) PyPI. 🙏

himkt · 2023-01-06T12:34:17Z

pyproject.toml

+[tool.setuptools.packages.find]
+include = ["optuna*"]


Describing package discovery in pyproject.toml is in the beta stage.
https://setuptools.pypa.io/en/latest/userguide/package_discovery.html

If we'd like to avoid the unstable API for building a package, setup.cfg maybe a better place for that.

It would be great to use this simple rule.

HideakiImamura

Thanks for the update and investigation! Almost, LGTM!

We have planned to release optuna v3.1.0 in next Monday. Just to be safe, I think this PR, which has a large impact around installation, should be merged after the 3.1.0 release.

HideakiImamura · 2023-01-10T02:52:03Z

pyproject.toml

+    "mlflow",
+    "pandas",
+    "pillow",
+    "plotly>=4.0.0",  # optuna/visualization.


This version is constrained as plotly>=4.9.0 in the master branch. https://github.com/optuna/optuna/blob/master/setup.py#L74

Thank you, updated in 2379003. 🙇

HideakiImamura · 2023-01-10T02:53:17Z

pyproject.toml

+optional = [
+    "matplotlib!=3.6.0",  # optuna/visualization/matplotlib
+    "pandas",  # optuna/study.py
+    "plotly>=4.0.0",  # optuna/visualization.


HideakiImamura · 2023-01-10T03:24:46Z

And, CI errors will be fixed in #4301.

himkt · 2023-01-10T10:47:54Z

Thank you for the careful consideration, I agree with you that merge the PR after the next release for safe. 👍

@HideakiImamura @gen740
BTW, I want you to check if you can install Optuna with editable manner.
Editable installation seemingly need pip>=21.3 and setuptools>=64 (but it looks I could do it with pip>=21.3 and setuptools==56). 🤔 I think these requirements can be acceptable to developers, however, any feedback is welcome.
🔗 https://stackoverflow.com/questions/69711606/how-to-install-a-package-using-pip-in-editable-mode-with-pyproject-toml

gen740 · 2023-01-12T01:33:29Z

pyproject.toml based project (PEP 621) has supported setuptools >= 61.0.0. So it must not be able to even build the project on setuptools=56. I suspect you are looking different setuptools version.

HideakiImamura

Thanks for the update. LGTM. Note that the CI will be fixed by merging the master.

HideakiImamura · 2023-01-18T07:02:53Z

Could you merge the master branch?

HideakiImamura · 2023-01-18T07:58:22Z

Note that setup.py has been updated, so you need to update pyproject.toml as well.

gen740

LGTM!

himkt · 2023-01-19T14:23:49Z

pyproject.toml based project (PEP 621) has supported setuptools >= 61.0.0. So it must not be able to even build the project on setuptools=56. I suspect you are looking different setuptools version.

Hmm..., it seemingly succeeded to install in editable manner with pip==22.0.4 and setuptools==47.1.0. Certainly I'm missing something but I don't have any idea. What am I misunderstanding?

https://gist.github.com/himkt/39a48eff132107b56e125312eb117485

Building the package is done in the isolated environment, which is why we can build the package with old pip/setuptools.

https://gist.github.com/himkt/f441968e9ca9406dabb7c25a5d1a7fe8

himkt · 2023-01-19T15:22:37Z

Merged the master branch (d104563) and fix dependency degradation (7398c7f). 🙏

HideakiImamura

LGTM.

I found a missed todo comment, but it can be added as a follow-up.

HideakiImamura · 2023-01-20T04:05:58Z

pyproject.toml

+    "plotly>=4.9.0",  # optuna/visualization.
+    "scikit-learn",
+    "scikit-optimize",
+    "sphinx<6",


Missed the todo comment above this line.

# TODO(not522): Remove the constraint after sphinx_rtd_theme supports Sphinx 6

🙇 fixed in f81457c.

himkt · 2023-01-25T07:42:21Z

a47d580 to reflect #4357.

Since I was not reviewer.

HideakiImamura · 2023-01-30T02:08:33Z

@himkt I'm so sorry for the delayed response, but could you merge the master to run the Checks CI?

himkt added 4 commits November 12, 2022 16:33

[wip] Add compatible pyproject.toml specification

5ab48cb

Add dependencies

66ba8ed

Remove setup.py

06ee054

Fix dockerbuild

8be2a9a

Subcommand

1419148

c-bata self-assigned this Nov 14, 2022

himkt changed the title ~~[RFC] Migration to pyproject.toml for packaging~~ Migration to pyproject.toml for packaging Nov 14, 2022

c-bata added the other Issue/PR that does not fit into any other label. label Nov 14, 2022

c-bata added this to the v3.1.0 milestone Nov 14, 2022

c-bata requested changes Nov 16, 2022

View reviewed changes

himkt added 5 commits November 18, 2022 15:15

Merge master

198aafb

:bye: cliff

38e793b

Update cmaes minimum version

11ac328

Apply dask change

e40e7cb

Use pyproject.toml to generate cache keys

9bcdfa1

himkt marked this pull request as ready for review November 18, 2022 06:27

c-bata assigned HideakiImamura Nov 18, 2022

Merge branch 'master' into pyproject.toml

e03174c

c-bata requested changes Nov 18, 2022

View reviewed changes

himkt and others added 3 commits November 18, 2022 23:27

Update pyproject.toml

4124b62

Co-authored-by: Masashi Shibata <c-bata@users.noreply.github.com>

Pull with --rebase

a2e800b

Add py311 to keywords

8f663dc

himkt added 2 commits November 18, 2022 23:32

Remove unnecessary double quotes

58f23ef

Fix conflict

a9296a2

himkt commented Jan 6, 2023

View reviewed changes

HideakiImamura reviewed Jan 10, 2023

View reviewed changes

Fix version constraint of plotly

2379003

HideakiImamura reviewed Jan 18, 2023

View reviewed changes

gen740 approved these changes Jan 19, 2023

View reviewed changes

Merge master

d104563

Follow the latest dependencies

7398c7f

himkt force-pushed the pyproject.toml branch from 2165d94 to 7398c7f Compare January 19, 2023 15:26

HideakiImamura approved these changes Jan 20, 2023

View reviewed changes

HideakiImamura added installation Installation and dependency. and removed other Issue/PR that does not fit into any other label. labels Jan 20, 2023

HideakiImamura added this to the v3.2.0 milestone Jan 20, 2023

himkt added 3 commits January 20, 2023 13:45

Restore comment

f81457c

Merge master

a47d580

Remove cached-path: 5fc23bb

fbe44e3

HideakiImamura enabled auto-merge January 30, 2023 02:07

Merge branch 'master' into pyproject.toml

6093e01

HideakiImamura merged commit d3872bd into optuna:master Jan 30, 2023

himkt deleted the pyproject.toml branch January 30, 2023 04:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migration to `pyproject.toml` for packaging #4164

Migration to `pyproject.toml` for packaging #4164

himkt commented Nov 12, 2022 •

edited

codecov-commenter commented Nov 12, 2022 •

edited

himkt commented Nov 14, 2022 •

edited

c-bata left a comment

himkt commented Nov 18, 2022

c-bata commented Nov 18, 2022

c-bata left a comment

c-bata Nov 18, 2022

himkt commented Nov 18, 2022

himkt commented Nov 24, 2022

himkt commented Jan 6, 2023

himkt Jan 6, 2023

HideakiImamura Jan 10, 2023

HideakiImamura left a comment

HideakiImamura Jan 10, 2023

himkt Jan 10, 2023

HideakiImamura Jan 10, 2023

HideakiImamura commented Jan 10, 2023

himkt commented Jan 10, 2023 •

edited

gen740 commented Jan 12, 2023 •

edited

HideakiImamura left a comment

HideakiImamura commented Jan 18, 2023

HideakiImamura commented Jan 18, 2023

gen740 left a comment

himkt commented Jan 19, 2023 •

edited

himkt commented Jan 19, 2023 •

edited

HideakiImamura left a comment

HideakiImamura Jan 20, 2023

himkt Jan 20, 2023

himkt commented Jan 25, 2023

HideakiImamura commented Jan 30, 2023

		[tool.setuptools.packages.find]
		include = ["optuna*"]

Migration to pyproject.toml for packaging #4164

Migration to pyproject.toml for packaging #4164

Conversation

himkt commented Nov 12, 2022 • edited

Motivation

Description of the changes

References

codecov-commenter commented Nov 12, 2022 • edited

Codecov Report

himkt commented Nov 14, 2022 • edited

c-bata left a comment

Choose a reason for hiding this comment

himkt commented Nov 18, 2022

c-bata commented Nov 18, 2022

c-bata left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

himkt commented Nov 18, 2022

himkt commented Nov 24, 2022

himkt commented Jan 6, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HideakiImamura left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HideakiImamura commented Jan 10, 2023

himkt commented Jan 10, 2023 • edited

gen740 commented Jan 12, 2023 • edited

HideakiImamura left a comment

Choose a reason for hiding this comment

HideakiImamura commented Jan 18, 2023

HideakiImamura commented Jan 18, 2023

gen740 left a comment

Choose a reason for hiding this comment

himkt commented Jan 19, 2023 • edited

himkt commented Jan 19, 2023 • edited

HideakiImamura left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

himkt commented Jan 25, 2023

HideakiImamura commented Jan 30, 2023

Migration to `pyproject.toml` for packaging #4164

Migration to `pyproject.toml` for packaging #4164

himkt commented Nov 12, 2022 •

edited

codecov-commenter commented Nov 12, 2022 •

edited

himkt commented Nov 14, 2022 •

edited

himkt commented Jan 10, 2023 •

edited

gen740 commented Jan 12, 2023 •

edited

himkt commented Jan 19, 2023 •

edited

himkt commented Jan 19, 2023 •

edited