Update namespace packages for PEP420 compatibility #3228

gradientsky · 2023-05-19T22:07:53Z

Description of changes:
Building the project generates deprecation warnings and suggests transitioning to the use of implicit namespace packages (refer to native namespace packages and PEP420):

../../venv-ag-310/lib/python3.10/site-packages/pkg_resources/__init__.py:2870
  /Users/user/Projects/venv-ag-310/lib/python3.10/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('autogluon')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

The action requires removing __init__.py from the root of the namespace package and switching to using find_namespace_packages instead of find_packages.

Pre-merge checklist:

builds passing
discussed with the team
verify nightly builds are working properly; rollback if any issues.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

yinweisu · 2023-05-22T22:48:28Z

core/src/autogluon/core/utils/version_utils.py

@@ -106,7 +84,7 @@ def show_versions():
    versions = _get_autogluon_versions()
    sorted_keys = sorted(versions.keys(), key=lambda x: x.lower())

-    maxlen = max(len(x) for x in versions)
+    maxlen = 0 if len(versions) == 0 else max(len(x) for x in versions)


I think previously the CI having error for max() having no argument is a sign that versions is an empty list, and I'm not sure if that's a correct behavior. I wonder does that mean version files are not correctly populated with the installation? Have you verified the installation locally?

I could not reproduce it across local, container and linux environments.

yinweisu · 2023-05-22T22:50:54Z

Another general comment is I'm not quite sure how would this affect pypi installation. Our CI is only testing for local installation and wheel building. We might need to merge this PR and do a test pypi release or wait for the nightly release and double check

Innixma · 2023-05-22T22:59:34Z

core/src/autogluon/core/utils/version_utils.py


+import autogluon


How are you importing autogluon?

Same way as before. It works without issues and auto-resolved by the package manager.

I see, interesting

Innixma · 2023-05-22T23:01:08Z

core/src/autogluon/core/utils/version_utils.py

-# We don't include test dependency here
-autogluon_extras_dict = {
-    'autogluon.core': ('all',),
-    'autogluon.common': (),
-    'autogluon.features': (),
-    'autogluon.timeseries': (),
-    'autogluon.tabular': ('all',),
-}
-
-# This is needed because some module are different in its import name and pip install name
-import_name_dict = {
-    'pillow': 'PIL',
-    'pytorch-lightning': 'pytorch_lightning',
-    'scikit-image': 'skimage',
-    'scikit-learn': 'sklearn',
-    'smart-open': 'smart_open',
-    'timm-clean': 'timm',
-}
-


Why no longer necessary?

The first dict - autogluon_extras_dict - the new code filters-out test dependencies via checking extras == 'test' using regexp.

The second part is not needed because we don't rely on file parsing anymore; packages and versions are retrieved via importlib.metadata.version and importlib.metadata.distribution. Thus no mismatch between pip dependency name and package import name.

Got it, thanks!

Innixma · 2023-05-22T23:02:30Z

core/src/autogluon/core/utils/version_utils.py

+    # Filter-out test dependencies
+    dependencies = [req for req in dependencies if not bool(re.search('extra.*test', req))]
+    # keep only package name
+    dependencies = [re.findall('[a-zA-Z0-9_\\-]+', req)[0].strip() for req in dependencies]


Have you compared the output of this function on mainline with the output in the PR? Ditto for other changes.

Yes, it fixed a few issues with missing dependencies:

main new

INSTALLED VERSIONS INSTALLED VERSIONS

------------------ ------------------

date : 2023-05-23 date : 2023-05-23

time : 00:49:04.037355 time : 00:58:15.831350

python : 3.9.16.final.0 python : 3.9.16.final.0

OS : Linux OS : Linux

OS-release : 5.15.0-1036-aws OS-release : 5.15.0-1036-aws

Version : #40~20.04.1-Ubuntu SMP... Version : #40~20.04.1-Ubuntu SMP ...

machine : x86_64 machine : x86_64

processor : x86_64 processor : x86_64

num_cores : 8 num_cores : 8

cpu_ram_mb : 31640 cpu_ram_mb : 31640

cuda version : None cuda version : None

num_gpus : 0 num_gpus : 0

gpu_ram_mb : [] gpu_ram_mb : []

avail_disk_size_mb : 905325 avail_disk_size_mb : 898628

accelerate : 0.16.0 accelerate : 0.16.0

autogluon : 0.7.1b20230523

autogluon.common : 0.7.1b20230523 autogluon.common : 0.7.1b20230523

autogluon.core : 0.7.1b20230523 autogluon.core : 0.7.1b20230523

autogluon.eda : 0.7.1b20230523 autogluon.eda : 0.7.1b20230523

autogluon.features : 0.7.1b20230523 autogluon.features : 0.7.1b20230523

autogluon.multimodal : 0.7.1b20230523 autogluon.multimodal : 0.7.1b20230523

autogluon.tabular : 0.7.1b20230523 autogluon.tabular : 0.7.1b20230523

autogluon.timeseries : 0.7.1b20230523 autogluon.timeseries : 0.7.1b20230523

boto3 : 1.26.120 boto3 : 1.26.120

catboost : 1.1.1 catboost : 1.1.1

defusedxml : 0.7.1 defusedxml : 0.7.1

evaluate : 0.3.0 evaluate : 0.3.0

fairscale : 0.4.13 fairscale : 0.4.13

fastai : 2.7.12 fastai : 2.7.12

gluonts : 0.12.8 gluonts : 0.12.8

hyperopt : 0.2.7 hyperopt : 0.2.7

imodels : 1.3.17

ipython : None ipython : 8.12.0

ipywidgets : 8.0.6 ipywidgets : 8.0.6

jinja2 : 3.1.2 jinja2 : 3.1.2

joblib : 1.2.0 joblib : 1.2.0

jsonschema : 4.17.3 jsonschema : 4.17.3

lightgbm : 3.3.5 lightgbm : 3.3.5

matplotlib : 3.6.3 matplotlib : 3.6.3

missingno : 0.5.2 missingno : 0.5.2

mlforecast : 0.7.2 mlforecast : 0.7.2

networkx : 2.8.8 networkx : 2.8.8

nlpaug : 1.1.11 nlpaug : 1.1.11

nltk : 3.8.1 nltk : 3.8.1

nptyping : 2.4.1 nptyping : 2.4.1

numpy : 1.23.5 numpy : 1.23.5

omegaconf : 2.2.3 omegaconf : 2.2.3

onnxruntime-gpu : 1.13.1

openmim : None openmim : 0.3.7

pandas : 1.5.3 pandas : 1.5.3

phik : 0.12.3 phik : 0.12.3

PIL : 9.5.0 Pillow : 9.5.0

psutil : 5.9.5 psutil : 5.9.5

pymupdf : None PyMuPDF : 1.21.1

pyod : None pyod : 1.0.9

pytesseract : 0.3.10 pytesseract : 0.3.10

pytorch_lightning : 1.9.5 pytorch-lightning : 1.9.5

pytorch-metric-learning: None pytorch-metric-learning: 1.7.3

ray : 2.3.1 ray : 2.3.1

requests : 2.28.2 requests : 2.28.2

scikit-image : 0.19.3

scikit-learn : 1.1.1

scikit-learn-intelex : None

scipy : 1.10.1 scipy : 1.10.1

seaborn : 0.12.2 seaborn : 0.12.2

sentencepiece : 0.1.99 sentencepiece : 0.1.99

seqeval : None seqeval : 1.2.2

setuptools : 65.6.3 setuptools : 65.6.3

shap : 0.41.0 shap : 0.41.0

skimage : 0.19.3

skl2onnx : 1.13

sklearn : 1.1.1 (see above scikit-learn)

statsforecast : 1.4.0 statsforecast : 1.4.0

statsmodels : 0.13.5 statsmodels : 0.13.5

suod : None suod : 0.0.8

tensorboard : 2.13.0 tensorboard : 2.13.0

text-unidecode : None text-unidecode : 1.3

timm : 0.6.13 timm : 0.6.13

torch : 1.13.1+cpu torch : 1.13.1+cpu

torchmetrics : 0.11.4 torchmetrics : 0.11.4

torchvision : 0.14.1+cpu torchvision : 0.14.1+cpu

tqdm : 4.65.0 tqdm : 4.65.0

transformers : 4.26.1 transformers : 4.26.1

ujson : 5.7.0 ujson : 5.7.0

vowpalwabbit : 9.4.0

xgboost : 1.7.5 xgboost : 1.7.5

yellowbrick : 1.5 yellowbrick : 1.5

LGTM, very nice.

Innixma · 2023-05-22T23:02:55Z

eda/setup.py

@@ -15,7 +15,7 @@
 ###########################

 version = ag.load_version_file()
-version = ag.update_version(version, use_file_if_exists=False, create_file=True)
+version = ag.update_version(version)


Why change?

That was a mistake. Version file should be created only in core

Innixma · 2023-05-22T23:04:27Z

core/src/autogluon/core/_setup_utils.py

-        packages=find_packages('src'),
+        packages=find_namespace_packages('src', include=['autogluon.*']),


how does the result differ?

From the user side there are no changes. find_namespace_packages is needed because there's no __init__.py and PEP asks to use this method instead of find_packages (it can't detect packages if there no init)

Gotcha, thanks

github-actions · 2023-05-22T23:19:05Z

Job PR-3228-dfc2ffa is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-3228/dfc2ffa/index.html

gradientsky · 2023-05-23T01:18:51Z

Another general comment is I'm not quite sure how would this affect pypi installation. Our CI is only testing for local installation and wheel building. We might need to merge this PR and do a test pypi release or wait for the nightly release and double check

Agreed. I had the same concerns; but I think nightly builds should be the way to check it.

…pkg_resources improved robustness of the code by using standard ways to retrieve versions instead of relying on parsing of __version__ and __VERSION__ files.

gradientsky · 2023-05-23T21:03:54Z

Rebased onto master

Innixma

LGTM!

github-actions · 2023-05-23T23:39:16Z

Job PR-3228-50f6868 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-3228/50f6868/index.html

gradientsky · 2023-05-26T23:18:26Z

Verified installation via pip install autogluon==0.7.1b20230526

gradientsky changed the title ~~Update namespace packages for PEP420 compatibility~~ [WIP] Update namespace packages for PEP420 compatibility May 22, 2023

gradientsky added this to the 0.8 Release milestone May 22, 2023

gradientsky added the code cleanup Fixing warnings/deprecations/syntax/etc. label May 22, 2023

gradientsky self-assigned this May 22, 2023

gradientsky force-pushed the 2023_05_19_namespace_packages branch 2 times, most recently from 22554c4 to dfc2ffa Compare May 22, 2023 20:47

gradientsky requested review from Innixma and yinweisu May 22, 2023 22:38

gradientsky changed the title ~~[WIP] Update namespace packages for PEP420 compatibility~~ Update namespace packages for PEP420 compatibility May 22, 2023

yinweisu reviewed May 22, 2023

View reviewed changes

Innixma reviewed May 22, 2023

View reviewed changes

gradientsky added 4 commits May 23, 2023 14:03

PEP420 compatibility patch

697727b

Updated _get_autogluon_versions

e0d115e

PEP420 compatibility patch

5ff5bf5

Updated show_version to use importlib.metadata instead of deprecated …

50f6868

…pkg_resources improved robustness of the code by using standard ways to retrieve versions instead of relying on parsing of __version__ and __VERSION__ files.

gradientsky force-pushed the 2023_05_19_namespace_packages branch from dfc2ffa to 50f6868 Compare May 23, 2023 21:03

Innixma approved these changes May 23, 2023

View reviewed changes

gradientsky merged commit c9b9828 into autogluon:master May 24, 2023
28 checks passed

gradientsky deleted the 2023_05_19_namespace_packages branch May 26, 2023 23:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update namespace packages for PEP420 compatibility #3228

Update namespace packages for PEP420 compatibility #3228

gradientsky commented May 19, 2023 •

edited

yinweisu May 22, 2023

gradientsky May 23, 2023

yinweisu commented May 22, 2023

Innixma May 22, 2023

gradientsky May 23, 2023

Innixma May 23, 2023

Innixma May 22, 2023

gradientsky May 23, 2023 •

edited

Innixma May 23, 2023

Innixma May 22, 2023

gradientsky May 23, 2023 •

edited

Innixma May 23, 2023

Innixma May 22, 2023

gradientsky May 23, 2023

Innixma May 22, 2023

gradientsky May 23, 2023

Innixma May 23, 2023

github-actions bot commented May 22, 2023

gradientsky commented May 23, 2023

gradientsky commented May 23, 2023

Innixma left a comment

github-actions bot commented May 23, 2023

gradientsky commented May 26, 2023

main	new
INSTALLED VERSIONS	INSTALLED VERSIONS
------------------	------------------
date : 2023-05-23	date : 2023-05-23
time : 00:49:04.037355	time : 00:58:15.831350
python : 3.9.16.final.0	python : 3.9.16.final.0
OS : Linux	OS : Linux
OS-release : 5.15.0-1036-aws	OS-release : 5.15.0-1036-aws
Version : #40~20.04.1-Ubuntu SMP...	Version : #40~20.04.1-Ubuntu SMP ...
machine : x86_64	machine : x86_64
processor : x86_64	processor : x86_64
num_cores : 8	num_cores : 8
cpu_ram_mb : 31640	cpu_ram_mb : 31640
cuda version : None	cuda version : None
num_gpus : 0	num_gpus : 0
gpu_ram_mb : []	gpu_ram_mb : []
avail_disk_size_mb : 905325	avail_disk_size_mb : 898628

accelerate : 0.16.0	accelerate : 0.16.0
	autogluon : 0.7.1b20230523
autogluon.common : 0.7.1b20230523	autogluon.common : 0.7.1b20230523
autogluon.core : 0.7.1b20230523	autogluon.core : 0.7.1b20230523
autogluon.eda : 0.7.1b20230523	autogluon.eda : 0.7.1b20230523
autogluon.features : 0.7.1b20230523	autogluon.features : 0.7.1b20230523
autogluon.multimodal : 0.7.1b20230523	autogluon.multimodal : 0.7.1b20230523
autogluon.tabular : 0.7.1b20230523	autogluon.tabular : 0.7.1b20230523
autogluon.timeseries : 0.7.1b20230523	autogluon.timeseries : 0.7.1b20230523
boto3 : 1.26.120	boto3 : 1.26.120
catboost : 1.1.1	catboost : 1.1.1
defusedxml : 0.7.1	defusedxml : 0.7.1
evaluate : 0.3.0	evaluate : 0.3.0
fairscale : 0.4.13	fairscale : 0.4.13
fastai : 2.7.12	fastai : 2.7.12
gluonts : 0.12.8	gluonts : 0.12.8
hyperopt : 0.2.7	hyperopt : 0.2.7
	imodels : 1.3.17
ipython : None	ipython : 8.12.0
ipywidgets : 8.0.6	ipywidgets : 8.0.6
jinja2 : 3.1.2	jinja2 : 3.1.2
joblib : 1.2.0	joblib : 1.2.0
jsonschema : 4.17.3	jsonschema : 4.17.3
lightgbm : 3.3.5	lightgbm : 3.3.5
matplotlib : 3.6.3	matplotlib : 3.6.3
missingno : 0.5.2	missingno : 0.5.2
mlforecast : 0.7.2	mlforecast : 0.7.2
networkx : 2.8.8	networkx : 2.8.8
nlpaug : 1.1.11	nlpaug : 1.1.11
nltk : 3.8.1	nltk : 3.8.1
nptyping : 2.4.1	nptyping : 2.4.1
numpy : 1.23.5	numpy : 1.23.5
omegaconf : 2.2.3	omegaconf : 2.2.3
	onnxruntime-gpu : 1.13.1
openmim : None	openmim : 0.3.7
pandas : 1.5.3	pandas : 1.5.3
phik : 0.12.3	phik : 0.12.3
PIL : 9.5.0	Pillow : 9.5.0
psutil : 5.9.5	psutil : 5.9.5
pymupdf : None	PyMuPDF : 1.21.1
pyod : None	pyod : 1.0.9
pytesseract : 0.3.10	pytesseract : 0.3.10
pytorch_lightning : 1.9.5	pytorch-lightning : 1.9.5
pytorch-metric-learning: None	pytorch-metric-learning: 1.7.3
ray : 2.3.1	ray : 2.3.1
requests : 2.28.2	requests : 2.28.2
	scikit-image : 0.19.3
	scikit-learn : 1.1.1
	scikit-learn-intelex : None
scipy : 1.10.1	scipy : 1.10.1
seaborn : 0.12.2	seaborn : 0.12.2
sentencepiece : 0.1.99	sentencepiece : 0.1.99
seqeval : None	seqeval : 1.2.2
setuptools : 65.6.3	setuptools : 65.6.3
shap : 0.41.0	shap : 0.41.0
skimage : 0.19.3
	skl2onnx : 1.13
sklearn : 1.1.1	(see above scikit-learn)
statsforecast : 1.4.0	statsforecast : 1.4.0
statsmodels : 0.13.5	statsmodels : 0.13.5
suod : None	suod : 0.0.8
tensorboard : 2.13.0	tensorboard : 2.13.0
text-unidecode : None	text-unidecode : 1.3
timm : 0.6.13	timm : 0.6.13
torch : 1.13.1+cpu	torch : 1.13.1+cpu
torchmetrics : 0.11.4	torchmetrics : 0.11.4
torchvision : 0.14.1+cpu	torchvision : 0.14.1+cpu
tqdm : 4.65.0	tqdm : 4.65.0
transformers : 4.26.1	transformers : 4.26.1
ujson : 5.7.0	ujson : 5.7.0
	vowpalwabbit : 9.4.0
xgboost : 1.7.5	xgboost : 1.7.5
yellowbrick : 1.5	yellowbrick : 1.5

		packages=find_packages('src'),
		packages=find_namespace_packages('src', include=['autogluon.*']),

Update namespace packages for PEP420 compatibility #3228

Update namespace packages for PEP420 compatibility #3228

Conversation

gradientsky commented May 19, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yinweisu commented May 22, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gradientsky May 23, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gradientsky May 23, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented May 22, 2023

gradientsky commented May 23, 2023

gradientsky commented May 23, 2023

Innixma left a comment

Choose a reason for hiding this comment

github-actions bot commented May 23, 2023

gradientsky commented May 26, 2023

gradientsky commented May 19, 2023 •

edited

gradientsky May 23, 2023 •

edited

gradientsky May 23, 2023 •

edited