New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add py.typed
Marker
#2904
Add py.typed
Marker
#2904
Conversation
This will allow static type checkers to recognized the package's type hints per PEP-561 Fixes issue pyvista#2903
@pyvista/developers Would you please review and approve this PR? |
@akaszynski Can you confirm you get pinged when I enter @pyvista/developers ? I'm not sure it's working for me. |
Codecov Report
@@ Coverage Diff @@
## main #2904 +/- ##
==========================================
- Coverage 94.11% 94.06% -0.05%
==========================================
Files 76 76
Lines 16521 16525 +4
==========================================
- Hits 15549 15545 -4
- Misses 972 980 +8 |
@tkoyama010 Just confirming, did you receive a ping about this PR? I'm not sure if the pyvista/devlopers option is working for me. |
@adam-grant-hendry I think it doesn't work with your permission. But if you post the Pull Request, most of them will review it. |
Thanks for confirming. Would I be able to get permission somehow? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not very familiar with this. But do we need to include this in the package_data
in setup.py as well?
It belongs to the @akaszynski . |
Ah no problem then. 👍 |
Oh yes, you are right. Good catch! Per PEP 561:
I'll update the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we use annotations from PyVista itself, or do we add stub files?
We place them in the x: int = 1 If we used stubs, they would be files with the |
As a follow-up, stubs are typically used instead of annotations directly in the source code for two reasons:
(I believe I asked, or was going to ask, @akaszynski which method he would prefer, but I saw the annotations were already in the source and we already type check with
so stubs can be used to patch broken stubs from packages:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Thanks for the explanation!
You bet! Thanks for asking and thanks for the approval! |
Let it be known that type annotations is still a work in progress and we don't have it in most places. @adam-grant-hendry, if you feel up to it, you can feel free to add them. I'd do it slowly across several PRs. Tracking down |
😄 It does for me too! Yes, I am happy to track them down. For better or worse, type hints appear here to stay in Python, and I think it will help us in the long run as the package grows. Can we approve this PR though?
Completely fine that is WIP and happy to implement slowly over multiple PRs. Shouldn't break any runtime functionality of |
@akaszynski @adeak @tkoyama010 Have we achieved consensus yet on adding the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've never heard of py.typed, but I don't really know typing, and the primer from mypy you linked in #2903 seems clear as day: mypy needs this, and it can't cause any harm.
One question though: the same page also has a note:
If you use setuptools, you must pass the option zip_safe=False to setup(), or mypy will not be able to find the installed package.
We use setuptools and we don't have zip_safe=False
. Do we need this too, and what else would passing this kwarg change?
It also says
Since we have an sdist on PyPI we probably have to do this too. |
Let's add this. |
Great question! First, we really should start following PEP 518 and specify that our build backend is [build-system]
requires = [
"setuptools",
"wheel",
"toml",
]
build-backend = "setuptools.build_meta" The original purpose of the In addition, thanks to
and put them in the It seems we have (rather innocently) used the
Here's how we could put all the configurations I mentioned into our pyproject.toml[project]
# PEP 621 project metadata
# See https://www.python.org/dev/peps/pep-0621/
authors = [
{ name = "PyVista Developers", email: "info@pyvista.org" },
]
# Alternative to below, if we don't want the `pyproject.toml` to keep the
# version single source of truth and instead keep it in the `_version.py`
# file, we can remove the bottom two entries and use
#
# [tool.setuptools.dynamic]
# version = { attr = "pyvista._version.__version___" }
# readme = { file = ["README.rst"] }
version = "v0.34.2"
readme = "README.rst"
requires-python = ">=3.7"
license = { text = "MIT" }
name = "pyvista"
description = "Easier Pythonic interface to VTK"
classifiers = [
'Development Status :: 4 - Beta',
'Intended Audience :: Science/Research',
'Topic :: Scientific/Engineering :: Information Analysis',
'License :: OSI Approved :: MIT License',
'Operating System :: Microsoft :: Windows',
'Operating System :: POSIX',
'Operating System :: MacOS',
'Programming Language :: Python :: 3.7',
'Programming Language :: Python :: 3.8',
'Programming Language :: Python :: 3.9',
]
keywords = "vtk numpy plotting mesh"
dependencies = [
"numpy",
"imageio",
"pillow",
"appdirs",
"scooby>=0.5.1",
"vtk"
]
[project.optional-dependencies]
all = [
"matplotlib",
"colorcet",
"cmocean",
"meshio>=5.2",
"ipyvtklink",
"pythreejs"
]
colormaps = [
"matplotlib",
"colorcet",
"cmocean"
]
io = [
"meshio>=5.2"
]
jupyter = [
"ipyvtklink",
"pythreejs"
]
docs = [
"Sphinx==4.5.0",
"cmocean==2.0",
"colorcet==3.0.0",
"imageio-ffmpeg==0.4.7",
"imageio>=2.5.0",
"ipygany==0.5.0",
"ipyvtklink==0.2.2",
"jupyter_sphinx==0.4.0",
"jupyterlab==3.4.3",
"lxml==4.9.0",
"matplotlib==3.5.2",
"meshio==5.3.4",
"mypy-extensions==0.4.3",
"mypy==0.961",
"numpydoc==1.4.0",
"osmnx==1.2.1",
"panel==0.13.1",
"param==1.12.2 # due to panel bug",
"pydata-sphinx-theme==0.9.0",
"pypandoc==1.8.1",
"pytest-sphinx==0.4.0",
"pythreejs==2.3.0",
"scipy==1.8.1",
"sphinx-autobuild==2021.3.14",
"sphinx-copybutton==0.5.0",
"sphinx-gallery==0.10.1",
"sphinx-notfound-page==0.8",
"sphinx-panels==0.6.0",
"sphinxcontrib-websupport==1.2.4",
"trimesh==3.12.6",
"typed-ast==1.5.4",
"typing_extensions==4.2.0"
]
# Note: using < here to reduce breakages
# These should be automatically kept up to date by dependabot
test = [
"Sphinx<4.6.0",
"cmocean<2.1",
"codecov<2.2.0",
"colorcet<3.1.0",
"hypothesis<6.48.2",
"imageio-ffmpeg<0.5.0",
"imageio<2.20.0",
"ipygany<0.6.0",
"ipython<9.0.0",
"ipyvtklink<0.4.0",
"itkwidgets<0.33.0; python_version < '3.10'",
"matplotlib<3.6.0",
"meshio<5.4.0",
"panel<0.14.0",
"param<1.13.0",
"pytest-cov<3.1.0",
"pytest-memprof<0.3.0",
"pytest-xdist<2.6.0",
"pytest<7.2.0",
"pythreejs<2.4.0",
"sphinx-gallery<0.11.0",
"tqdm<4.65.0",
"trimesh<3.13.0"
]
[project.urls]
homepage = "https://www.pyvista.org/"
repository = "https://github.com/pyvista/pyvista"
documentation = "https://docs.pyvista.org/"
bug_tracker = "https://github.com/pyvista/pyvista/issues"
[tools.setuptools]
packages = [
"pyvista",
"pyvista.examples",
"pyvista.core",
"pyvista.core.filters",
"pyvista.demso",
"pyvista.jupyter",
"pyvista.plotting",
"pyvista.utilities",
"pyvista.ext"
]
[tools.setuptools.package_data]
pyvista = [
"py.typed"
]
pyvista.examples = [
"airplane.ply",
"ant.ply",
"channels.vti",
"hexbeam.vtk",
"sphere.ply",
"nut.ply",
"uniform.vtk",
"rectilinear.vtk",
"globe.vtk",
"2k_earth_daymap.jpg"
]
# Package Configurations
# ======================
# For ease of lookup, I try to keep these in alphabetical order
[tool.black]
line-length = 100
skip-string-normalization = true
target-version = ["py39"]
# `pre-commit` doesn't see skips; `force-exclude` forces it
# to see these files
#
# See:
# - https://github.com/psf/black/issues/1584
force-exclude = '''
(
pyvista/.*/__init__.py
| pyvista/__init__.py
)
'''
[tool.pydocstyle]
match = '''
/(
(?!
(
^pyvista/ext/
)
)
| pyvista/
| other/
/)
'''
[tool.coverage.run]
branch = true
# `shiboken6`, which creates the python bindings for `Qt` C++ source, imports from a
# `zip` file into the top-level directory at runtime. These files are deleted after
# running, but `coverage` attempts to look at their source after they're gone, causing
# warnings to appear. Namely, it looks for these modules/files:
#
# project_dir/pysrcript
# project_dir/shibokensupport
# project_dir/signature_bootstrap.py
#
# To avoid this error, `source` is specified to the package subdirectory. However, this
# can also be avoided by explicitly omitting these folders in the `omit` section.
#
# See https://github.com/nedbat/coveragepy/issues/1392
source = [
'pyvista/'
]
omit = [
'pyvista/ext/coverage.py',
'pyvista/conftest.py',
# kept for backwards compatibility:
'pyvista/plotting/theme.py'
]
disable_warnings = ['no-data-collected']
[tool.coverage.report]
exclude_lines = [
'pragma: no cover',
'def __repr__',
'raise AssertionError',
'raise NotImplementedError',
'if __name__ == .__main__.:',
'@(abc\.)?abstractmethod'
]
[tool.isort]
profile = "black"
line_length = 100
add_imports = [
# "from __future__ import annotations" # Automatically add to module on save, if not there
]
# Sort by name, don't cluster "from" vs "import"
force_sort_within_sections = true
# Combines "as" imports on the same line
combine_as_imports = true
# Prevent linter ignore comments from being moved off the offending line
ensure_newline_before_comments = true
# Some of Adam's favorites, for consideration
# multi_line_output = 3
# include_trailing_comma = true
# force_grid_wrap = 0
# use_parentheses = true
[tool.mypy]
ignore_missing_impoorts = true
# More of Adam's favorites...
# python_version = "3.9"
# disallow_untyped_defs = true
# warn_return_any = true
# warn_unused_configs = true
# warn_unused_ignores = true
# warn_redundant_casts = true
# show_error_codes = true
# no_pretty = true
# show_column_numbers = true
# plugins = [
# "pydantic.mypy"
# ]
# exclude = [
# 'venv',
# '\.venv/',
# 'build/',
# '_build/',
# 'dist/',
# 'ci/'
# ]
# fast_module_lookup = true
[[tool.mypy.overrides]]
module = "vtk.*"
ignore_missing_imports = true
[[tool.mypy.overrides]]
module = "matplotlib.*"
ignore_missing_imports = true
[[tool.mypy.overrides]]
module = "numpy.*"
ignore_missing_imports = true
[[tool.mypy.overrides]]
module = "scooby.*"
ignore_missing_imports = true
[[tool.mypy.overrides]]
module = "appdirs.*"
ignore_missing_imports = true
[[tool.mypy.overrides]]
module = "itkwidgets.*"
ignore_missing_imports = true
[tool.pydocstyle]
convention = "numpy"
[tool.pylint.master]
disable = """
raw-checker-failed,
bad-inline-option,
locally-disabled,
file-ignored,
suppressed-message,
useless-suppression,
deprecated-pragma,
use-symbolic-message-instead,
arguments-differ,
no-name-in-module,
no-member,
# Redundant alias imports required for type hinting by PEP 484
useless-import-alias,
# Qt uses PascalCase
invalid-name,
"""
extension-pkg-allow-list = """
vtk,
PyQt5,
PyQt6,
PySide2,
PySide6
"""
ignore_patterns = '''
/(
(?=
( coverage
| test_
)
).*[.]py
)/
'''
# Supposed to run `pylint` on multiple cores, but
# I've not had good experience with this...
# jobs = 0
[tool.pytest.ini_options]
junit_family = "legacy"
filterwarnings = [
'error',
'ignore::ResourceWarning',
# bogus numpy ABI warning (see numpy/#432)'
'ignore:.*numpy.dtype size changed.*:RuntimeWarning',
'ignore:.*numpy.ufunc size changed.*:RuntimeWarning',
# from usage of numpy_to_vtk:'
'ignore:.*`np.bool` is a deprecated alias.*:DeprecationWarning',
'ignore:.*`np.int` is a deprecated alias.*:DeprecationWarning',
'ignore:.*`np.float` is a deprecated alias.*:DeprecationWarning',
'ignore:.*`np.object` is a deprecated alias.*:DeprecationWarning',
'ignore:.*`np.long` is a deprecated alias:DeprecationWarning',
'ignore:.*Converting `np\.character` to a dtype is deprecated.*:DeprecationWarning'
]
# Some more Adam favorites...
# minversion = "7.0"
# addopts = """\
# --last-failed --last-failed-no-failures all \
# -p no:faulthandler \
# --import-mode=importlib \
# --randomly-seed=1234 \
# --cov \
# --cov-report term-missing \
# --cov-report html \
# --cov-report xml \
# --doctest-rst \
# --doctest-modules"""
# testpaths = [
# "tests",
# ]
# qt_api = "pyside6"
# doctest_plus = "enabled"
# env = [
# # "D:QT_QPA_PLATFORM=offscreen"
# # "D:COVERAGE_DEBUG=trace",
# # "D:COVERAGE_DEBUG_FILE=debug_log.txt"
# ]
# filterwarnings = [
# # Occurs when mocking QWidgets
# 'ignore:pyside_type_init:RuntimeWarning'
# ] And now for the main event...I am assuming we are using the latest version of
Excerpting from realpython.com: Zip imports were introduced in PEP 273 and are helpful when you need to distribute a package as a single file (its most common use case). PEP 302 added import hooks that give you import from a zip file, so long as it is in python's module search path. Using the builtin module
Another limitation not discussed in the article is that your code base will have to be refactored to use AFAIK, we do not distribute
@akaszynski @adeak Yes, I will add this to the |
…afe=False` `mypy` requires packages built with `setuptools` use the `zip_safe=False` option so it can find the package and `*.pyi` and `*.typed` files be added to the `MANIFEST.in` so they are included in the `sdist`. See https://mypy.readthedocs.io/en/latest/installed_packages.html#creating-pep-561-compatible-packages
@akaszynski @adeak @tkoyama010 Regarding transferring package configuration values to |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I’m happy with this.
I’m also fine with a follow up PR improving our pyproject.toml, if only for specifying setuptools as our build system.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I pushed a trivial cosmetic change (new line at end of MANIFEST.in). Consider the approvals intact.
Newlines! Hahaha. Thank you for fixing! |
Thank you @akaszynski ! If you and @tkoyama010 can approve and merge, I will work on the next PR to update the |
On the pyvista repo we try to wait 24 hours after approval before merging in order to give everyone around the globe a chance to check PRs. |
Oh I'm so sorry, I forgot! That was exactly what we brought up in our discussion Improving review habits #2245. My bad! Sorry for jumping the gun, I apologize. |
No worries at all, I just wanted to explain why we might seem like we're dragging our feet here :) And in any case with an 8-line diff here and 3 approvals across 3 continents this is probably fine. |
* upstream/main: Add setup and teardown functionality to plot_directive (pyvista#2907) Update hypothesis requirement from <6.48.4 to <6.49.2 (pyvista#2942) Bump lxml from 4.9.0 to 4.9.1 (pyvista#2936) Bump typing-extensions from 4.2.0 to 4.3.0 (pyvista#2939) Update hypothesis requirement from <6.48.3 to <6.48.4 (pyvista#2938) Bump trimesh from 3.12.6 to 3.12.7 (pyvista#2937) Update information of release (pyvista#2911) update pre-commit hooks (pyvista#2931) Handle invalid theme on init (pyvista#2917) Add `py.typed` Marker (pyvista#2904) [create-pull-request] update local intersphinx (pyvista#2926) Add tags with python commands to avoid misnumbering (pyvista#2416) Fix the url link of discretize (pyvista#2914) Add DataSet.cell_point_ids (pyvista#2897) Update hypothesis requirement from <6.48.2 to <6.48.3 (pyvista#2918) Ensure data remains a shallow copy when added to plotter (pyvista#2888)
@@ -0,0 +1 @@ | |||
partial\n |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
-partial\n
+partial
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If a stub package distribution is partial it MUST include partial\n in a py.typed file.
I believe it needs a newline. We can remove the \n
, but the PEP seems to identify the file must end with a new line
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I’ve posted the question on the Python Discussion Forum:
https://discuss.python.org/t/pep-561-clarification-regarding-n/32875
My main question is whether the \n
is deliberately required in order to distinguish Linux/MacOS line endings from Windows ones (\r\n
).
git config core.autocrlf
can be configured to store files with Linux line endings, but when checking out the repo on Windows, these are converted back to \r\n
, which I wonder if type checkers will honor or if they only honor \n
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@eggplants Per the discussion, this works with or without the newline in the mypy
implementation since that checker strips all whitespace, including explicit escape sequences (I have not looked at pyre
, pyright
, or other type checkers).
It seems a new line is requested to be included by convention, specifically the POSIX convention (ASIDE: A newline is useful in the event this file must be concatenated with another prior to processing).
This works with the \n
(the package is registered as partially typed), so nothing is breaking.
It’s a “6 one way, half a dozen the other” problem, so if we want this changed, I recommend removing the escape sequence, but adding a new line to the end of the file (i.e. hit Enter/Return on your keyboard one more time).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@adam-grant-hendry I've been trying to make it clear that this shouldn't work. What we have in this PR doesn't include an escape sequence. The exact contents of the file are 'partial\\n\n'
. There's a literal backslash, a literal n, and then an actual linefeed character. If you strip away this content you get 'partial\\n'
. Odds are the typing "works" because of the "edge case" code path in the above mypy code (I haven't looked closely).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is not worked with mypy.
fscache.read(stub_typed_file).decode().strip()
returns 'partial\\n'
and it is not recognized as Partial typed package because 'partial\\n' == 'partial'
returns False
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've been trying to make it clear that this shouldn't work.
Then why are we “fixing” a non-problem?
I can't really explain it in any other way. Please reread my comments (also on discuss) carefully. What "shouldn't work" is the file contents we have now. If it works now it's accidental. We are "fixing it" because it's wrong. The literal backslash and literal n must go. @eggplants' suggestion here is the fix to make it correct.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fscache.read(stub_typed_file).decode().strip()
returns 'partial\n' and it is not recognized as Partial typed package because 'partial\n' == 'partial' returns False.
Okay, THAT's the answer I was looking for. Thank you for the clear explanation. If that's the case, then certainly, partial\n
does not work.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@eggplants Did you happen to open a separate PR for this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just created! #4863
FWIW, I've submitted a PR to clarify PEP 561. |
This will allow static type checkers to recognized the package's type hints per PEP-561
Fixes issue #2903