Add stubs for pycocotools #9086

hoel-bagard · 2022-11-04T04:07:27Z

This PR adds stubs for pycocotools (see #9061).

AlexWaygood · 2022-11-04T09:44:23Z

Unfortunately you won't be able to use numpy as a dependency just yet. This should hopefully be changing soon; see #5768.

For now, as a workaround, I recommend you do something similar to what we do here:

typeshed/stubs/D3DShot/d3dshot/capture_outputs/numpy_capture_output.pyi

Line 14 in 3ac147c

_NDArray: TypeAlias = Incomplete

hoel-bagard · 2022-11-04T10:08:04Z

@AlexWaygood Thank you for the workaround. I've removed the use of numpy, but if possible I'd like to leave the numpy typing information in case it can be used in the future. I'm not sure if it's better leave all the original lines as comments like here or to create aliases for every type I'm using like here.

AlexWaygood · 2022-11-04T10:15:38Z

You can probably actually just replace the import line with a TypeAlias to Incomplete:

- import numpy as np
- import numpy.typing as not
+ # TODO when #5768 is resolved
+ # import numpy as np
+ # import numpy.typing as npt
+ np: TypeAlias = Incomplete
+ npt: TypeAlias = Incomplete

And then have the rest of the stub as it would be if you were actually importing numpy as np. I'm not 100% sure if that'll work, but might be worth a shot?

(TypeAlias needs to be imported from typing_extensions, BTW. And you'll need to use _typeshed.Self instead of typing(_extensions).Self for now, since mypy doesn't support typing(_extensions).Self yet. Note that _typeshed.Self is just a normal TypeVar, so it works slightly differently to typing(_extensions).Self.)

hoel-bagard · 2022-11-04T11:19:29Z

I've tried replacing the import line with a TypeAlias to Incomplete, but it doesn't work. I get errors like these:

stubs/pycocotools/pycocotools/coco.pyi:41: error: Name "npt.NDArray" is not defined
stubs/pycocotools/pycocotools/coco.pyi:41: error: Name "np.float64" is not defined

AlexWaygood · 2022-11-04T11:22:53Z

I've tried replacing the import line with a TypeAlias to Incomplete, but it doesn't work. I get errors like these:
stubs/pycocotools/pycocotools/coco.pyi:41: error: Name "npt.NDArray" is not defined
stubs/pycocotools/pycocotools/coco.pyi:41: error: Name "np.float64" is not defined

Okay, what you've done looks fine, in that case. It would be good to include a reference to #5768 in a comment(s) somewhere, though — that way we'll find it when we're grepping through the codebase after #5768 is solved

pyrightconfig.stricter.json

hoel-bagard · 2022-11-05T11:58:08Z

@AlexWaygood Two checks are still failing, but I don't understand why or how to fix it. Would you have time to look at it ?

AlexWaygood · 2022-11-05T12:00:23Z

Sure, I'll take a look soon!

AlexWaygood

Stubtest is a tool that compares the stub to the runtime, and verifies that the two are consistent with each other. Currently stubtest is unhappy about a few imports that don't look like they exist at runtime, and a file that doesn't exist at runtime :)

stubs/pycocotools/pycocotools/__init__.pyi

stubs/pycocotools/pycocotools/coco_types.pyi

hoel-bagard · 2022-11-07T05:01:12Z

Thank you!
I've done as you instructed and now all the checks are passing. Is there anything else that needs to be modified ?

AlexWaygood · 2022-11-07T07:36:18Z

Thanks! I still need to do a manual review, but it looks like we're nearly there :)

hoel-bagard · 2022-11-16T03:33:31Z

Hi, do you have an idea of when you will be able to do the review ? (anytime is fine, I'm just wondering what I should expect)

In the stub I made, the part I'm the most unsure about this the use of _AnnotationG in stubs/pycocotools/pycocotools/coco.pyi. One of the methods of the package (annToRLE) returns a different type depending on the type of one of the element from the input dictionary, and I'm not sure how that should be typed.

hoel-bagard · 2022-11-24T02:22:37Z

@AlexWaygood

AlexWaygood

Thanks for your patience, @hoel-bagard, and I'm sorry I've taken so long in getting to this. I've been busier than I expected the last few weeks :(

This mostly looks good, but I've left a few review comments. In general, we need to be quite cautious about using concrete types for parameter annotations. Abstract types are usually better -- they mean we don't run into annoying issues around invariance. There's some documentation on this here:

stubs/pycocotools/pycocotools/coco.pyi

AlexWaygood · 2022-11-28T11:46:09Z

stubs/pycocotools/pycocotools/mask.pyi

+def iou(
+    dt: _NDArrayUInt32 | list[float] | list[_EncodedRLE],
+    gt: _NDArrayUInt32 | list[float] | list[_EncodedRLE],
+    pyiscrowd: list[int] | _NDArrayUInt8,
+) -> list[Any] | _NDArrayFloat64: ...


(No change requested, just pointing out to clarify:) This function should have the parameters annotated as list, since the code explicitly forbids other kinds of sequences: https://github.com/ppwwyyxx/cocoapi/blob/71e284ef862300e4319aacd523a64c7f24750178/PythonAPI/pycocotools/_mask.pyx#L182

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

AlexWaygood

Thank you! Sorry again about the wait :)

hoel-bagard · 2022-11-28T12:40:23Z

Thank you for the great review process and all the advice !

github-actions · 2022-11-28T12:45:12Z

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

Removals from `stubinfo.py`: - `atomicwrites` is archived and deprecated at runtime; stubs were removed from typeshed in python/typeshed#8925 - `attrs` has had inline types for a very long time now - `chardet` recently cut a release with inline types; typeshed's stubs were marked obsolete in python/typeshed#9318 - `cryptography` has had inline types for a very long time now; the only reason why it's still in typeshed is because other typeshed packages need `types-cryptography` as a dependency, and our testing infrastructure therefore can't currently cope with it being removed from typeshed. - `emoji` recently cut a release bundling stubs with the runtime package; typeshed's stubs were marked obsolete in python/typeshed#9051 - `termcolor` recently cut a release with inline types; typeshed's stubs were marked obsolete in python/typeshed#8746 - `prettytable` recently cut a release with inline types; typeshed's stubs were marked obsolete in python/typeshed#9023 Additions: - Stubs for `Xlib` were added in python/typeshed#9279 - Stubs for `consolemenu` were added in python/typeshed#8820 - Stubs for `dockerfile_parse` were added in python/typeshed#9305 - Stubs for `flask_migrate` were added in python/typeshed#8967 - Stubs for `paho.mqtt` were added in python/typeshed#8853 - Stubs for `pycocotools` were added in python/typeshed#9086 - Stubs for many `pywin32` modules were added in python/typeshed#8825, and multiple follow-up PRs - Stubs for `pyscreeze` were added in python/typeshed#8823

hoel-bagard and others added 6 commits November 4, 2022 11:32

Added the coco.pyi file.

ae9e2d2

Added the mask.pyi file.

2a1c251

Added the cocoeval.pyi file.

81675f9

Added imports to the __init__.

07febd5

Merge branch 'python:main' into main

9c204ee

[pre-commit.ci] auto fixes from pre-commit.com hooks

4faa515

This comment has been minimized.

Sign in to view

hoel-bagard and others added 3 commits November 4, 2022 14:49

Fixed some ids being allowed to also be simple int/str instead of list.

3fe497d

Merged with main.

2699f58

[pre-commit.ci] auto fixes from pre-commit.com hooks

87039a7

This comment has been minimized.

Sign in to view

Fixed name missing an '_'

feb9114

This comment has been minimized.

Sign in to view

Moved some imports from typing to typing_extensions.

f03e88c

This comment has been minimized.

Sign in to view

hoel-bagard and others added 3 commits November 4, 2022 19:01

Removed numpy dependency.

8aca487

[pre-commit.ci] auto fixes from pre-commit.com hooks

a8cd2d5

Removed extra underscores.

8b52fca

This comment has been minimized.

Sign in to view

hoel-bagard and others added 3 commits November 4, 2022 19:16

Merge branch 'main' of github.com:hoel-bagard/typeshed

8596fab

Fixed TypeAlias import.

294e104

[pre-commit.ci] auto fixes from pre-commit.com hooks

749bdfa

This comment has been minimized.

Sign in to view

Now using the Self type from _typeshed.

cd6001f

AlexWaygood reviewed Nov 4, 2022

View reviewed changes

pyrightconfig.stricter.json Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

AlexWaygood self-requested a review November 5, 2022 12:00

AlexWaygood reviewed Nov 6, 2022

View reviewed changes

stubs/pycocotools/pycocotools/__init__.pyi Outdated Show resolved Hide resolved

stubs/pycocotools/pycocotools/coco_types.pyi Outdated Show resolved Hide resolved

hoel-bagard and others added 2 commits November 7, 2022 13:21

Moved the private types to __init__ and the files they are used.

99a28bd

[pre-commit.ci] auto fixes from pre-commit.com hooks

a686ab9

This comment has been minimized.

Sign in to view

AlexWaygood self-requested a review November 7, 2022 07:36

Removed comment.

3f668c3

This comment has been minimized.

Sign in to view

AlexWaygood reviewed Nov 28, 2022

View reviewed changes

hoel-bagard and others added 8 commits November 28, 2022 21:29

Updated imports.

34b09cd

Replace list by Collection/Sequence.

7472602

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

Replace list by Collection/Sequence.

6e5bb73

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

Replace list by Collection/Sequence.

8857817

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

Replace list by Collection/Sequence.

d9a3b8e

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

Replace list by Sequence.

f7a13bf

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

Replace Self type by COCO,

4d29cf0

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

3f28dcb

AlexWaygood approved these changes Nov 28, 2022

View reviewed changes

AlexWaygood merged commit a132ff2 into python:main Nov 28, 2022

AlexWaygood mentioned this pull request Dec 8, 2022

Update stubinfo.py for recent typeshed changes python/mypy#14265

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add stubs for pycocotools #9086

Add stubs for pycocotools #9086

hoel-bagard commented Nov 4, 2022

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

AlexWaygood commented Nov 4, 2022

This comment has been minimized.

hoel-bagard commented Nov 4, 2022

This comment has been minimized.

AlexWaygood commented Nov 4, 2022 •

edited

This comment has been minimized.

hoel-bagard commented Nov 4, 2022

AlexWaygood commented Nov 4, 2022 •

edited

This comment has been minimized.

hoel-bagard commented Nov 5, 2022

AlexWaygood commented Nov 5, 2022 •

edited

AlexWaygood left a comment

This comment has been minimized.

hoel-bagard commented Nov 7, 2022

AlexWaygood commented Nov 7, 2022 •

edited

hoel-bagard commented Nov 16, 2022

This comment has been minimized.

hoel-bagard commented Nov 24, 2022

AlexWaygood left a comment

AlexWaygood Nov 28, 2022

AlexWaygood left a comment

hoel-bagard commented Nov 28, 2022 •

edited

github-actions bot commented Nov 28, 2022

Add stubs for pycocotools #9086

Add stubs for pycocotools #9086

Conversation

hoel-bagard commented Nov 4, 2022

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

AlexWaygood commented Nov 4, 2022

This comment has been minimized.

hoel-bagard commented Nov 4, 2022

This comment has been minimized.

AlexWaygood commented Nov 4, 2022 • edited

This comment has been minimized.

hoel-bagard commented Nov 4, 2022

AlexWaygood commented Nov 4, 2022 • edited

This comment has been minimized.

hoel-bagard commented Nov 5, 2022

AlexWaygood commented Nov 5, 2022 • edited

AlexWaygood left a comment

Choose a reason for hiding this comment

This comment has been minimized.

hoel-bagard commented Nov 7, 2022

AlexWaygood commented Nov 7, 2022 • edited

hoel-bagard commented Nov 16, 2022

This comment has been minimized.

hoel-bagard commented Nov 24, 2022

AlexWaygood left a comment

Choose a reason for hiding this comment

AlexWaygood Nov 28, 2022

Choose a reason for hiding this comment

AlexWaygood left a comment

Choose a reason for hiding this comment

hoel-bagard commented Nov 28, 2022 • edited

github-actions bot commented Nov 28, 2022

AlexWaygood commented Nov 4, 2022 •

edited

AlexWaygood commented Nov 4, 2022 •

edited

AlexWaygood commented Nov 5, 2022 •

edited

AlexWaygood commented Nov 7, 2022 •

edited

hoel-bagard commented Nov 28, 2022 •

edited