Dropping Python2 support schedule and what to do

[junkmd]

Overviews

This package still supports Python 2.7.

However, it is planned to end support for Python2.7 in #216.
The timing is mid-2023 or maybe even earlier.
And by discussions in this issue, it is planned to support only non-EOL Pythons(3.7+) at drop-2.7 timing.

This issue will close when a version is released that no longer supports Python 2.7.

The community guidelines up to the dropping Python2 support

• The master branch would be in maintenance-only-mode. Except for fixing regressions, new contributions to codebase and documentation should be merged into the drop_py2 branch.
• Related tasks should be labeled drop_py2.
• It is assumed that changes to the codebase and documentation will be supporting only Python3.7+.
• Please install black==22.12.0, and format the codebase that you changed before submitting PR.
  • If you PRed from your forked repo, the GitHub Actions(GHA) will run black with --check --diff --color.
  • Auto formatting during PR by GHA or other tools is under consideration.
  • The formatter is only used for development and is not a runtime requirement.

How to add drop_py2 branch to your forked local and remote repository

1. git remote add upstream git://github.com/enthought/comtypes.git
2. git fetch upstream
3. git checkout -b drop_py2 upstream/drop_py2
4. git push origin drop_py2

To track current progress

See drop_py2 labeled issues or PRs and the milestone for current status.

Summary of changes

Release title should be "1.3.0 Dropping Python 2.7 support and fix ancient bugs"

 * Improve error message on non Windows environments. By @CristiFati.
 * Adjust styles of codebase. By @junkmd.
 * Apply ``black==22.12.0`` formatter. By @junkmd.
 * Modernize ``client.dynamic``. By @junkmd.
 * Add tests for ``client.dynamic``. By @junkmd.
 * Fix ``test_client``. By @junkmd.
 * Fix ``test_getactiveobj``. By @junkmd.
 * Make explicit the symbols that imports from the wrapper module into the friendly module. By @junkmd.
 * Modernize type annotations in statically defined modules. By @junkmd.
 * Fix several type annotations in statically defined modules. By @jonschz and @junkmd.
 * Remove ``sys.version_info`` bridges from production codebase. By @junkmd.
 * Remove ``for_stub=True`` conditional branch from the ``tools.codegenerator.ImportedNamespaces.getvalue``. By @junkmd.
 * Fix ``codegenerator`` as generating ``__next__()`` instead of ``next()`` in ``IEnum...``. By @jonschz.
 * Add unit test for generated ``IEnum...`` interfaces. By @jonschz.
 * Remove ``'Programming Language :: Python :: 2.7'`` from ``setup.py``. By @junkmd.
 * Change the base class of ``_ComMemberSpec`` and ``_DispMemberSpec`` to ``typing.NamedTuple``. By @junkmd.
 * Delint and remove wildcard-import from ``typeinfo``. By @junkmd.
 * Split ``DISPPARAMS`` instantiation in ``IDispatch.Invoke`` method. By @junkmd.
 * Move ``clear_comtypes_cache`` to be a callable ``comtypes.clear_cache`` module. By @bennyrowland.
 * Wrapper modules are now imported into friendly modules using an abstracted name, ``__wrapper_module__``. By @junkmd.
 * Fix old index bug in ``call_with_inout`` within ``_fix_inout_args``. By @jonschz.
 * Fix ``CONTRIBUTING.md``. By @junkmd
 * Update ``README.md``. By @junkmd

Schedule

From #216 (comment),

I am planning to observe movements after the release of 1.2.1 over the next few weeks (for regression and bug reports).

1.3.0 will be released after that observation.

Miscellaneous

This kanban will be updated as the situation requires.

Any opinions would be appreciated.

[jaraco]

I'd propose to also drop support for Python 3.6 (supporting only non-EOL Pythons, currently 3.7+), especially because the existing releases will continue to support Python 2.7 and older 3.x versions.

[jaraco]

On further consideration, I notice that current releases don't advertise their supported Python versions. Before cutting a release dropping support, this project should cut a release declaring support.

[jaraco]

I see you're working on this. That's great. I have a suggestion - let's put something like:

[options]
python_requires= >= 2.7

In setup.cfg.

[junkmd]

@jaraco

I'd propose to also drop support for Python 3.6 (supporting only non-EOL Pythons, currently 3.7+)

I agree.

One of the major differences between Python3.6 and 3.7 is that the syntax for asynchronous processing has changed(async and await).
However, this should not be a problem since it is not handled by comtypes.

Backward compatibility and wide availability of versions is great, but the fact that the latest features are not available is a barrier to entry for newcomers.

It would be beneficial to the community to sort out the EOL Python problems.

[junkmd]

@jaraco

Before cutting a release dropping support, this project should cut a release declaring support.

In setup.cfg.

Those are great suggestions 👍!

This is a package with a long history, so precedent was may being followed.

The introduction of modern ways of doing things is great.

[junkmd]

Since there seems to be no objection, I will create the drop_py2 branch mentioned in #327 during this weekend.
At the same time, I will also create the drop_py2 label to be attached to related tasks.
I will also create the following issue from several discussions...

• Add inline annotations as type hinting for dynamically generated modules in comtypes.gen.
• Add inline annotations as type hinting for statically defined modules, or change them from comment annotations
• Modify setup.py to prevent installation for non-Windows(it will be bound to Prevent .whl installation (OOTB) on non Win #394) Prevent .whl installation (OOTB) on non Win #394 is related to Create wheel package of comtypes #229

Also, #229 and #228 will be labeled drop_py2.

[junkmd]

I made drop_py2 branch and drop_py2 label.

In case of developments based on supporting only Python3, please use those.

I will post the related issue and revise this issue kanban shortly.

[junkmd]

I would remove the hints.AnnoField that was the workaround for type annotation to class fields in Py2.

However, the hints.pyi stub itself will remain for the following reasons;

Why is hints.pyi required?

It is true that using typing.TYPE_CHECKING will provide type information to the static type checker as if you were importing a symbol in the normal way.

But it makes it difficult for the IDE to show if it is valid or not in runtime.
Using such as hints.IDispatch instead of symbols that would be circularly imported in runtime makes it easy to visually distinguish.

Also, being a stub file, if it is accidentally annotated in non-quoted literal strings, it will immediately raise NameError before the function is called, making it easier to determine the cause of the problem if it occurs.

[junkmd]

A "Summary of changes" has been added to the kanban to summarize the current progress and work to date.
This might be also useful when making release notes.

[junkmd]

If "type hints in dynamically defined modules"(#400) were implemented to the codegenerator, the wrapper module code for stdole would be generated like this;

...
from typing import TYPE_CHECKING
if TYPE_CHECKING:
    from typing import Optional, Tuple
...

class IPicture(IUnknown):
    """Picture Object"""
    if TYPE_CHECKING:
        @property
        def Handle(self) -> int: ...
        @property
        def hPal(self) -> int: ...
        @hPal.setter
        def hPal(self, phpal: int) -> None: ...
        @property
        def Type(self) -> int: ...
        @property
        def Width(self) -> int: ...
        @property
        def Height(self) -> int: ...
        def Render(self, hdc: int, x: int, y: int, cx: int, cy: int, xSrc: int, ySrc: int, cxSrc: int, cySrc: int, prcWBounds: Optional[int]) -> int: ...
        @property
        def CurDC(self) -> int: ...
        def SelectPicture(self, hdcIn: int) -> Tuple[int, int]: ...
        @property
        def KeepOriginalFormat(self) -> bool: ...
        @KeepOriginalFormat.setter
        def KeepOriginalFormat(self, pfkeep: bool) -> None: ...
        def PictureChanged(self) -> int: ...
        def SaveAsFile(self, pstm: Optional[int], fSaveMemCopy: bool) -> int: ...
        @property
        def Attributes(self) -> int: ...
        def SetHdc(self, hdc: int) -> int: ...

    _case_insensitive_ = True
    _iid_ = GUID('{7BF80980-BF32-101A-8BBB-00AA00300CAB}')
    _idlflags_ = ['hidden']

...

(Argument and return types need a little more scrutiny)

`stdole.IPicture._methods_` definition

IPicture._methods_ = [
    COMMETHOD(
        ['propget'],
        HRESULT,
        'Handle',
        (['out', 'retval'], POINTER(OLE_HANDLE), 'phandle')
    ),
    COMMETHOD(
        ['propget'],
        HRESULT,
        'hPal',
        (['out', 'retval'], POINTER(OLE_HANDLE), 'phpal')
    ),
    COMMETHOD(
        ['propget'],
        HRESULT,
        'Type',
        (['out', 'retval'], POINTER(c_short), 'ptype')
    ),
    COMMETHOD(
        ['propget'],
        HRESULT,
        'Width',
        (['out', 'retval'], POINTER(OLE_XSIZE_HIMETRIC), 'pwidth')
    ),
    COMMETHOD(
        ['propget'],
        HRESULT,
        'Height',
        (['out', 'retval'], POINTER(OLE_YSIZE_HIMETRIC), 'pheight')
    ),
    COMMETHOD(
        [],
        HRESULT,
        'Render',
        (['in'], c_int, 'hdc'),
        (['in'], c_int, 'x'),
        (['in'], c_int, 'y'),
        (['in'], c_int, 'cx'),
        (['in'], c_int, 'cy'),
        (['in'], OLE_XPOS_HIMETRIC, 'xSrc'),
        (['in'], OLE_YPOS_HIMETRIC, 'ySrc'),
        (['in'], OLE_XSIZE_HIMETRIC, 'cxSrc'),
        (['in'], OLE_YSIZE_HIMETRIC, 'cySrc'),
        (['in'], c_void_p, 'prcWBounds')
    ),
    COMMETHOD(
        ['propput'],
        HRESULT,
        'hPal',
        (['in'], OLE_HANDLE, 'phpal')
    ),
    COMMETHOD(
        ['propget'],
        HRESULT,
        'CurDC',
        (['out', 'retval'], POINTER(c_int), 'phdcOut')
    ),
    COMMETHOD(
        [],
        HRESULT,
        'SelectPicture',
        (['in'], c_int, 'hdcIn'),
        (['out'], POINTER(c_int), 'phdcOut'),
        (['out'], POINTER(OLE_HANDLE), 'phbmpOut')
    ),
    COMMETHOD(
        ['propget'],
        HRESULT,
        'KeepOriginalFormat',
        (['out', 'retval'], POINTER(VARIANT_BOOL), 'pfkeep')
    ),
    COMMETHOD(
        ['propput'],
        HRESULT,
        'KeepOriginalFormat',
        (['in'], VARIANT_BOOL, 'pfkeep')
    ),
    COMMETHOD([], HRESULT, 'PictureChanged'),
    COMMETHOD(
        [],
        HRESULT,
        'SaveAsFile',
        (['in'], c_void_p, 'pstm'),
        (['in'], VARIANT_BOOL, 'fSaveMemCopy'),
        (['out'], POINTER(c_int), 'pcbSize')
    ),
    COMMETHOD(
        ['propget'],
        HRESULT,
        'Attributes',
        (['out', 'retval'], POINTER(c_int), 'pdwAttr')
    ),
    COMMETHOD(
        [],
        HRESULT,
        'SetHdc',
        (['in'], OLE_HANDLE, 'hdc')
    ),
]

If an executable method is defined, we must write an implementation to pass args to the method defined by metaclass.
And If the typing symbols overlap with the COM symbol in runtime, it might occur unexpected troubles.
In the drop_py2 plan, I am going to do only what to inform type hints to the IDE and type checkers, knowingly that inspect will not work well enough.

So I will define each method and property under if TYPE_CHECKING:.

[junkmd]

The last major version of supporting Python 2.7 (1.2.0) has been released.

I added the "Summary of changes" to the kanban of this issue to explain what will change from 1.2.x to 1.3.0.
This should also be useful when updating the CHANGES.txt.

[junkmd]

I noticed that the mscorlib.ITrackingHandler.MarshaledObject and UnmarshaledObject methods can take arguments whose names are or.

ITrackingHandler._methods_ = [
    COMMETHOD(
        [dispid(1610743808)],
        HRESULT,
        'MarshaledObject',
        (['in'], VARIANT, 'obj'),
        (['in'], POINTER(_ObjRef), 'or')
    ),
    COMMETHOD(
        [dispid(1610743809)],
        HRESULT,
        'UnmarshaledObject',
        (['in'], VARIANT, 'obj'),
        (['in'], POINTER(_ObjRef), 'or')
    ),
    COMMETHOD(
        [dispid(1610743810)],
        HRESULT,
        'DisconnectedObject',
        (['in'], VARIANT, 'obj')
    ),
]

Since or is a Python keyword, it raises a SyntaxError even if used in a block under if TYPE_CHECKING.

When generating function annotations for methods with such arguments, I am trying to implement them so that *args: Any, **kwargs: Any is annotated.

memo

• This test uses the mscorlib.
  

  comtypes/comtypes/test/test_client.py

  Lines 62 to 67 in 62ce303

  	def test_imports_IEnumVARIANT_from_other_generated_modules(self):
  	# NOTE: `codegenerator` generates code that contains unused imports,
  	# but removing them are attracting wierd bugs in library-wrappers
  	# which depend on externals.
  	# NOTE: `mscorlib`, which imports `IEnumVARIANT` from `stdole`.
  	comtypes.client.GetModule(("{BED7F4EA-1A96-11D2-8F08-00A0C9A6186D}",))

[junkmd]

Such as the IAccessible.accName property's setter and the Excel.Range.Value property's setter, there are required arguments exist after optional arguments.

They are not compatible with the Python language specifications.

IAccessible._methods_ = [
    ...
    COMMETHOD(
        [dispid(-5003), 'hidden', 'propget'],
        HRESULT,
        'accName',
        (['in', 'optional'], VARIANT, 'varChild'),
        (['out', 'retval'], POINTER(BSTR), 'pszName')
    ),
    ...
    COMMETHOD(
        [dispid(-5003), 'hidden', 'propput'],
        HRESULT,
        'accName',
        (['in', 'optional'], VARIANT, 'varChild'),
        (['in'], BSTR, 'pszName')
    ),
    ...
]

Range._disp_methods_ = [
    ...
    DISPMETHOD(
        [dispid(6), 'propget'],
        VARIANT,
        'Value',
        (['in', 'optional'], VARIANT, 'RangeValueDataType')
    ),
    ...
    DISPMETHOD(
        [dispid(6), 'propput'],
        None,
        'Value',
        (['in', 'optional'], VARIANT, 'RangeValueDataType'),
        (['in'], VARIANT, 'rhs')
    ),
    ...
]

To avoid SyntaxError, the code generator will annotate **kwargs: Any instead of the required arguments that come after the optional arguments.

Honestly, a more complex callback or overload is required.
However, in the first part of the PR, in order to keep the changes as small as possible, I have defined such method annotations.

[junkmd]

When I did pip install comtypes, I noticed that hints.pyi was not where it should be.

As PEP561, we have to make it explicit in setup.py that there is a stub file in the package.

I don't think it is urgent as it is not an error at runtime, but I will write a PR so that we can resolve it at the same time when fix runtime bugs or next release.

[junkmd]

From #216 (comment),

I am planning to observe movements after the release of 1.2.1 over the next few weeks (for regression and bug reports).

1.3.0 will be released after that observation.

In light of this, I have reviewed the current milestones for 1.3.0.

The following items, which have been present since before the drop_py2 plan was initiated, have shown little recent activity and seems unlikely to be resolved before the release. Therefore, they has been removed from the 1.3.0 milestone.

• Can't create VARIANT with typecode VT_ARRAY | BT_BSTR #80
• have 64-bit client check 32-bit registry hive #89
• CreateObject fails and then suceeds #193
• array of BSTR is not supported #347

The following items, originally included in the drop_py2 plan, were also removed from the 1.3.0 milestone. This decision was made due to the significant impact of the changes required for inclusion in 1.3.0 and the necessity to announce these changes to comtypes users prior to implementation.

• Define enumerations in generated modules. #345
• Add inline type annotations to dynamically generated modules #400
• Add enumeration definitions in the friendly modules #475
• Add method annotations (all args and return-values are Any) #490

[junkmd]

Unless there are regression reports, I am considering releasing version 1.3.0 next week, which will only support Python 3.

[junkmd]

comtypes==1.3.0 is released now!
https://pypi.org/project/comtypes/1.3.0/

Thank you for all participants!