Define enumerations in generated modules.

[junkmd]

Motivation

Currently, when we generate a Python module from a COM type library with client.GetModule, the stuffs defined as Enumeration in the COM type library are defined in Python as aliases for ctypes.c_int.

Importing and using them in a production code is rarely done.

They are passed to STDMETHOD, COMMETHOD, DISPMETHOD or DISPPROPERTY to define method and property arguments and return values.
It is difficult to add features such as enumerated type member information to these symbols by using classes inherited from c_int or by using weird tricks regarding __dict__, given the magnitude of impact.

This problem has been left as a comment that marked XXX on the codegenerator, since this package was born.

comtypes/comtypes/tools/codegenerator.py

Lines 406 to 421 in cc9a013

	def Enumeration(self, tp):
	self._enumtypes += 1
	self.last_item_class = False
	if tp.name:
	print("# values for enumeration '%s'" % tp.name, file=self.stream)
	else:
	print("# values for unnamed enumeration", file=self.stream)
	# Some enumerations have the same name for the enum type
	# and an enum value. Excel's XlDisplayShapes is such an example.
	# Since we don't have separate namespaces for the type and the values,
	# we generate the TYPE last, overwriting the value. XXX
	for item in tp.values:
	self.generate(item)
	if tp.name:
	print("%s = c_int # enum" % tp.name, file=self.stream)
	self.names.add(tp.name)

Summary Examples

This feature is implemented by using the same name but making it an alias of c_int in the wrapper module and a subclass of enum.IntFlag in the friendly module.

The .../comtypes/gen/Scripting.py prior to this change looks something like this.

import comtypes.gen._420B2830_E718_11CF_893D_00A0C9054228_0_1_0 as __wrapper_module__
from comtypes.gen._420B2830_E718_11CF_893D_00A0C9054228_0_1_0 import (
    FileAttribute, DriveTypeConst, TristateMixed, IFileSystem,
    SystemFolder, IFileCollection, StandardStreamTypes,
    _check_version, GUID, IFolderCollection, Dictionary, Directory,
    Remote, Library, StdOut, IOMode,
    __MIDL___MIDL_itf_scrrun_0001_0001_0003, WindowsFolder, VARIANT,
    Alias, helpstring, TristateUseDefault, IScriptEncoder, Archive,
    TextStream, IDrive, IFile, HRESULT, Hidden, StdErr, Folders,
    dispid, typelib_path, IFileSystem3, ReadOnly, ForWriting, StdIn,
    FileSystemObject, ForAppending, Encoder, CDRom, IDriveCollection,
    COMMETHOD, Volume, CompareMethod, BinaryCompare, IDictionary,
    BSTR, Normal, TristateTrue, TristateFalse, Fixed, UnknownType,
    Files, File, _lcid, Tristate, SpecialFolderConst, ForReading,
    ITextStream, __MIDL___MIDL_itf_scrrun_0001_0001_0001, Removable,
    RamDisk, VARIANT_BOOL, Drives, Folder, TemporaryFolder, CoClass,
    __MIDL___MIDL_itf_scrrun_0000_0000_0001, System,
    __MIDL___MIDL_itf_scrrun_0001_0001_0002, Compressed, TextCompare,
    DatabaseCompare, IFolder, IUnknown, Drive
)


__all__ = [
    'Encoder', 'FileAttribute', 'CDRom', 'IDriveCollection',
    'DriveTypeConst', 'TristateMixed', 'Volume', 'SystemFolder',
    'IFileSystem', 'IFileCollection', 'CompareMethod',
    'StandardStreamTypes', 'BinaryCompare', 'IDictionary', 'Normal',
    'TristateTrue', 'TristateFalse', 'Fixed', 'IFolderCollection',
    'Dictionary', 'Directory', 'Remote', 'Library', 'StdOut',
    'IOMode', '__MIDL___MIDL_itf_scrrun_0001_0001_0003',
    'UnknownType', 'WindowsFolder', 'Files', 'File', 'Tristate',
    'Alias', 'ForReading', 'SpecialFolderConst', 'ITextStream',
    'TristateUseDefault', 'IScriptEncoder',
    '__MIDL___MIDL_itf_scrrun_0001_0001_0001', 'Archive', 'Removable',
    'RamDisk', 'IDrive', 'TextStream', 'IFile', 'Drives', 'Folder',
    'Hidden', 'StdErr', 'Folders', 'TemporaryFolder', 'typelib_path',
    '__MIDL___MIDL_itf_scrrun_0000_0000_0001', 'ReadOnly', 'System',
    'IFileSystem3', 'ForWriting', 'Compressed',
    '__MIDL___MIDL_itf_scrrun_0001_0001_0002', 'StdIn', 'TextCompare',
    'DatabaseCompare', 'FileSystemObject', 'IFolder', 'Drive',
    'ForAppending'
]

With this change, it looks like this.

from enum import IntFlag

import comtypes.gen._420B2830_E718_11CF_893D_00A0C9054228_0_1_0 as __wrapper_module__
from comtypes.gen._420B2830_E718_11CF_893D_00A0C9054228_0_1_0 import (
    Hidden, Volume, Alias, TextCompare, SystemFolder,
    FileSystemObject, Fixed, typelib_path, IDrive, IFile, IDictionary,
    CoClass, _lcid, ForWriting, RamDisk, StdIn, COMMETHOD, Removable,
    CDRom, Files, TristateUseDefault, Archive, TristateMixed,
    _check_version, WindowsFolder, Remote, DatabaseCompare,
    TextStream, IUnknown, helpstring, HRESULT, IFolderCollection,
    Encoder, ITextStream, Dictionary, Compressed, VARIANT, BSTR,
    IFileCollection, IDriveCollection, Folder, VARIANT_BOOL, Drives,
    Folders, System, StdOut, UnknownType, File, ForAppending, StdErr,
    ReadOnly, BinaryCompare, IFolder, IScriptEncoder, IFileSystem,
    Drive, Normal, Directory, ForReading, TristateTrue, IFileSystem3,
    Library, dispid, TemporaryFolder, TristateFalse, GUID
)


class __MIDL___MIDL_itf_scrrun_0001_0001_0003(IntFlag):
    StdIn = __wrapper_module__.StdIn
    StdOut = __wrapper_module__.StdOut
    StdErr = __wrapper_module__.StdErr


class CompareMethod(IntFlag):
    BinaryCompare = __wrapper_module__.BinaryCompare
    TextCompare = __wrapper_module__.TextCompare
    DatabaseCompare = __wrapper_module__.DatabaseCompare


class __MIDL___MIDL_itf_scrrun_0000_0000_0001(IntFlag):
    Normal = __wrapper_module__.Normal
    ReadOnly = __wrapper_module__.ReadOnly
    Hidden = __wrapper_module__.Hidden
    System = __wrapper_module__.System
    Volume = __wrapper_module__.Volume
    Directory = __wrapper_module__.Directory
    Archive = __wrapper_module__.Archive
    Alias = __wrapper_module__.Alias
    Compressed = __wrapper_module__.Compressed


class __MIDL___MIDL_itf_scrrun_0001_0001_0002(IntFlag):
    WindowsFolder = __wrapper_module__.WindowsFolder
    SystemFolder = __wrapper_module__.SystemFolder
    TemporaryFolder = __wrapper_module__.TemporaryFolder


class __MIDL___MIDL_itf_scrrun_0001_0001_0001(IntFlag):
    UnknownType = __wrapper_module__.UnknownType
    Removable = __wrapper_module__.Removable
    Fixed = __wrapper_module__.Fixed
    Remote = __wrapper_module__.Remote
    CDRom = __wrapper_module__.CDRom
    RamDisk = __wrapper_module__.RamDisk


class IOMode(IntFlag):
    ForReading = __wrapper_module__.ForReading
    ForWriting = __wrapper_module__.ForWriting
    ForAppending = __wrapper_module__.ForAppending


class Tristate(IntFlag):
    TristateTrue = __wrapper_module__.TristateTrue
    TristateFalse = __wrapper_module__.TristateFalse
    TristateUseDefault = __wrapper_module__.TristateUseDefault
    TristateMixed = __wrapper_module__.TristateMixed


StandardStreamTypes = __MIDL___MIDL_itf_scrrun_0001_0001_0003
FileAttribute = __MIDL___MIDL_itf_scrrun_0000_0000_0001
DriveTypeConst = __MIDL___MIDL_itf_scrrun_0001_0001_0001
SpecialFolderConst = __MIDL___MIDL_itf_scrrun_0001_0001_0002


__all__ = [
    'DriveTypeConst', 'FileAttribute', 'Hidden', 'Compressed',
    'SpecialFolderConst', '__MIDL___MIDL_itf_scrrun_0001_0001_0003',
    'StandardStreamTypes', 'Volume',
    '__MIDL___MIDL_itf_scrrun_0001_0001_0001', 'IFileCollection',
    'Alias', 'TextCompare', 'IDriveCollection', 'Folder',
    'SystemFolder', '__MIDL___MIDL_itf_scrrun_0000_0000_0001',
    'IOMode', 'FileSystemObject', 'Drives', 'Fixed', 'Folders',
    'typelib_path', 'IDrive', 'Tristate', 'System', 'StdOut', 'IFile',
    'UnknownType', 'File', 'IDictionary', 'ForAppending',
    'ForWriting', 'StdErr', 'CompareMethod', 'RamDisk', 'ReadOnly',
    'StdIn', '__MIDL___MIDL_itf_scrrun_0001_0001_0002',
    'BinaryCompare', 'Encoder', 'IFolder', 'IScriptEncoder',
    'Removable', 'IFileSystem', 'CDRom', 'Files',
    'TristateUseDefault', 'Drive', 'Archive', 'Normal',
    'TristateMixed', 'Directory', 'WindowsFolder', 'Remote',
    'DatabaseCompare', 'ForReading', 'TristateTrue', 'IFileSystem3',
    'Library', 'TextStream', 'TemporaryFolder', 'TristateFalse',
    'IFolderCollection', 'ITextStream', 'Dictionary'
]

Why IntFlag?

The reasons as to why IntFlag is used for enumerations, because ...

• They are sometimes used as bit flags
• It is complicated to write a .value every time.
• It is exactly for interoperability with other systems.

Compatibility with traditional c_int aliases

Codebases like below will no longer work as before with this change because they rely on the symbol is an alias for c_int.

from comtypes.gen.Scripting import Tristate

Tristate.from_param(...)

import ctypes

from comtypes.gen import Scripting

for k, v in vars(Scripting).items():
    if v is ctypes.c_int and k != "c_int":
        ...

There are some ways to use both new and legacy functionalities.
Users can still use the old definitions by changing import and the definitions of module-level symbols as follows;

- from comtypes.gen.Scripting import Tristate
+ import ctypes
+ Tristate = ctypes.c_int

- from comtypes.gen.Scripting import Tristate
+ from comtypes.gen import Scripting
+ Tristate = Scripting.__wrapper_module__.Tristate

- from comtypes.gen import Scripting
+ from comtypes.gen.Scripting import __wrapper_module__ as Scripting

previous descriptions (ideas)

I propose the following procedure as a solution to this problem.

1. The args passed to functions like STDMETHOD will be changed to ctypes.c_int itself from the symbols defined as Enumeration and External or Alias those referred to Enumeration in the COM type library.

   • Actual code will be below.

   ...
   XlCreator = c_int  # enum
   ...
       DISPMETHOD([dispid(149), 'propget'], XlCreator, 'Creator'),
   ...

   ↓

   ...
   XlCreator = c_int  # enum
   ...
       DISPMETHOD([dispid(149), 'propget'], c_int, 'Creator'),
   ...

2. The symbols currently defined as aliases of ctypes.c_int will be changed the definition as enumeration types.

   • The de facto standard for defining enumeration types in Python is the enum module.
     The enum module is supported since Py3.4, so it is not available in Py2.7 or Py3.3.
     Therefore, it will be after older Python versions are dropped that these will actually be changed.
     • Plans to drop Py2.7 and older Py3.x are being discussed.

[junkmd]

I have found an implementation that allows us to act as if it were a submodule in a package without creating another .py file.

https://github.com/python/cpython/blob/555e76e90722a278f47ff9f14682af0220016a03/Lib/typing.py#L3289-L3312

Using the way of implementation for typing.io, we can create a module that defines enumerations without duplicating the namespace of the module and without another .py file.

The code for wrapper-module is assumed to be as follows

# assuming generated from `stdole2.tlb`
# -*- coding: mbcs -*-

from ctypes import *
from comtypes.automation import DISPPARAMS, EXCEPINFO, IDispatch, IEnumVARIANT
from comtypes import (
    _check_version, BSTR, CoClass, COMMETHOD, dispid, DISPMETHOD,
    DISPPROPERTY, GUID, IUnknown
)
from ctypes.wintypes import VARIANT_BOOL
from ctypes import HRESULT

_lcid = 0  # change this if required
typelib_path = 'C:\\Windows\\System32\\stdole2.tlb'
OLE_XPOS_HIMETRIC = c_int
...
# values for enumeration 'OLE_TRISTATE'
Unchecked = 0
Checked = 1
Gray = 2
OLE_TRISTATE = c_int  # enum
...

class __enumerations(object):
    """Wrapper namespace for stdole enumerations."""
    from enum import IntEnum  # or IntFlag?

    class OLE_TRISTATE(IntEnum):
        Unchecked = 0
        Checked = 1
        Gray = 2

__enumerations.__name__ = __name__ + '.enums'
sys.modules[__enumerations.__name__] = __enumerations

[junkmd]

If it came to writing the enumeration definitions in the wrapper module, I had to change the codegenerator quite a bit.
Also, I would have to use some magic to twist the module namespace.

However, if definitions of the enumerations were in the friendly module, I can do what I want without damaging the wrapper module implementation.

Of course, this will break backward compatibility because symbols like comtypes.gen.XlPasteType would be no longer aliases for c_int.

But, I suspect that few people would import and use XlPasteType as an alias for c_int from a friendly module.
Besides, if they are really using it that way, writing c_int = XlPasteType in their own project should solve the problem.

The following code is generated for comtypes.gen.Scripting.

from enum import IntFlag

from comtypes.gen import _420B2830_E718_11CF_893D_00A0C9054228_0_1_0
from comtypes.gen._420B2830_E718_11CF_893D_00A0C9054228_0_1_0 import (
    Alias, _check_version, Folder, helpstring, VARIANT_BOOL, Hidden,
    TextStream, dispid, TemporaryFolder, ForWriting, IFile,
    IScriptEncoder, ForReading, TristateUseDefault, CDRom, Remote,
    Removable, Fixed, Compressed, DatabaseCompare, TristateFalse,
    StdOut, Directory, Drives, Dictionary, TristateMixed, StdErr,
    StdIn, System, VARIANT, IDictionary, ReadOnly, IFolderCollection,
    Library, UnknownType, IDrive, TextCompare, IFileSystem3, Volume,
    File, WindowsFolder, TristateTrue, IFolder, ForAppending, Folders,
    CoClass, _lcid, RamDisk, BinaryCompare, FileSystemObject, GUID,
    IFileSystem, IFileCollection, SystemFolder, typelib_path, Archive,
    Drive, COMMETHOD, IDriveCollection, Encoder, HRESULT, IUnknown,
    Normal, BSTR, Files, ITextStream
)


class CompareMethod(IntFlag):
    BinaryCompare = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.BinaryCompare
    TextCompare = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.TextCompare
    DatabaseCompare = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.DatabaseCompare


class __MIDL___MIDL_itf_scrrun_0001_0001_0002(IntFlag):
    WindowsFolder = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.WindowsFolder
    SystemFolder = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.SystemFolder
    TemporaryFolder = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.TemporaryFolder


class IOMode(IntFlag):
    ForReading = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.ForReading
    ForWriting = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.ForWriting
    ForAppending = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.ForAppending


class Tristate(IntFlag):
    TristateTrue = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.TristateTrue
    TristateFalse = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.TristateFalse
    TristateUseDefault = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.TristateUseDefault
    TristateMixed = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.TristateMixed


class __MIDL___MIDL_itf_scrrun_0001_0001_0001(IntFlag):
    UnknownType = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.UnknownType
    Removable = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.Removable
    Fixed = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.Fixed
    Remote = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.Remote
    CDRom = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.CDRom
    RamDisk = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.RamDisk


class __MIDL___MIDL_itf_scrrun_0001_0001_0003(IntFlag):
    StdIn = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.StdIn
    StdOut = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.StdOut
    StdErr = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.StdErr


class __MIDL___MIDL_itf_scrrun_0000_0000_0001(IntFlag):
    Normal = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.Normal
    ReadOnly = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.ReadOnly
    Hidden = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.Hidden
    System = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.System
    Volume = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.Volume
    Directory = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.Directory
    Archive = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.Archive
    Alias = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.Alias
    Compressed = _420B2830_E718_11CF_893D_00A0C9054228_0_1_0.Compressed


SpecialFolderConst = __MIDL___MIDL_itf_scrrun_0001_0001_0002
DriveTypeConst = __MIDL___MIDL_itf_scrrun_0001_0001_0001
FileAttribute = __MIDL___MIDL_itf_scrrun_0000_0000_0001
StandardStreamTypes = __MIDL___MIDL_itf_scrrun_0001_0001_0003


__all__ = [
    'Alias', 'Folder', 'UnknownType',
    '__MIDL___MIDL_itf_scrrun_0001_0001_0001',
    '__MIDL___MIDL_itf_scrrun_0000_0000_0001', 'IDrive',
    'TextCompare', 'Hidden', 'IFileSystem3', 'Volume', 'TextStream',
    'TemporaryFolder', 'FileAttribute', 'ForWriting', 'IFile', 'File',
    'IScriptEncoder', 'IOMode', 'WindowsFolder', 'ForReading',
    'StandardStreamTypes', 'TristateTrue', 'IFolder',
    'TristateUseDefault', 'ForAppending', 'CDRom', 'Remote',
    'Folders', 'Removable', 'RamDisk', 'Fixed', 'Compressed',
    'DatabaseCompare', '__MIDL___MIDL_itf_scrrun_0001_0001_0003',
    'BinaryCompare', 'TristateFalse', 'StdOut', 'Directory',
    'FileSystemObject', 'DriveTypeConst', 'Drives', 'IFileSystem',
    'IFileCollection', 'SystemFolder', 'typelib_path', 'Dictionary',
    'Archive', 'TristateMixed', 'SpecialFolderConst', 'Drive',
    'StdErr', 'StdIn', 'System', 'IDriveCollection', 'Encoder',
    'IDictionary', '__MIDL___MIDL_itf_scrrun_0001_0001_0002',
    'Normal', 'ReadOnly', 'CompareMethod', 'IFolderCollection',
    'Library', 'Files', 'ITextStream', 'Tristate'
]

[junkmd]

The reasons as to why IntFlag is used for enumerations, because ...

• They are sometimes used as bit flags
• It is complicated to write a .value every time.
• It is exactly for interoperability with other systems.

[junkmd]

I PRed #475

[jaraco]

I don't understand why so many changes are being coupled with the drop py2 effort. I'd suggest simply dropping support for Python 2 and then incrementally remove python 2 compatibility. I notice these PRs are being targeted not at the mainline branch, which makes me worried that there's a lot of accumulated flux that's going to land all at once. I would recommend instead to make frequent releases with small changes to limit the amount of change any particular release creates (and make it easier to bisect any issues or regressions).

[junkmd]

I agree with removing this from the scope of drop_py2 as I commented below.

#475 (comment)