Semantic Highlighting for C Extensions

[akern40]

Hello! I do not know whether this is a feature request or a bug report, but I cannot seem to get semantic highlighting to work with a Python C extension. The structure of the C extension package is listed below. I can't find documentation on whether this is a bug in my C extension project directory, whether pylance simply doesn't support semantic highlighting for C extensions, or whether I need to add some sort of stub files in order to support semantic highlighting, like mypy's .pyi stub files. Any advice is greatly appreciated!

Environment data

• Language Server version: 2022.3.0
• OS and version: win32 x64
• Python version (and distribution if applicable, e.g. Anaconda): Anaconda 4.10.1, Python 3.7
• python.analysis.indexing: null
• python.analysis.typeCheckingMode: off

Expected behaviour

When the C extension - built as a separate package - is imported and used in a codebase, it should receive the same semantic highlighting as any other Python package.

Actual behaviour

The module name is highlighted correctly, but none of its methods have a semantic token type.

Logs

Python Language Server Log (Anonymized in places for privacy)

[Info  - 3:21:43 PM] (22440) Pylance language server 2022.3.0 (pyright 1ed262f6) starting
[Info  - 3:21:43 PM] (22440) Server root directory: c:\Users\username\.vscode\extensions\ms-python.vscode-pylance-2022.3.0\dist
[Info  - 3:21:43 PM] (22440) No configuration file found.
[Info  - 3:21:43 PM] (22440) pyproject.toml file found at c:\Users\username\VCProjects\sandbox.
[Info  - 3:21:43 PM] (22440) Setting pythonPath for service "sandbox": "C:\Users\username\.conda\envs\my-env\python.exe"
[Info  - 3:21:43 PM] (22440) Loading pyproject.toml file at c:\Users\username\VCProjects\sandbox\pyproject.toml
[Error - 3:21:43 PM] (22440) Pyproject file "c:\Users\username\VCProjects\sandbox\pyproject.toml" is missing "[tool.pyright]" section.
[Warn  - 3:21:43 PM] (22440) stubPath c:\Users\username\VCProjects\sandbox\typings is not a valid directory.
[Info  - 3:21:43 PM] (22440) Assuming Python version 3.7
[Info  - 3:21:43 PM] (22440) Assuming Python platform Windows
[Info  - 3:21:44 PM] (22440) Searching for source files
[Info  - 3:21:44 PM] (22440) Found 13 source files
[Info  - 3:21:44 PM] (22440) Background analysis(1) root directory: c:\Users\username\.vscode\extensions\ms-python.vscode-pylance-2022.3.0\dist
[Info  - 3:21:44 PM] (22440) Background analysis(1) started
[Info  - 3:21:48 PM] (22440) [BG(1)] Long operation: checking: c:\Users\username\VCProjects\sandbox\src\sandbox\basis.py (2984ms)
[Info  - 3:21:48 PM] (22440) [BG(1)] Long operation: analyzing: c:\Users\username\VCProjects\sandbox\src\sandbox\basis.py (3356ms)

Code Snippet / Additional information

Structure of C extension code (with omissions and renames for privacy purposes):

my_module
│   LICENSE
│   README.md
│   setup.py
├───doc
├───include
│       my_module.h
│
└───src
    │   my_module.c
    │   file_1.c
    │   ...
    │
    ├───my_module <-- This must be here in order to get semantic highlighting for the module name
    └───python
            my_modulemodule.c

[erictraut]

@akern40, when you say that you want "syntax highlighting for your extension", are you saying that you want the C++ code to be displayed with semantic highlighting? Or are you saying that you want Python code that imports symbols from your extension to have the proper semantic highlighting and types? If it's the former, then you'll need to install a C++ language server because pylance handles only Python source files. If it's the latter, then you'll need to provide type stubs that describe the interface of your C++ code in a way that a Python static type checker can understand.

[akern40]

Thanks for clarifying @erictraut - definitely the latter. Does Pylance require the type stubs to be located in a particular place in the project directory, or formatted a particular way?

[erictraut]

The format of a type stub is dictated by PEP 484. They use standard Python syntax, but they typically omit the implementation details and replace them with .... For example type stubs, you could check out the typeshed repo. You might also find this documentation and this documentation useful.

Pylance follows the import resolution rules specified in PEP 561, which defines the locations where type checkers are supposed to look for stubs and source files when resolving import statements. This documentation might also be useful.

[akern40]

Thank you, those references were tremendously helpful. For anyone writing stubs for C extensions who may come across this, I ended up moving the empty my_module directory to the top level and adding an empty __init__.py under that folder. I then added an __init__.pyi in the same folder, and that is where I placed my type stubs.