[BUG] not None semantics is not supported in Pure Python mode for memory views

[matusvalo]

Describe the bug

Pure python mode does not support not None when used with memory views. See examples below.

Code to reproduce the behaviour:

import cython
import typing
import numpy as np

def process_buffer(input_view: cython.int[:,:],
                   output_view: typing.Optional[cython.int[:,:]] = None):

    if output_view is None:
        # Creating a default view, e.g.
        output_view = np.empty_like(input_view)
    
    # process 'input_view' into 'output_view'
    return output_view

process_buffer(None, None)

Note
There is additional issue that memoryviews does not support typing.Optional(). Compiling the example fails with error: typing.Optional[...] cannot be applied to non-Python type int[:]

Expected behaviour

TypeError is raised in the same way as it is raised in cython version:

import numpy as np

def process_buffer(int[:,:] input_view not None,
                   int[:,:] output_view=None):

   if output_view is None:
       # Creating a default view, e.g.
       output_view = np.empty_like(input_view)

   # process 'input_view' into 'output_view'
   return output_view


process_buffer(None, None)

Also typing.Optional[] of memoryview succeeds without error.

OS

any

Python version

3.9

Cython version

master

Additional context

Similar semantics is already present in Extension types: https://cython.readthedocs.io/en/latest/src/userguide/extension_types.html#extension-types-and-none

[da-woods]

The difficulty is backwards compatibility (i.e. Cython 3 was probably the time to make this change is we wanted to). I think we'd probably have to introduce a cython.NonOptional decorator (name to be debated...) rather than change the behaviour of existing annotated memoryviews.

We should definitely make sure typing.Optional[cython.int[:]] is accepted (if it isn't already)

Much as it'd be nice to make it consistent.

[scoder]

As much as I'd dislike making incompatible changes at this point, we could still admit a mistake and change it right in 3.0.1, so that users can depend on that onwards. Pure Python mode hasn't been used much in the past, so we're mostly talking about newly written, still maintained code here that doesn't need to work in Cython 0.29.x. Also, this is the behaviour that most users would expect or at least want, so it's not like we're really talking something away. I'd certainly prefer using the standard Optional[] mechanism over anything special that we could come up with ourselves. Special cases aren't special enough to break the rules. This one really doesn't feel like it should be special.

[matusvalo]

OK I have created PoC PR fixing this issue. Not tested yet. I will wait for final decision whether we will go with it to 3.0.1 (as @scoder suggested) or not.

[scoder]

On a loosely related note, should we allow Optional[cpointer] as well, as a no-op? It seems reasonable to me to allow explicitly marking NULL as a valid value for pointers. I wouldn't want to reject NULL if Optional is not used, but IMHO, there's nothing wrong with an explicit Optional[cpointer].

Or do you think that we should limit the applicable range of Optional to "things that can actually be None"?

[matusvalo]

On a loosely related note, should we allow Optional[cpointer] as well, as a no-op? It seems reasonable to me to allow explicitly marking NULL as a valid value for pointers. I wouldn't want to reject NULL if Optional is not used, but IMHO, there's nothing wrong with an explicit Optional[cpointer].

I don't have strong opinion about this. But I would rather do it later not in this PR or 3.0.1 release (to keep it light). This change is not breaking anything so can be added anytime.