Allow creating images from raw image sources

[freakboy3742]

If you're calling a native platform API, you'll get back an image in native format (e.g., UIImage on iOS, NSImage on macOS, etc). At present, there's no easy way to create a toga.Image from that raw representation - you have to convert it to bytes, then create a toga.Image from the raw byte data... which will create a UIImage/NSImage etc.

This PR allows the platform's native image format as a valid source type when creating an Image; if provided, it's essentially a shortcut to "use this object as the native representation of the image."

Testing this required a couple of tweaks to the dummy platform's Image representation to give it a unique data type.

PR Checklist:

• All new features have been tested
• All new features have been documented
• I have read the CONTRIBUTING.md file
• I will abide by the code of conduct

[freakboy3742]

@rmartin16 If you're looking for a hard-mode typing challenge: How should the type annotation on src be modified to allow for this? It's a type on a class that, by definition, is platform specific, and won't exist in the environment where toga-core is being built, tested, or documented.

The only solution I can think of is to add UIImage | NSImage | GdkBitmap | ... but that won't be correct - you can't use a UIImage unless you're on iOS.

Any ideas?

[rmartin16]

I think this might be possible....but it, perhaps, starts to verge on the ridiculous....

For instance, here's where I got to before I, more or less, gave up:

if TYPE_CHECKING:
    BaseImageSrcT: TypeAlias = Union[
        Path,
        bytes,
        bytearray,
        memoryview,
        "Image",
        PIL.Image.Image,
        None,
    ]

    if sys.platform == "win32":
        from toga_winforms.images import WinImage
        ImageSrcT: TypeAlias = Union[BaseImageSrcT, WinImage]

    if sys.platform.startswith("linux"):
        from toga_gtk.images import GdkPixbuf
        ImageSrcT: TypeAlias = Union[BaseImageSrcT, GdkPixbuf.Pixbuf]

    if sys.platform == "darwin":
        from toga_cocoa.images import NSImage
        ImageSrcT: TypeAlias = Union[BaseImageSrcT, NSImage]


class Image:
    def __init__(self, src: ImageSrcT): ...

I'm not sure if mypy technically considers this strategy above part of their larger type narrowing support, but it helps accommodate the runtime environment.

That said, for this to really make mypy happy, it'll require other changes in Toga...notably, replacing the star imports in the platform-specific Toga code to explicitly import these native library classes....and we'll also need typing stubs for, at least, PyGObject.

BUT...this has major consequences for building docs. On my Linux machine, it tries to interpret the definition of ImageSrcT for Linux but cannot find toga_gtk; I can install toga_gtk to build the docs....but then the build blows up with RuntimeErrors about DISPLAY env var. Maybe Sphinx has some buried support for effectively ignoring certain symbols...and that could serve as some type of resolution here.

At any rate, the src argument feels like a good candidate for Any where the documentation fills in the details.

P.S. - type checkers and IDEs are still likely to resolve the type of src to Any even with all these accommodations if Pillow isn't installed. And if we just add these platform-specific image classes, such as GdkPixbuf.Pixbuf, then the type will always be Any because it would be impossible to resolve all the types in the Union on any one system.

[freakboy3742]

I think this might be possible....but it, perhaps, starts to verge on the ridiculous....

For instance, here's where I got to before I, more or less, gave up:

That was broadly my initial thought as well - except that it falls down on iOS/Android (which will never be installed in the local environment where sys.platform will find it), and on custom backends, and in the Sphinx builds you mention later.

That said, for this to really make mypy happy, it'll require other changes in Toga...notably, replacing the star imports in the platform-specific Toga code to explicitly import these native library classes....and we'll also need typing stubs for, at least, PyGObject.

Woof... good luck with that endeavour... I like yaks as much as the next person, but autogeneration of types from GI bindings is a yak to far for me :-)

At any rate, the src argument feels like a good candidate for Any where the documentation fills in the details.

Is there any advantage to specifying the type at all in this case? What (if any) benefit exists to an explicit Any over "no type declaration"? Cosmetically, "Any" reads to me as "You can use literally anything", which isn't true (at least, not in spirit). This is a case where I think I'd prefer to say nothing, and force the reader to actually read words, rather than provide something explicitly too broad as an automated description.

[rmartin16]

Is there any advantage to specifying the type at all in this case? What (if any) benefit exists to an explicit Any over "no type declaration"? Cosmetically, "Any" reads to me as "You can use literally anything", which isn't true (at least, not in spirit). This is a case where I think I'd prefer to say nothing, and force the reader to actually read words, rather than provide something explicitly too broad as an automated description.

I think it depends on your goals.

As far as typing is concerned, an untyped argument will be resolved as Any...so, in that regard, adding Any or not in this case is functionally the same.

However, the absence of a type can mean different things. It could mean this function hasn't been evaluated for typing; this would be useful when gradually typing a code base and using mypy's functionality to ignore untyped functions and methods. Although, as you mention, if someone takes Any seriously, then they will be especially misled on what this argument can accept.

That said, given this is part of the API surface, I think it will be useful to ensure it has a type assigned....so, we can know the type has been assessed.

So, thinking a bit out of the box perhaps, you could create a NewType called ImageDataT that's just assigned to Any. In that way, it's an indication a specific type is being requested....it just isn't expressed in Python types.

[freakboy3742]

Replacing all the specific Image classes with an ImageT that maps to Any seems like a decent option that is going to render well as documentation, and won't be intractable from a mypy perspective. I'll go with that.

[rmartin16]

Additionally, fwiw, using NewType wouldn't really work either....since it would, in fact, require users to explicitly cast the image data as ImageType which wouldn't be a great experience at all.

[freakboy3742]

Ugh... fine, I guess we'll just go with Any in the union then... le sigh...

[mhsmith]

The docs still look quite cluttered with everything in the union. Can we just make ImageType an alias for Any, and then describe all the options in its docstring?

[freakboy3742]

See the discussion above - that's where I started, and it looks awful; it renders in Sphinx as NewType("Image", Any), with the sphinx link to "NewType" being to the Python docs. AFAIK, it's not possible to provide documentation for the type itself (which would be the ideal situation).

I'm open to other suggestions, but putting Any in the union is the only version I could find that mollifies mypy/PyCharm and doesn't read like a train wreck.

[mhsmith]

Since a union that contains Any no longer provides any type-checking safety, and there are so many things in the union that it isn't very good documentation either, my suggestion was to just replace the whole union with Any, and rely on the documentation text to explain what's accepted.

Ideally we would do this with an alias, but if Sphinx doesn't work with that, we can use Any directly, and put the documentation in its own section on the Image page with incoming links from all the relevant places.

[mhsmith]

Decapitalize, since the type probably isn't literally called Image.

Suggested change

	format, an instance of the platform's native Image type, or
	format, an instance of the platform's native image type, or

[mhsmith]

Although, if this is now part of the public API, we should probably list the native types explicitly, rather than making the reader guess.

[mhsmith]

Oh, I see them now, they're on the Image page, but not referenced from the constructor docstring. If we can have an ImageType alias linked from both Image and ImageView, that would be a good place to move that list.

The list of file formats (PNG, JPG, etc.) should probably stay where it is, because they apply to saving images as well as loading them. But there should be a link to that list from each relevant place.

[freakboy3742]

I was on the fence about documenting the internal types at all. As an end user, unless you're reaching into the native layer to implement a feature, you shouldn't need to know the internal types. On that basis, I'm OK with this being a little obscure.

As mentioned in the other comment about types, I'm not aware of any way that we can structure this in Sphinx that is readable and mollifies type checkers; including not being able to document the actual type. I'm open to any suggestions.

[mhsmith]

unless you're reaching into the native layer to implement a feature, you shouldn't need to know the internal types

We recommend people to reach into the native layer on a fairly regular basis, so we should document this properly. Saying "Can also accept the platform's native image format" without clearly indicating what that means is just going to annoy the reader. A link to the list would be enough to fix this.

[freakboy3742]

Following the type documentation wrangling I did for #2259, I've adopted the same typing and documentation pattern for Image. This means there's some overlap in the patch (e.g., the updates to conf.py).

[mhsmith]

This is imported in one place, and added to the spelling wordlist, but It doesn't looks like it's actually being used.

[freakboy3742]

It was pre-existing; it's used in all the as_image() (canvas, image view) and as_format() (Image) calls.

It's in the dictionary now because the default sphinx handling of type annotations apparently considers the type to be a word that needs to be spell checked. The previous plugin didn't do this.