Render to QtQuickWindowSurface

[drlukeparry]

Firstly I wish to share a big thank you for your really great contribution and work across PyGFX. For the first time, I have come across a viable alternative 3D UI framework for general programming and also specifically for use with Python. Generally, there are really are no alternatives except three.js (Three React Fiber), which are inadequate for both scientific and desktop programming. Additionally, it would be nice in future if there was a community section on your website to share examples usage.

Specifically on the topic:

I am working on integrating pygfx into a qml/qtquick environment. Initial tests are quite promising.

The reason behind is QtQuick3D is clunky, difficult to customise (especially for scientific applications) and shares the similar issues with Qt3D, which I previoulsy used in the past. Also its license is undesirable.

I would like to directly use the native render surface that can be drawn asynchronously with a QtQuick environment compared to a QtWidget backend.

https://doc.qt.io/qtforpython-6/PySide6/QtQuick/QQuickWindow.html#PySide6.QtQuick.QQuickWindow.graphicsApi

I have found it is possible to render this via a QImage on a paintEvent in QQuickPaintedItem but appears slow and is both CPU/GPU heavy during dynamic call canges. I was wondering whether via QRhi from PySide6, it is possible to obtain the render context and directly draw within the QtQuickAppliction window like below:

https://doc.qt.io/qtforpython-6/examples/example_quick_scenegraph_openglunderqml.html

and a follow-on QRhi example:

https://doc.qt.io/qtforpython-6/examples/example_gui_rhiwindow.html

Thank you once again,
Luke

[almarklein]

Hi Luke,

I am working on integrating pygfx into a qml/qtquick environment. Initial tests are quite promising.

Interesting!

Render via a bitmap

A good first step would be to render via an image, this can probably be done similar to how we do it with the QRenderWidget:

rendercanvas/rendercanvas/qt.py

Lines 305 to 318 in 9fc9627

	def _rc_get_present_methods(self):
	global _show_image_method_warning
	if self._surface_ids is None:
	self._surface_ids = self._get_surface_ids()
	methods = {}
	if self._present_to_screen:
	methods["screen"] = self._surface_ids
	else:
	if _show_image_method_warning:
	logger.warning(_show_image_method_warning)
	_show_image_method_warning = None
	methods["bitmap"] = {"formats": list(BITMAP_FORMAT_MAP.keys())}
	return methods

and

rendercanvas/rendercanvas/qt.py

Lines 336 to 367 in 9fc9627

	def _rc_present_bitmap(self, *, data, format, **kwargs):
	width, height = data.shape[1], data.shape[0] # width, height
	rect1 = QtCore.QRect(0, 0, width, height)
	rect2 = self.rect()
	painter = QtGui.QPainter(self)
	# backingstore = self.backingStore()
	# backingstore.beginPaint(rect2)
	# painter = QtGui.QPainter(backingstore.paintDevice())
	# We want to simply blit the image (copy pixels one-to-one on framebuffer).
	# Maybe Qt does this when the sizes match exactly (like they do here).
	# Converting to a QPixmap and painting that only makes it slower.
	# Just in case, set render hints that may hurt performance.
	painter.setRenderHints(
	painter.RenderHint.Antialiasing | painter.RenderHint.SmoothPixmapTransform,
	False,
	)
	qtformat = BITMAP_FORMAT_MAP[format]
	bytes_per_line = data.strides[0]
	image = QtGui.QImage(data, width, height, bytes_per_line, qtformat)
	painter.drawImage(rect2, image, rect1)
	# Uncomment for testing purposes
	# painter.setPen(QtGui.QColor("#0000ff"))
	# painter.setFont(QtGui.QFont("Arial", 30))
	# painter.drawText(100, 100, "This is an image")
	painter.end()

Not sure if this is what you meant with "render this via a QImage on a paintEvent in QQuickPaintedItem ".

Render to screen

As for rendering directly to the window ... my knowledge of QtQuick / QRhi etc is near zero. I'll just share some ideas and questions.

I was wondering whether via QRhi from PySide6, it is possible to obtain the render context

From what I can see you can set the graphicsApi to e.g. Metal / 3D12 / Vulkan, but I don't think that will help because wgpu hides the backend (metal/vulkan) and does not support sharing native handles (yet).

In other words, even if you get context information, like a Vulkan surface, that won't help, because wgpu has its own opaque surface object. Under the hood that will be a Vulkan surface, but there's no way for us to pass that Vulkan surface into wgpu.

So my impression is that the best way to render directly to the window would be basically the same as we do for the QRenderWidget.

I wonder what happens when you set the graphics api to QSGRendererInterface.Unkown or .Null, and whether it will help / be necessary.

Can you not simply embed a QRenderCanvas in a qtQuick application?

[drlukeparry]

Thank you for your detailed reply.

Yes it was as you suggested for the first example. It does work, although the copy of the buffer in _rc_present_bitmap is undesirable as acknowledged, whichever approach you take at this stage.

QRenderClass (as derived from QWidget) cannot sadly be incorporated in the QtQuick Scenegraph in Qt6 (at one point I think there was a proxy widget in earlier releases of Qt5).

Researching a bit more into what you said, there are no direct access to the texture or buffers in the context created by the webgpu-core. I believe the lack of an inter-op API for sharing the texture or buffers is not currently possible, as you previously referred to (gfx-rs/wgpu#4067) which would accomplish the following: https://doc.qt.io/qtforpython-6

For future reference, QtQuick can use a QRhi renderTarget if there is an externally managed texture (Vulkan, Metal, DirectX), but I do not think that we can get the pointer for this resource at the moment.

[almarklein]

QRenderClass (as derived from QWidget) cannot sadly be incorporated in the QtQuick Scenegraph

And I don't suppose we can get the window-id for the region that you want to draw to?

[drlukeparry]

In QtQuick (v6.8) the window handle is provided, and there are ways to get (or set) the render surface/texture target handle, if the graphics backend is fixed or known from the operating environment.

[almarklein]

In QtQuick (v6.8) the window handle is provided

If this returns the same as https://doc.qt.io/qt-6/qwidget.html#winId, we can use it to render to screen, but I guess it's not?

[drlukeparry]

The QQuickWindow (https://doc.qt.io/qt-6/qquickwindow.html) is a good potential bet (even now or in the long-term). This provides access to window events if needed additionally. Long-term, once there is access to the native backend context, texture creation then this can be used directly (example for DirectX is provided on the page.

The native QWindow can be accessed via its inherent parent method, that provides access to winId via QWindow class.

[almarklein]

The bitmap present mode will likely become a much more viable option: #66

[skoudoro]

By curiosity, what is the status of this ?

@drlukeparry, do you have an alternative/temporary solution for now?

Thanks all for this interesting thread

[drlukeparry]

By curiosity, what is the status of this ?

@drlukeparry, do you have an alternative/temporary solution for now?

Thanks all for this interesting thread

Unfortunately not a pretty solution and requires GPU->CPU->GPU transfer by using QPainter to draw the WGPU canvas buffer to the primitive shape for a QQmlElement within a QMLScene. There are no other solutions at the moment as there is no native raw pointer handles to the underlying GPU textures within WGPU for direct access when using QRHI directly.

    def paint(self, painter: QPainter):

        self.main() # do the animation and render logic
        frame = self._canvas.draw()

        width, height = frame.shape[1], frame.shape[0]
        self.img  = QImage(frame, width, height, 4 * width,  QImage.Format_RGBA8888)


        itemSize = self.size()
        bbox = self.boundingRect()
        painter.drawImage(bbox, self.img)

[Vipitis]

Might be a tad offtopic, but there is a whole QShader module which I heard about recently. I don't know how complex it would be but maybe you can export the spirv module from wgpu and hand it to qt directly?