Native Multitouch for AWT component (Windows)
This page documents the process of enabling an AWT/Swing component to receive touch events (WM_TOUCH) in Windows using JNA.
Neither AWT nor Swing (which is built on top of AWT) support multitouch out of the box.
Multitouch for Java (MT4J) supports multitouch on windows according to the FAQ at their project site. Unfortunately we weren't able to get MT4J to work, nor could we find documentation on how they got their multi-touch to work.
TUIO is a generic touch interface which sends touch events over UDP. By drawing a native window over the java window and using this window to capture native touch events and send TUIO events, a java application can become touch-enabled. The program Touch2TUIO can be used to accomplish this. Unfortunately we were unable to run this program on our target device. Furthermore, we expect this approach to be problematic for full-screen windows.
JNA offers quite some support for the Windows API already, having many types already defined. For our multitouch implementation we used JNA 3.5.2.
Windows does inter-window communication using messages. Each window
has a handle. Using the Windows function
SendMessage
(which is actually either SendMessageA or SendMessageW depending
on whether ascii or unicode is used), a message can be sent to a
window by passing this function the window handle, a message type
(specified by an int) and two parameters, wParam and lParam whose
precise meaning depends on the message type.
In order to get touch events in an AWT window, the following things have to happen:
- Retrieve the native window handle associated with an AWT window.
- Use the handle to register the window as a touch-enabled window.
- Install a message hook for this window to intercept the WM_TOUCH messages.
- Parse the message.
In order to manipulate the AWT component using native Windows
functionality, we first need a window handle. This StackOverflow
question
describes a method to retrieve the native window handle for any AWT
window. I'll describe this method shortly here as well. AWT maps each
java component to a peer class, a java wrapper around
the native component. For Windows, each of these peers extends
sun.awt.windows.WComponentPeer. This class has a function,
getHWnd() which returns the native pointer to this handle as a long.
In order to use this handle nicely with JNA we need to convert this
pointer to JNA's handle type, WinDef.HWND. This is done by first
making it a Pointer (which takes the long pointer in its
constructor), then passing the Pointer to the constructor of
WinDef.HWND.
We now have a handle to the native window, nicely wrapped in a JNA object.
By default, windows doesn't pass WM_TOUCH messages to a window, but
WM_GESTURE messages. In order to receive WM_TOUCH messages, a window
needs to be registered as a touch window using
RegisterTouchWindow. This
function takes two arguments, the window handle and a bitmap
specifying extra options.
Unfortunately this function has to be called on the same thread that created the window. for AWT, this is a thread called 'AWT-Windows'. This is NOT the AWT event dispatch thread, and there's no way to access this thread from pure java.
Luckily there exists a way to run code on this thread, but it requires
Windows messages. AWT creates a special toolkit window (class
'SunAwtToolkit', name 'theAwtToolkitWindow') and defines a bunch of
custom message types that can be sent to this window to run various
AWT actions on the AWT-Windows thread. This message handling happens
in awt_Toolkit.cpp (located in the openJDK source at
jdk/src/windows/native/sun/windows/awt_Toolkit.cpp). One of the custom
messages is WM_AWT_INVOKE_VOID_METHOD, defined in awtmsg.h as part
of an enum as 0x982a. Upon receiving WM_AWT_INVOKE_VOID_METHOD
message, AWT will interpret the wParam as a function pointer and
invoke the function placed there as a void function.
In order to send a message to this window we first need to retrieve a
handle to it. Windows has a function for this,
FindWindow,
which takes a window class and a window name and returns a window
handle. For the AWT toolkit window the window class is 'SunAwtToolkit'
and the window name is 'theAwtToolkitWindow'. When the handle is
retrieved, a message can be sent to the AWT window using
SendMessage, described above.
Using JNA, we can create a
Callback
that calls RegisterTouchWindow for our AWT window. Also using JNA,
we can define a version of the SendMessage that takes our Callback
as the wParam. This version can then be used to send the required
message to the AWT toolkit window to register our AWT window as a
touch window.
Explaining JNA is out of this document's scope, but the method we used can be looked up in the source code.
In order to capture the WM_TOUCH messages which should now be sent to
the AWT window, a mesage hook needs to be installed. Windows' hook
system is explained in detail
here. The
function required to set a hook is
SetWindowsHookEx
(or actually either SetWindowsHooksExA or SetWindowsHooksExW
depending on whether ascii or unicode is used). This function takes a
hook type (Which should be WH_GETMESSAGE (which has value 3)), a
pointer to a hook callback, a DLL handle (unnecessary in this case, so
should be NULL) and a thread ID for the thread on which the hook needs
to be installed. This hook needs to be installed on the thread
associated with the AWT window. In order to retrieve this thread ID,
GetWindowThreadProcessId
can be used. This function takes a window handle and optionally a
pointer to a variable to write the process identifier (we don't need
that, so we pass NULL).
The hook callback must be of type
GetMsgProc. This
function receives 3 arguments, but only the last argument (lParam)
is of interest to us, as it contains a pointer to the message that was
sent to the AWT window. JNA already has a type for this message
(WinUser.MSG),
so using JNA a StdCallCallback can be defined which takes a
WinUser.MSG as third argument.
After setting up the hook using SetWindowsHookEx, the callback will
be called for each message, including many messages that are not
related to touch input. In order to check whether or not a message is
of interest to us, the hook callback should check the message field
of the received WinUser.MSG object. If this is equal to 0x240 (which
is what WM_TOUCH is defined as by Windows), then this message is about
touch input.
In order to make sure that the AWT window still receives
mouse and keyboard input in the usual way, the hook callback should
call
CallNextHookEx
and return the value returned by this function. CallNextHookEx takes
4 arguments, the first of which is always ignored and the last 3 are
the same arguments that were also passed to the hook callback.
It is very important to keep a reference to both the hook handle
returned by SetWindowsHookEx and to the callback passed to
SetWindowsHookEx. We need to keep the callback to prevent the JVM
from garbage-collecting it, and we need to keep a reference to the
hook handle so that we can unhook it when our AWT window closes.
Finally, there's some evidence that suggests that the hook should not
be set on a thread that is destroyed later on while the hook is still
required. Our hook seemed to become invalid when SetWindowsHookEx
was called on a clojure nREPL thread (which is regularily destroyed),
but remained valid when SetWindowsHookEx was called on the
AWT-Windows thread using the void function procedure described above.
As stated above, our callback receives a MSG as its third argument,
and this MSG is a
WM_TOUCH
message if its message field is 0x240.
A WM_TOUCH message has two parameters, wParam and lParam. The two
least significant bytes of wParam represent the amount of touch
points that are currently being registered. lParam contains a touch
input handle that can be passed to
GetTouchInputInfo.
GetTouchInputInfo requires 4 arguments: the touch input handler
described above, the amount of inputs (which can be found in wParam
as described above), a pointer to an array of
TOUCHINPUT
structures, and the size of a single TOUCHINPUT structure.
In order to use GetTouchInputInfo with JNA, a TOUCHINPUT structure
needs to be defined. This can be done by extending Structure as
described in the JNA documentation
here.
This Structure should simply map the TOUCHINPUT structure as defined
on MSDN
here.
When that Structure is defined, an array can be constructed using the
toArray
method, and the size of a TouchInput structure can be retrieved using the
size
method.
Having described how to get a touch input handle, the amount of touch
points currently registered, an array of TOUCHINPUT structures and the
size of a single TOUCHINPUT, we now have all information required to
call GetTouchInputInfo. After calling this, our array of JNA
TOUCHINPUT objects will contain the touch input info.
The most interesting fields of each TOUCHINPUT structure are x, y,
dwID, dwFlags and dwTime. x and y represent the touch
coordinate. dwID is an identifier identifying one particular touch
input consistently over multiple WM_TOUCH messages (from the moment a
finger/pen is placed to the moment it is lifted). dwFlags is a
bitfield representing whether this is an UP, DOWN or MOVE event (among
other things) and dwTime is a timestamp in milliseconds for when
this event happened.
dwFlags, as mentioned above, is a bitfield. from least-significant
to most-significant, these bits are MOVE (0x1), DOWN (0x2), UP (0x4),
and various others that aren't very important to us. They're all
listed in the Remarks section on the MSDN entry for TOUCHINPUT.