Error 126 from RegisterServiceCtrlHandlerEx

[zackmartin]

I am using Windows 7SP1 64bit (but 32-bit JVM). I was trying to use ntservice from the contrib directory, but couldn't get it to work. So I modified it slightly - added "throw LastErrorException" to all the API calls, and added some code to open a logfile and redirect stderr/stdout to it (since otherwise you can't get the output of the service). And I discovered that RegisterServiceCtrlHandlerEx fails with error 126 (The specified module could not be found). Investigating further, I believe what is happening is as follows:

1. RegisterServiceCtrlHandlerExW calls GetModuleHandleExW on the function pointer passed in
   -- I guess, for some unknown reason, it wants to know what DLL/EXE the pointer belongs to
2. With a normal service, this will return the module handle for some DLL or EXE
3. But with a JNA callback, this points to a memory buffer VirtualAlloc-allocated by FFI
4. So GetModuleHandleExW fails with 126 and RegisterServiceCtrlHandlerExW as a whole fails

Solution: It is a hard one. I guess the only way forward would be to modify FFI to write out the trampoline code to a DLL in a temporary directory, and then load that into memory. From reading the FFI code, it looks like it actually does this on some other platforms, but not windows. Obviously you would not want it to do this for every callback, but only for ones like this where the VirtualAlloc issue is a problem. But making modifications to the FFI code like this is a bit beyond me - maybe someone else will feel sufficiently inclined?

Since it looks like others have got this working before, maybe RegisterServiceCtrlHandlerExW calling GetModuleHandleExW is new code in some new versions of Windows.

[zackmartin]

BTW, I found the call to GetModuleHandleExW was falling using rohitab API Monitor.

[zackmartin]

I was thinking more about this, I was thinking a possible workaround might be to build a custom DLL with just two exports, one a global variable and the other a function. The function just does an indirect jump using the contents of the global variable. Then LoadLibrary the DLL, GetProcAddress the variable and stick the address of the native code of the JNA callback in there, then GetProcAddress the function and pass it to RegisterServiceCtrlHandlerExW. Hopefully then GetModuleHandleExW will work fine, since it will return the module handle for the custom DLL. In the process of trying this, I will let you know if it works or not.

[twall]

Is this specific to windows 7?

On Mar 16, 2012, at 8:19 PM, zackmartin wrote:

I am using Windows 7SP1 64bit (but 32-bit JVM). I was trying to use ntservice from the contrib directory, but couldn't get it to work. So I modified it slightly - added "throw LastErrorException" to all the API calls, and added some code to open a logfile and redirect stderr/stdout to it (since otherwise you can't get the output of the service). And I discovered that RegisterServiceCtrlHandlerEx fails with error 126 (The specified module could not be found). Investigating further, I believe what is happening is as follows:

1. RegisterServiceCtrlHandlerExW calls GetModuleHandleExW on the function pointer passed in
   -- I guess, for some unknown reason, it wants to know what DLL/EXE the pointer belongs to
2. With a normal service, this will return the module handle for some DLL or EXE

Normally most JNA-mapped functions requiring a module handle just use null when calling the function, which does the right thing. How did you determine GetModuleHandleExW was being called?

Are you certain that all required DLLs (including jnidispatch.dll) are available on PATH when the service is started? If not, you'd get the "module not found" error.

3. But with a JNA callback, this points to a memory buffer VirtualAlloc-allocated by FFI
4. So GetModuleHandleExW fails with 126 and RegisterServiceCtrlHandlerExW as a whole fails

I'm not certain the cause is what you think it is. Fundamentally, the ntservice bits register the JVM executable as the service entry point, and set up command-line arguments to configure it.

Solution: It is a hard one. I guess the only way forward would be to modify FFI to write out the trampoline code to a DLL in a temporary directory, and then load that into memory. From reading the FFI code, it looks like it actually does this on some other platforms, but not windows. Obviously you would not want it to do this for every callback, but only for ones like this where the VirtualAlloc issue is a problem. But making modifications to the FFI code like this is a bit beyond me - maybe someone else will feel sufficiently inclined?

Since it looks like others have got this working before, maybe RegisterServiceCtrlHandlerExW calling GetModuleHandleExW is need code in some new versions of Windows.

Might be specific to win7.

---

Reply to this email directly or view it on GitHub:
#62

[zackmartin]

Hello twall, I believe it probably is specific to Windows7. I am going to test things on WinXP soon, I am guessing on WinXP none of this will be necessary. I determined it was error 126 in three ways:

1. my declaration of RegisterServiceCtrlHandlerExW had "throw LastErrorException" on it, and that threw error 126
2. I used rohitab API monitor (http://www.rohitab.com/apimonitor) and I saw that RegisterServiceCtrlHandlerExW was
   calling GetModuleHandleExW which was failing with 126
3. I disassembled advapi32.dll, and by tracing things around I found that it does indeed end up calling GetModuleHandleExW for some reason

I am not sure WHY it would be calling GetModuleHandleExW. maybe it is related to some kind of internal diagnostics??? I can't pass null as the module handle, since I'm not calling it, Microsoft's own code is

I know I'm not missing "jnidispatch.dll", etc., since other Win32 API calls via JNA work fine, just not this one.

I've tested the "trampoline DLL" method I mentioned above, and it solves the issue for me. The DLL is pretty simple, it is written in assembly using MINGW:

.globl _TrampolinePtr
.bss
.align 4
_TrampolinePtr:
.space 4
.text
.globl _TrampolineCall
.def _TrampolineCall; .scl 2; .type 32; .endef
_TrampolineCall:
movl (_TrampolinePtr), %eax
jmp *%eax
.globl _DllMain
.def _DllMain; .scl 2; .type 32; .endef
_DllMain:
pushl %ebp
movl %esp, %ebp
movl $1, %eax
popl %ebp
ret $12
.section .drectve
.ascii " -export:TrampolineCall"
.ascii " -export:TrampolinePtr,data"

Then I LoadLibraryW it, GetProcAddress on both TrampolinePtr and TrampolineCall, put the pointer to the JNA callback in TrampolinePtr, and pass the pointer to TrampolineCall to RegisterServiceCtrlHandlerExW.

I actually copy the DLL to a temporary location with a random name. Then I modify the DLL to change its name to that, and use Imagehlp to fixup the checksum. Finally MoveFileExW is used to schedule the DLL for deletion at next reboot

I made sure to compile the DLL without using MSVCRT at all, I set my "DllMain" function as the entry point in the linker. Trying to load via LoadLibraryW a DLL with dependencies was causing some DLL dependency dramas, since I didn't need any dependencies I just got rid of the C runtime.

[twall]

This might be a way for JNA to provide callbacks for other routines (like keyboard/mouse hooks) which require callbacks to reside in a DLL if you can figure out a decent method of making this reusable or configurable. You'd probably just have to "pre-allocate" a dozen or so static hooks to a pool for JNA to use and have it fail if they've all been used up; I don't see how you could re-use any particular DLL address or write new ones to it. you'd then just embed these in the standard jnidispatch.dll to avoid additional issues writing/reading/disposing temporary files.

On Apr 2, 2012, at 1:30 AM, zackmartin wrote:

Hello twall, I believe it probably is specific to Windows7. I am going to test things on WinXP soon, I am guessing on WinXP none of this will be nexessary. As I mentioned, I determined it was error 126 in two ways:

1. my declaration of RegisterServiceCtrlHandlerExW had "throw LastErrorException" on it, and that threw error 126
2. I used rohitab API monitor (http://www.rohitab.com/apimonitor) and I saw that RegisterServiceCtrlHandlerExW was
   calling GetModuleHandleExW which was failing with 126
3. I disassembled advapi32.dll, and by tracing things around I found that it does indeed end up calling GetModuleHandleExW for some reason

I am not sure WHY it would be calling GetModuleHandleExW

I know I'm not missing "jnidispatch.dll", etc., since other Win32 API calls via JNA work fine, just not this one.

I've tested the "trampoline DLL" method I mentioned above, and it solves the issue for me. The DLL is pretty simple, it is written in assembly using MINGW:

.globl _TrampolinePtr
.bss
.align 4
_TrampolinePtr:
.space 4
.text
.globl _TrampolineCall
.def _TrampolineCall; .scl 2; .type 32; .endef
_TrampolineCall:
movl (_TrampolinePtr), %eax
jmp *%eax
.globl _DllMain
.def _DllMain; .scl 2; .type 32; .endef
_DllMain:
pushl %ebp
movl %esp, %ebp
movl $1, %eax
popl %ebp
ret $12
.section .drectve
.ascii " -export:TrampolineCall"
.ascii " -export:TrampolinePtr,data"

Then I LoadLibraryW it, GetProcAddress on both TrampolinePtr and TrampolineCall, put the pointer to the JNA callback in TrampolinePtr, and pass the pointer to TrampolineCall to RegisterServiceCtrlHandlerExW.

I actually copy the DLL to a temporary location with a random name. Then I modify the DLL to change its name to that, and use Imagehlp to fixup the checksum. Finally MoveFileExW is used to schedule the DLL for deletion at next reboot

I made sure to compile the DLL without using MSVCRT at all, I set my "DllMain" function as the entry point in the linker. Trying to load via LoadLibraryW a DLL with dependencies was causing some DLL dependency dramas, since I didn't need any dependencies I just got rid of the C runtime.

---

Reply to this email directly or view it on GitHub:
#62 (comment)

[twall]

Here's the native junk required for the DLL-embedded jumps; it establishes four "trampolines" which would be returned in place of the heap-allocated callback trampolines. MASM is required on win64/MSVC (yuck):

static void (*fn[4])();

extern void asmfn0(int,char**);
extern void asmfn1(int,char**);
extern void asmfn2(int,char**);
extern void asmfn3(int,char**);

#if defined(__x86_64__)
#ifdef _MSC_VER
/* No amd64 support for inline asm */
#else
#define ASMFN(X) asm(".globl _asmfn" #X "\n\                                    
_asmfn" #X ":\n\                                                                
 jmp *(8*" #X ")+_fn(%rip)")
#endif
#else /* __x86_64 */
#ifdef _MSC_VER
#define ASMFN(X) \
__asm asmfn ## X PROC NEAR \
__asm jmp fn[X]
#else
#define ASMFN(X) asm(".globl _asmfn" #X "\n\                                    
_asmfn" #X ":\n\                                                                
 jmp *(_fn+4*" #X ")")
#endif
#endif /* __x86_64 */

ASMFN(0);
ASMFN(1);
ASMFN(2);
ASMFN(3);

[twall]

Added branch dll-callbacks. Working functionality, pending build fixes to work properly with MS tools in addition to mingw64.

[twall]

Please re-test against dll-callbacks branch @zackmartin