Shell extension does not unload from explorer.exe when unregistered from the system

[end2endzone]

Describe the bug
After the shell extension is unregistered from the system, explorer.exe does not unload sa.shellextension.dll.

To Reproduce
Steps to reproduce the behavior:

1. Install as normal
2. Register the shell extension by running register.bat inside an administrator command prompt.
3. Right-click on a directory to show that the shell extension is registered on the system and loads additional context menus.
4. Unregister the shell extension by running unregister.bat inside an administrator command prompt.
5. Try to uninstall the application.
6. Windows Installer refuse to proceed with the uninstallation and says Windows Explorer is using the files.

Expected behavior
Explorer.exe should release/unload shell anything dlls when unregistering ShellAnything.

Screenshots
After unregistering the application, if we try to uninstall the application, the following dialog is displayed:

Environment

• OS: Windows 10 64 bit

Additional context
N/A

[end2endzone]

Files are not unloaded after unregistration according to Process Explorer:

[end2endzone]

From https://learn.microsoft.com/en-us/previous-versions/windows/desktop/legacy/cc144064(v=vs.85)?redirectedfrom=MSDN :

The Shell automatically unloads any DLL when its usage count is zero, but only after the DLL has not been used for a period of time. This inactive period might be unacceptably long at times, especially when a Shell extension DLL is being debugged. You can shorten the inactive period by adding the following information to the registry.

  HKEY_LOCAL_MACHINE
   Software
      Microsoft
         Windows
            CurrentVersion
               Explorer
                  AlwaysUnloadDll

[end2endzone]

From https://www.tenforums.com/performance-maintenance/163171-unload-dll-ram-memory-tweak-question.html :

This is one of those useless tweaks that has been around since at least XP days. It has largely disappeared but can still be found. At least it is harmless. And finally, the setting hasn't been supported since Windows 2000. Unsupported registry entries are silently ignored.

[end2endzone]

Some thoughts about why this might be occurring:

1. ShellAnything recently moved to dynamic linked libraries (DLL) for GLOG and libexprtk libraries (ShellAnything API in C #103). There might be a hook or something similar which prevents explorer.exe to unload the main DLL because it can't also unload glog.dll or libexprtk.dll.
2. ShellAnything uses its own DLL registering and unregistering code.
   • The first builds were compiled on Visual Studio Express which did not provide a version of [MFC and ATL] (https://learn.microsoft.com/en-us/cpp/mfc/mfc-and-atl) so manual DLL registration/unregistration was needed.
   • The code should be updated now that Visual Studio Community is available with support for ATL.
3. In order to force explorer.exe to "refresh" shell extensions and unload the ones that are unregistered, the code has to "notify" Windows about the change. This is implemented in file shellext.cpp in functions DllRegisterServer() and DllUnregisterServer() with a call to SHChangeNotify(SHCNE_ASSOCCHANGED, 0, 0, 0); (version 0.7.0 and current version). A new API call or a new flag might be required in order to notify explorer.exe.
4. Microsoft might have implemented a new feature in Windows 10 or 11 (after Windows 7) that might conflict with our code. The result being that explorer.exe never unloads sa.shellextension.dll after it is being unregistered.

EDIT:

1. Invalid. See comment below.
2. Invalid. Following commit 22d3d87, ShellAnything uses ATL for registering the shell extension. The *.rgs recipe file is still manually written but its structure matches other reputable sources such as:
   • https://github.com/WinMerge/winmerge/blob/master/ShellExtension/ShellExtension/WinMergeShell.rgs
   • https://github.com/search?q=repo%3Amicrosoft%2FWindows-classic-samples+ContextMenuHandlers&type=code
   • https://github.com/uvbs/Norton_AntiVirus_SourceCode/blob/b90225233fa7930d2ef7922080ed975b7abfae8c/Consumer/r12.0.6/src/UI/NAVShellExt/IEContextMenu.rgs
   • https://github.com/sysprogs/WinCDEmu/blob/master/WinCDEmuContextMenu/VCDImgContextMenu.rgs

[end2endzone]

Possible solutions:

• 1. Temporary remove delendencies to GLOG and libexprtk libraries. Tried in a9e5849. Did not work. The library is not unloaded after unregistering.
• 2. Replace the custom registration code by old Microsoft supplied code. See here or here for details. Tried in ffcc940. Did not work. Also, the functions does not register the shell extension for directories, folder, desktop and other.
• 3. Add a call to DisableThreadLibraryCalls(hInstance); in dllmain.cpp. ShellAnything is lacking this guard. This seems like good practice in many shell extension. For example:
  • https://github.com/chrdavis/SmartRename/blob/v.1.1.2/SmartRenameExt/dllmain.cpp#L95
  • https://github.com/XiaoFaye/ZipFolderEx/blob/0.1/ZipFolderEx/dllmain.cpp#L56
  • https://github.com/yanxijian/ShellExtContextMenuHandler/blob/master/dllmain.cpp#L56
  • https://github.com/chrdavis/PIFShellExtensions/blob/master/PIFShellExtensions/DllMain.cpp#L54
• 4. Replace custom registration code in favor of ATL generated code. This would provide the maximum support for latest operating system as Visual Studio 2019 may generate specific code for a shell extension running on Windows 11 or Windows 10. See the following for reference:
  • https://github.com/WinMerge/winmerge/tree/master/ShellExtension/ShellExtension
  • https://github.com/PxGame/template_IContextMenuExt
  • https://github.com/kareldonk/WMFPreview
  • https://github.com/danielgrigg/ContextMenuDemo
  • https://github.com/stax76/OpenWithPlusPlus
  • COMPLETED: See 22d3d87.
• 5. Review the code to make sure there is no instance leaks. Does classes are deleted/freed when their reference counter return to 0?
  • ShellAnything implementation differs from the following:
    • chrdavis/SmartRename with here, and here.
    • CredentialProvider from microsoft/Windows-classic-samples.
    • deskbands from microsoft/Windows-classic-samples.
    • explorerdataprovider from microsoft/Windows-classic-samples.
    • https://github.com/yanxijian/ShellExtContextMenuHandler
    • https://github.com/XiaoFaye/ZipFolderEx
    • https://github.com/owncloud/client
  • See also:
    • https://stackoverflow.com/questions/43010172/addref-and-release-in-iunknown-what-do-they-actually-do
    • https://stackoverflow.com/questions/73238133/how-to-do-com-dll-reference-counting-the-right-way-official-microsoft-samples-a
    • https://learn.microsoft.com/en-us/windows/win32/api/unknwn/nf-unknwn-iunknown-release

    When the reference count on an object reaches zero, Release must cause the interface pointer to free itself. When the released pointer is the only (formerly) outstanding reference to an object (whether the object supports single or multiple interfaces), the implementation must free the object.
    Call this method when you no longer need to use an interface pointer. If you are writing a method that takes an in-out parameter, call Release on the pointer you are passing in before copying the out-value on top of it.

• 6. Rewrite the shell extension's high level code based on the code from https://www.codeproject.com/Articles/441/The-Complete-Idiot-s-Guide-to-Writing-Shell-Extens#gettingstarted.

[end2endzone]

• Take a look at registry key:
  HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\Shell Extensions\Cached. It contains an entry about ShellAnything's extension. The key contains a value matching ShellAnything's class id. The value is still in the registry after the shell extension is unregistered. This might be an issue.
  Note: There is no reference to the shell extension under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Shell Extensions\Cached.

[end2endzone]

Fixed Cache registry key in e7d4b32.

[end2endzone]

Disabled option 6: Rewrite the shell extension's high level code based on the code from https://www.codeproject.com/Articles/441/The-Complete-Idiot-s-Guide-to-Writing-Shell-Extens#gettingstarted.

[end2endzone]

There does not seems to be a way to "force" or tell File Explorer to unload the Shell Extension once it is unregistered.

According to https://stackoverflow.com/questions/11365944/how-to-unload-c-shell-extension-dll-properly, Windows Installer should be able to use MoveFile to move a loaded dll. This could be an acceptable solution.

[end2endzone]

ShellAnything recently moved to dynamic linked libraries (DLL) for GLOG and libexprtk libraries (#103). There might be a hook or something similar which prevents explorer.exe to unload the main DLL because it can't also unload glog.dll or libexprtk.dll.

This has been proven incorrect. A dumb/empty new shell extension was created (see code attached below). The shell extension is really simple and has no dependencies on other DLL. Once unregistered, File Explorer is still holding to the shell extension's dll file. The file can be moved but not deleted. Waiting 2h+ did not resolved the issue.

See the following code used for testing: FooBarShellExt.zip
Note: This is a stripped down version of a shell extension from an GPLv2 licensed projet.

[end2endzone]

I have implemented an utility to restart File Explorer so that uninstalling the shell extension can be done without rebooting. The plan is to modify the uninstaller to show instructions to run the File Explorer restart before actually uninstalling.

[end2endzone]

The plan is to show a dialog when uninstalling such as the following:

--------------------------------------------------------------------------------------------------------
Before you uninstall
Should you renew File Explorer before you continue?
--------------------------------------------------------------------------------------------------------

Shell Extensions do not uninstall as easily as other softwares. Shell Extensions DLLs cannot be deleted because
File Explorer usually have a lock on the file. There is an issue with File Explorer which do not automatically
release shell extensions dll even if they have been unregistered from the system. Because of the lock, these dll
files cannot be deleted. A system reboot is usually the prefered option to make sure all dll files are released
and can be deleted.

ShellAnything provides a workaround which allow complete uninstallation without rebooting the system.
File Explorer Renew is an utility provided with ShellAnything that can close and reopen all File Explorer windows.
If the shell extension is unregistered, it will renew all windows and release the lock on the shell extension dll.

To enable the workaround, close the uninstaller, unregister the shell extension (unregister.bat)
and then run "file_explorer_renew.exe".

You can also continue normally and reboot the system to complete the uninstall process.

It seems that showing a dialog or a message box during the uninstall process is a bad idea. This is a bad practice because the uninstaller is usually executed in Silent Mode :

I wouldn't mess with this uninstall sequence. Any modal dialog that pops up when the installation is run in silent mode (which it is from add/remove) could cause your entire product to be axed in a corporate environment.

See the following references:

• https://stackoverflow.com/questions/4838526/launch-dialog-on-uninstall-using-wix#5999450
• https://stackoverflow.com/questions/55185676/wix-uninstall-confirmation-dialog-on-program-and-features-yes-no-before-msi-pr
• https://stackoverflow.com/questions/53876550/wix-custom-action-dialogbox-on-silent-uninstall-of-application
• https://stackoverflow.com/questions/40926730/how-to-show-dialog-or-message-box-during-uninstall-in-wix
• https://stackoverflow.com/questions/39377461/how-to-customize-the-uninstall-dialog-in-programs-and-features#39390642

[end2endzone]

Forcing a restart of File Explorer (without the user's consent) may also be a bad idea:

• https://stackoverflow.com/questions/11365944/how-to-unload-c-shell-extension-dll-properly

According to the link above, it appears the appropriate solution is to use MoveFile:

You can use the same mechanism in your homebrew installer and move the old file that is in use out of its original location and move the other one in right away. Any new application instance will then make use of the new shell extension (*), while the old one will continue to be used by any of the running applications that loaded it at one point or another.

This would allow the uninstallation of an old version and the installation of a new ShellAnything version. However, the new installed version would not be available to the system:

[Windows have] rules that apply to DLL loading and prevent modules with the same name to be loaded again under some circumstances.

[end2endzone]

Added a new dialog in the installer about instructions before uninstalling. See e453904 for details.

[end2endzone]

Leaving the issue opened in case someone else has an actual way of solving the problem.