Make it possible to create GUI mode applications in Windows

[konovod]

Feature Request

• Is your feature request related to a problem? Please describe clearly and concisely what is it.
  Right now, every crystal program on Windows creates console window. It is fine for experimenting, but for polished programs (GUI, games, headless servers, etc) there should be a way to build without showing console window.

• Describe the feature you would like, optionally illustrated by examples, and how it will solve the above problem.
  I've tried adding @[Link(ldflags: "/subsystem:windows")] - application just silently closes after start.
  I think the problem is in https://github.com/crystal-lang/crystal/blob/master/src/crystal/system/win32/wmain.cr so tried to change it in different ways (change entry point to fun wWinMain(ptr1 : Void*, ptr2 : Void*, argc : Int32, argv : UInt16**) : Int32, change flags) but no luck - application still closes at start.

• Minimal example:

@[Link(ldflags: "/subsystem:windows")] # it works (but show console window) if this line is commented
@[Link("user32")]
lib WinAPI
    fun message_box = MessageBoxA(hWnd : Void*, lpText : UInt8*, lpCaption : UInt8*,  uType : UInt32)
end

WinAPI.message_box(Pointer(Void).null, "Hello", "World", 0)

[HertzDevil]

At the minimum you will need the following monkey-patch, because otherwise the Crystal runtime will attempt to configure a non-existent console:

module Crystal::System::FileDescriptor
  def self.from_stdio(fd)
    console_handle = false
    handle = LibC._get_osfhandle(fd)
    if handle != -1 && handle != -2
      handle = LibC::HANDLE.new(handle)
      # TODO: use `out old_mode` after implementing interpreter out closured var
      old_mode = uninitialized LibC::DWORD
      if LibC.GetConsoleMode(handle, pointerof(old_mode)) != 0
        console_handle = true
        if fd == 1 || fd == 2 # STDOUT or STDERR
          if LibC.SetConsoleMode(handle, old_mode | LibC::ENABLE_VIRTUAL_TERMINAL_PROCESSING) != 0
            at_exit { LibC.SetConsoleMode(handle, old_mode) }
          end
        end
      end
    end

    io = IO::FileDescriptor.new(fd, blocking: true)
    # Set sync or flush_on_newline as described in STDOUT and STDERR docs.
    # See https://crystal-lang.org/api/toplevel.html#STDERR
    if console_handle
      io.sync = true
    else
      io.flush_on_newline = true
    end
    io
  end
end

Strictly speaking, you don't need a different entry point to be able to display any GUI, but one can be provided nonetheless: (if there are multiple /ENTRY options, the last one wins)

@[Link(ldflags: "/ENTRY:wWinMainCRTStartup")]
@[Link(ldflags: "/SUBSYSTEM:WINDOWS")]
lib LibCrystalMain
end

lib LibC
  alias HLOCAL = HANDLE
  alias HINSTANCE = HANDLE

  # shellapi.h
  fun CommandLineToArgvW(lpCmdLine : LPWSTR, pNumArgs : Int*) : LPWSTR*

  # winbase.h
  fun LocalFree(hMem : HLOCAL) : HLOCAL
end

fun wWinMain(
  hInstance : LibC::HINSTANCE,
  hPrevInstance : LibC::HINSTANCE,
  pCmdLine : LibC::LPWSTR,
  nCmdShow : LibC::Int,
) : LibC::Int
  argv = LibC.CommandLineToArgvW(pCmdLine, out argc)
  wmain(argc, argv)
ensure
  LibC.LocalFree(argv) if argv
end

This is still not enough for production use because a lot of the places in Crystal's runtime still assume the presence of the standard streams (e.g. unhandled exceptions always go to STDERR). The lack of standard streams is relevant to not just graphical-mode Windows binaries, but also shared libraries (#921).

[HertzDevil]

The parameters in the alternative entrypoint can be retrieved as follows:

lib LibC
  # libloaderapi.h
  fun GetModuleHandleW(lpModuleName : LPWSTR) : HMODULE

  # processenv.h
  fun GetCommandLineW : LPWSTR

  # processthreadsapi.h
  fun GetStartupInfoW(lpStartupInfo : STARTUPINFOW*)
end

# if `wWinMain` is invoked, then we are not building a DLL,
# so the following always works
hInstance = LibC.GetModuleHandleW(nil)

# this parameter exists for compatibility only
hPrevInstance = LibC::HINSTANCE.null

# the raw command line string, rarely needed when `ARGV` exists
pCmdLine = LibC.GetCommandLineW

# usually this is `SW_SHOW = 1`
# one way to customize this value is by creating a Windows shortcut to the program,
# then changing Shortcut -> Run in the shortcut properties dialog, which could produce
# `SW_MAXIMIZE = 3` or `SW_SHOWMINNOACTIVE = 7`
# the other way is by calling `LibC.CreateProcessW` directly
LibC.GetStartupInfoW(out startup_info)
nCmdShow = startup_info.wShowWindow

So while wWinMain is a convenience, I don't think it is ever necessary for writing Windows GUI applications, not even in C. Passing --link-flags=/SUBSYSTEM:WINDOWS during compilation should be sufficient now that the monkeypatch above is merged.

If #13330 is implemented, it becomes the responsibility of user code to redefine Crystal.fatal as appropriate. An unhandled exception may very well attempt to write the message to a file rather than showing a modal dialog, and it is not the standard library's job to decide. So that should leave nothing actionable that is exclusive to this issue.

[crysbot]

This issue has been mentioned on Crystal Forum. There might be relevant details there:

https://forum.crystal-lang.org/t/how-to-create-a-gui-app-using-libui-ng-on-windows/6361/24