Convert Windows stubs to Unicode #58

dra27 · 2024-03-05T03:33:25Z

Subsumes #55. Fixes #57. Fixes ocaml/dune#10180 which then fixes ocaml/opam#5861.

Windows OCaml uses UTF-8 internally for environment variables. The spawn_windows stub was using the ANSI version of CreateProcess (CreateProcessA) which causes the environment to become corrupted if the environment contained any non-ASCII characters (i.e. any UTF-8 sequences).

Support for Windows 98 ended in July 2006, so I think we can risk using the Unicode version of CreateProcess 😉 The code mirrors that for Unix.create_process.

This updates our spawn vendored version to include janestreet/spawn#58. Fixes ocaml#10180 Signed-off-by: Etienne Millon <me@emillon.org>

emillon · 2024-03-05T15:07:48Z

src/dune

+    (run ocaml %{dep:flags.ml})))
+
+(rule
+  (with-stdout-to flags.ml


I think that this might add the Flags module to the library.

Gah, yes indeed! Fix pushed for the original approach, but actually enabled_if is arguably "better", given that it's then a semi-pure Dune solution.

kit-ty-kate · 2024-03-06T11:48:21Z

src/spawn_stubs.c

+/* Must come before any other caml/ headers are included */
+#define CAML_INTERNALS
+
+#ifdef _WIN32


Instead of changing the dune file:

Suggested change

#ifdef _WIN32

#ifdef _WIN32

#define UNICODE

#define _UNICODE

Based on a comment by @emillon in ocaml/dune#10212 (comment)

My only concern is that UNICODE and _UNICODE must either be defined everywhere or nowhere (OCaml headers can pull in windows.h or parts of it as well, ending up with a similar problem to that which we have already had with CAML_INTERNALS).

The other benefit of putting it in flags is that any additional C files correctly inherit it, but that's really never likely to apply here. Given that we already have the _GNU_SOURCE definition (which is where it should go - i.e. right at the top).

I don't mind, though - how strong opinions against doing it "philosophically" correctly, if more complicatedly, in the dune file?

I agree it is better to do it in the dune file.

rgrinberg · 2024-03-08T08:19:57Z

@nojb could you have a look?

nojb

LGTM (modulo the tweaks below)

nojb · 2024-03-08T08:22:46Z

src/spawn.ml

-    Buffer.contents buf
+    if env = [] then
+      "\000\000"
+    else


Can you say a word about this change?

The environment block is supposed to be terminated with two NUL characters - if there are any bindings, that obviously in the other branch (and did in the ANSI version too). When there are no bindings, the ANSI version was passing a pointer to the OCaml string "\000" but that will have had an additional NUL terminator so it quietly worked. With the wide version, the issue was instantly revealed by one of the tests where CreateProcessW will have received two NUL bytes (i.e. a NUL wchar_t) but will have then "segfaulted" reading past the end of that buffer (CreateProcessW actually translated that to an invalid argument, which worryingly suggests that it carried on reading random memory and instead got upset that there was no = character anywhere or some such😱)

I'm not sure if it's worth a comment in the code - CreateProcess does indicate that the block should be a series of nul-terminated strings themselves terminated by another nul, it's just hazy on the case of an empty environment (not surprisingly, really).

Thanks for the explanation; but then isn't it wrong in the compiler?

https://github.com/ocaml/ocaml/blob/dfb7abefb4df82804703e6609d957f3dcda64d29/otherlibs/unix/unix_win32.ml#L999-L1003

Yeah, I had that same shock thought! 🙂 Just awaiting a train, and then I’ll flex some environments…

OK, OCaml's fine because createprocess.c speculatively adds a nul character. I was speculatively adding that character but not zeroing it. I've fixed it by not adding it, as I think it's nicer that the function in spawn.ml is entirely responsible for producing the UTF-8 version and then the C stub transcodes exactly that string.

nojb · 2024-03-08T08:23:54Z

src/spawn_stubs.c

+/* Must come before any other caml/ headers are included */
+#define CAML_INTERNALS
+
+#ifdef _WIN32


I agree it is better to do it in the dune file.

nojb · 2024-03-08T08:26:43Z

src/spawn_stubs.c

@@ -648,22 +679,22 @@ CAMLprim value spawn_windows(value v_env,
  if (!dup2_and_clear_close_on_exec(v_stdin , &si.hStdInput ) ||
      !dup2_and_clear_close_on_exec(v_stdout, &si.hStdOutput) ||
      !dup2_and_clear_close_on_exec(v_stderr, &si.hStdError )) {
-    win32_maperr(GetLastError());
+    caml_win32_maperr(GetLastError());
    close_std_handles(&si);
    uerror("DuplicateHandle", Nothing);


If we error out here, we will leak the strings allocated above. Maybe invert the order?

Oh blimey, good catch. (Another) revised version incoming...

emillon · 2024-03-08T09:13:11Z

ok, thanks. my only concern with the dune file was that it requires changing a bit our bootstrap system in dune but that's fine with me.

Signed-off-by: David Allsopp <david.allsopp@metastack.com>

If the duplication of the handles failed, then the memory allocated for the strings wasn't freed. Duplicate the handles first - the only failure for the strings is OOM, and all bets are then off. Signed-off-by: David Allsopp <david.allsopp@metastack.com>

dra27 · 2024-03-08T16:56:47Z

While looking at it again, for some reason the check for nul characters in environment values was only being done in Env_unix so I shifted the function out of there and used it in Env_win32 as well.

Hopefully this is the last version 🙂

nojb

LGTM (after the latest changes). Thanks!

Signed-off-by: Rudi Grinberg <me@rgrinberg.com>

This does several things: - update our spawn vendored version to include janestreet/spawn#58 - pick the dune file - add new build flags for bootstrapping Fixes ocaml#10180 Signed-off-by: Etienne Millon <me@emillon.org>

This does several things: - update our spawn vendored version to include janestreet/spawn#58 - pick the dune file - add new build flags for bootstrapping Fixes #10180 Signed-off-by: Etienne Millon <me@emillon.org>

This does several things: - update our spawn vendored version to include janestreet/spawn#58 - pick the dune file - add new build flags for bootstrapping Fixes ocaml#10180 Signed-off-by: Etienne Millon <me@emillon.org>

This does several things: - update our spawn vendored version to include janestreet/spawn#58 - pick the dune file - add new build flags for bootstrapping Fixes #10180 Signed-off-by: Etienne Millon <me@emillon.org> Co-authored-by: Etienne Millon <me@emillon.org>

emillon added a commit to emillon/dune that referenced this pull request Mar 5, 2024

fix(windows): use unicode version of CreateProcess

c7179fd

This updates our spawn vendored version to include janestreet/spawn#58. Fixes ocaml#10180 Signed-off-by: Etienne Millon <me@emillon.org>

emillon mentioned this pull request Mar 5, 2024

fix(windows): use unicode version of CreateProcess ocaml/dune#10212

Merged

emillon added a commit to emillon/dune that referenced this pull request Mar 5, 2024

fix(windows): use unicode version of CreateProcess

3c6ee60

This updates our spawn vendored version to include janestreet/spawn#58. Fixes ocaml#10180 Signed-off-by: Etienne Millon <me@emillon.org>

emillon added a commit to emillon/dune that referenced this pull request Mar 5, 2024

fix(windows): use unicode version of CreateProcess

8e832a3

This updates our spawn vendored version to include janestreet/spawn#58. Fixes ocaml#10180 Signed-off-by: Etienne Millon <me@emillon.org>

emillon reviewed Mar 5, 2024

View reviewed changes

kit-ty-kate reviewed Mar 6, 2024

View reviewed changes

nojb approved these changes Mar 8, 2024

View reviewed changes

dra27 added 7 commits March 8, 2024 16:45

Convert Windows stubs to Unicode

c618d30

Signed-off-by: David Allsopp <david.allsopp@metastack.com>

Apply preprocessor flags on Windows only

66581de

Signed-off-by: David Allsopp <david.allsopp@metastack.com>

Fix stray module

a2778fa

Signed-off-by: David Allsopp <david.allsopp@metastack.com>

Alternate without calling OCaml

494bedd

Signed-off-by: David Allsopp <david.allsopp@metastack.com>

Save the memory, 1 wide character at a time

b3f6cfe

Signed-off-by: David Allsopp <david.allsopp@metastack.com>

Windows environments also can't contain nulls

9637826

Signed-off-by: David Allsopp <david.allsopp@metastack.com>

dra27 force-pushed the win32-unicode-spawn branch from 2ff553a to 5958e51 Compare March 8, 2024 16:53

nojb approved these changes Mar 9, 2024

View reviewed changes

rgrinberg added 2 commits March 9, 2024 12:45

fmt

3929015

Signed-off-by: Rudi Grinberg <me@rgrinberg.com>

_

9410596

Signed-off-by: Rudi Grinberg <me@rgrinberg.com>

rgrinberg merged commit fa620a8 into janestreet:master Mar 9, 2024
1 check passed

emillon mentioned this pull request Mar 11, 2024

Make C stubs more resistant to header structure #55

Closed

kit-ty-kate mentioned this pull request Mar 12, 2024

Allow to compile opam when the environment contains unicode characters on Windows (upgrade to dune 3.14.2) ocaml/opam#5880

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert Windows stubs to Unicode #58

Convert Windows stubs to Unicode #58

dra27 commented Mar 5, 2024 •

edited

emillon Mar 5, 2024

dra27 Mar 6, 2024

kit-ty-kate Mar 6, 2024

dra27 Mar 6, 2024

nojb Mar 8, 2024

rgrinberg commented Mar 8, 2024

nojb left a comment

nojb Mar 8, 2024

dra27 Mar 8, 2024

nojb Mar 8, 2024

dra27 Mar 8, 2024

dra27 Mar 8, 2024

nojb Mar 8, 2024

nojb Mar 8, 2024

dra27 Mar 8, 2024

dra27 Mar 8, 2024

emillon commented Mar 8, 2024

dra27 commented Mar 8, 2024

nojb left a comment

Convert Windows stubs to Unicode #58

Convert Windows stubs to Unicode #58

Conversation

dra27 commented Mar 5, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rgrinberg commented Mar 8, 2024

nojb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emillon commented Mar 8, 2024

dra27 commented Mar 8, 2024

nojb left a comment

Choose a reason for hiding this comment

dra27 commented Mar 5, 2024 •

edited