tests/stdlib/tfdleak is flaky on windows + netbsd

[euantorano]

The test tests/stdlib/tfdleak.nim fails on NetBSD with the following output:

Category: stdlib
Name: tests/stdlib/tfdleak.nim C
Action: run
Result: reOutputsDiffer
-------- Expected -------

--------- Given  --------
leaked reopen
-------------------------

[timotheecour]

(moving from timotheecour#190) it's not just netbsd; happens at least on windows as well (at least) so I've changed title /cc @alaviss since you've added this test in #13201 :-)

history: https://dev.azure.com/nim-lang/Nim/_test/analytics?definitionId=1&contextType=build

example:
https://dev.azure.com/nim-lang/255dfe86-e590-40bb-a8a2-3c0295ebdeb1/_apis/build/builds/5280/logs/66

2020-05-14T03:25:41.0864301Z FAIL: tests/stdlib/tfdleak.nim C
2020-05-14T03:25:41.0865386Z Test "tests\stdlib\tfdleak.nim" in category "stdlib"
2020-05-14T03:25:41.0866068Z Failure: reOutputsDiffer
2020-05-14T03:25:41.0866878Z Expected:
2020-05-14T03:25:41.0867295Z 
2020-05-14T03:25:41.0867768Z 
2020-05-14T03:25:41.0868196Z Gotten:
2020-05-14T03:25:41.0868728Z leaked createNativeSocket()
2020-05-14T03:25:41.0869348Z leaked newSocket()
2020-05-14T03:25:41.0869725Z 

other example:

FAIL: tests/stdlib/tfdleak.nim C

https://dev.azure.com/nim-lang/255dfe86-e590-40bb-a8a2-3c0295ebdeb1/_apis/build/builds/5123/logs/66

2020-05-08T00:46:45.9418160Z PASS: tests/stdlib/tdecls.nim C                                    ( 6.65 sec)
2020-05-08T00:46:45.9422875Z FAIL: tests/stdlib/tfdleak.nim C
2020-05-08T00:46:45.9424937Z Test "tests\stdlib\tfdleak.nim" in category "stdlib"
2020-05-08T00:46:45.9427256Z Failure: reOutputsDiffer
2020-05-08T00:46:45.9428335Z Expected:
2020-05-08T00:46:45.9428737Z 
2020-05-08T00:46:45.9429171Z 
2020-05-08T00:46:45.9429819Z Gotten:
2020-05-08T00:46:45.9431348Z leaked system.open()
2020-05-08T00:46:45.9431966Z leaked reopen
2020-05-08T00:46:46.0808665Z 
2020-05-08T00:46:46.2481550Z PASS: tests/stdlib/tfdleak.nim C                                   (25.60 sec)

@euantorano it's important to distinguish "failing" from "flaky" (as it helps fixing bugs); for netbsd is it flaky or always-failing?

also is there a way to avoid this:

when running locally on windows?

[euantorano]

I can’t 100% reliably say whether it’s flaky or failing based on the limited number of times I’ve ran the tests locally (probably about 3 times). It’s failed every time there, but I’ll need to gather more results.

[alaviss]

also is there a way to avoid this:

You can't, the test spawned a simple TCP server to do leak checking.

It’s failed every time there, but I’ll need to gather more results.

It's pretty much NetBSD fault actually:

https://github.com/NetBSD/src/blob/072d87e47de012d8038aba996d3ab672569d9574/lib/libc/stdio/freopen.c#L150

The dup2 will undo O_CLOEXEC set by the call earlier... You should open a bug report there. As for a walkaround, modify this so that it's enabled for netbsd regardless of NoInheritFlag:

Nim/lib/system/io.nim

Lines 640 to 644 in 278b458

	when not defined(nimInheritHandles) and declared(setInheritable) and
	NoInheritFlag.len == 0:
	if not setInheritable(getOsFileHandle(f), false):
	close(f)
	return false

[euantorano]

For reference: the OpenBSD code uses dup3:

	/*
	 * If reopening something that was open before on a real file, try
	 * to maintain the descriptor.  Various C library routines (perror)
	 * assume stderr is always fd STDERR_FILENO, even if being freopen'd.
	 */
	if (wantfd >= 0 && f != wantfd) {
		if (dup3(f, wantfd, oflags & O_CLOEXEC) >= 0) {
			(void) close(f);
			f = wantfd;
		}
	}