curl_global_init(): make it thread-safe #8680
Conversation
tguillem
force-pushed
the
curl-global-init-ts
branch
2 times, most recently
from
841d2b4 to
6e5c2cd
Compare
|
Also see #4064 |
Sure it would be possible. It would imagine that check should be enabled by default. But I would also like to see the #4064 and #5017 approaches considered properly, as they might be able to make us reach this goal without using a thread library. |
On Windows, we could do implement a lock via Interlocked (Atomic like) and WaitOnAddress (futex like) : On Linux, we could do the same with atomics and futexes, but futuxes are not available on all POSIX systems (we could fall back to a wait loop but I'm not very fan). That's why I would prefer using pthread. |
tguillem
force-pushed
the
curl-global-init-ts
branch
6 times, most recently
from
6597e58 to
8b65933
Compare
If a program need a thread-safe init for curl, then it is likely already linking with the pthread library (on posix). I prefer the mutex approach to keep it simple since atomics or spinlocks are too often wrongly used (lock convoy, fairness, active wait). For Windows, the SRWLock is a good a light solution too. |
tguillem
force-pushed
the
curl-global-init-ts
branch
from
8b65933 to
7417fd4
Compare
|
Hello, I just proposed an alternative that doesn't need pthread: 7417fd4 We still use SRWLOCK for Windows since it is simpler that way (don't need to use Windows Atomic API, and it is very likely to be a spinlock internally). |
Use a posix pthread or a Windows SRWLOCK to lock curl_global_init*() and curl_global_cleanup().
tguillem
force-pushed
the
curl-global-init-ts
branch
from
7417fd4 to
34b141d
Compare
This flag can be used to make sure that curl_global_init() is thread-safe. This can be useful for libraries that can't control what other dependencies are doing with Curl.
tguillem
force-pushed
the
curl-global-init-ts
branch
from
34b141d to
deb38e1
Compare
This flag can be used to make sure that curl_global_init() is thread-safe. This can be useful for libraries that can't control what other dependencies are doing with Curl. Closes #8680
|
Thanks! |
|
I don't think this PR was sufficiently tested as the later merge of #8976 with 26 failed CI runs unfortunately shows. |
|
We saw zero interest from anyone to do more tests and we had no indication that this was not good enough on Windows. I think we merged it fine. We keep suffering on Windows because we have too few people around who know about and care for Windows (before things are merged), and the Windows CI jobs are flaky. The test seems to have problems in itself, but does it show that this code isn't good enough and doesn't do what it's set out to do? |
|
The Windows CI is far less flaky nowadays. Most failures are now caused by the underlying infrastructure or memory issues, not the tests themselves anymore. So, if you see more than 1 CI failing it is definitely worth checking. I am sorry, but I personally can't keep up with this all the time although I am trying very hard. I tried to make the CI as stable as possible and still do, and then people just ignoring it again makes me question the whole purpose. I don't know if this code without the test is broken, I just focused on the test last night. I will try to replicate a Windows specific test with CreateThread. |
|
At least I would determine that a test pass rate of 99.91 % on Azure CI can be considered quite stable, but unfortunately this does not include CI and test runs that timed out: |
The net result is still that there is virtually zero chance of all CI jobs to run green on Windows which makes it very easy to miss when real issues occur on just a few windows builds. I know I miss such ones all the time. And I do try to look for them. |
|
IMO this is not any of our fault. IIRC any CI we've used has always been flaky on Windows, weird input/output issues that can't be reproduced, and a bunch of hacks we have to create to get things working. They seem to be way more sensitive to timing and available resources. And credit to Marc for his work on this, they are definitely more stable than they used to be. |
Follow-up to 9b517c8 curl#11973 Follow-up to 23af112 curl#8680
On Windows this function is exported from winpthread, but winpthread is never used in Windows builds by curl itself. In cmake it was marked as never available, but autotools did check for this function. It found it sometimes (presumable when a dependency linked to pthread), resulting in a different `curl_config.h`. Sync the build systems by skipping this check in both, for Windows. Follow-up to 9b517c8 curl#11973 Follow-up to 23af112 curl#8680
On Windows a successful `sched_yield()` detection requires mingw-w64 built with POSIX threads (not Win32 threads) and GCC (not llvm/clang). (linking to `winpthread` via custom options may also work.) In CMake builds, it was pre-cached as unavailable before this patch. When detected (via autotools), it got only used for Windows XP or older targets combined with a non-GCC, non-clang compiler that doesn't support `__builtin_ia32_pause()`, or with the Intel C compiler. According to `lib/easy_lock.h`. mingw-w64 only supports GCC and clang, leaving a very narrow chance when `shed_yield()` gets called on Windows. Even then, `sched_yield()` is implemented in `winpthread` as `Sleep(0)`, which may or not be a useful. It's also trivial to implement locally if it is, and such rare build combination is also deemed useful. Thus, this patch marks `sched_yields()` permanently unavailable on the Windows platform also with autotools, and instead of pre-caching, skip this feature check with CMake. This syncs `HAVE_SCHED_YIELDS` between builds methods on Windows. Follow-up to 9b517c8 #11973 Follow-up to 23af112 #8680 Closes #16037
Making curl_global_init() thread-safe can be very useful for libraries that can't control what other libraries (or the main application) are doing with Curl. That way, curl_global_init() doesn't have to be called at the very beginning of the process and can be called anytime (from another library init function, that is run later, for example).
PS: I didn't know that number to choose for the new test (3026).