Optionally disable thread-local storage.

[fgvanzee]

Details:

• Implemented a new configure option, --disable-tls, which allows the user to optionally disable the use of thread-local storage qualifiers on static variables in BLIS. This option will rarely be needed, but in some situations may allow BLIS to compile when TLS is unavailable. Like the --disable-system option, --disable-tls forcibly disables threading (e.g. forcibly uses --disable-threading). Thanks to Nick Knight for suggesting this option.
• Modified frame/include/bli_lang_defs.h so that BLIS_THREAD_LOCAL is defined to nothing when BLIS_ENABLE_TLS is not defined.
• Edited --disable-system configure status output for clarity.

@nick-knight @hominhquan You may wish to discuss this feature since you both make use of a similar option, --disable-system. 🙂

[hominhquan]

Thanks @fgvanzee for the ping. I don't know if I have given my opinion before, if not, I'd like to express my disagreement to the use of TLS, not only in any highly parallel and performant code, but also generic one.

From experience and personal reasoning, I think we can always find an alternative to TLS, and building contextless API (all behaviors are defined by arguments and not by any global variable, or if global they must be Read-only) is the safest way to deal with parallel execution on unknown/future/novel/exotic architectures.

BLIS already defined a lot of logic information and parameters in arguments (cntl_t, cntx_t, rntm_t, ...) and I like it. And now I find it a bit strange to say: "We disable multi-threading because we don't have TLS". It's like leaving other low-end embedded architectures behind the door, just because they are not "well-equipped" enough.

Anyway, if we can't do differently, in the current state, this PR is OK to me.

[fgvanzee]

And now I find it a bit strange to say: "We disable multi-threading because we don't have TLS". It's like leaving other low-end embedded architectures behind the door, just because they are not "well-equipped" enough.

@hominhquan Thank you for sharing your honest opinion! I'm happy to report that my choice to disable threading when disabling TLS was not one I was in any way confident of -- I only did it out of (a) caution, and (b) I was copying and pasting code from the part of configure that dealt with --disable-system. So I'm happy to remove that part if it makes sense to someone with more experience in these matters than me. 🙂

[devinamatthews]

@hominhquan Is there such as system that supports multithreading but not TLS? It seems like the latter is a prerequisite for most implementations of the former? I certainly have no objection to disabling it if possible, but there is some useful functionality which it enables. Even if the library were architected to be free of global state, a globally-stateful interface would be the most natural for a lot of people, especially when they have to access BLIS through the BLAS interface for legacy reasons.

[nick-knight]

Thanks @fgvanzee for adding this configure knob.

I don't have any strong opinions (yet) on the broader question of how BLIS uses TLS, since I don't fully understand how it's used.

I think it is reasonable to have this configuration option, since TLS support can be quite involved. For example, GNU docs say

It requires significant support from the linker (ld), dynamic linker (ld.so), and system libraries (libc.so and libpthread.so), so it is not available everywhere.

[leekillough]

@devinamatthews : pthreads does not support TLS as we think of it today, e.g., thread_local variables accessed through a thread pointer register defined in the ABI, but instead you have to jump through hoops like pthread_key_create, pthread_getspecific, pthread_setspecific, and pthread_key_delete.

But thread_local has been standard since C11/C++11 and interoperates with pthreads and OpenMP.

[hominhquan]

I only did it out of (a) caution, and (b) I was copying and pasting code from the part of configure that dealt with --disable-system.

@fgvanzee You are right to care about race-condition, but here I'm curious about write access to bli_l3_ind_oper_st[] being already protected (I would protect the read-access as well, or use lock-free atomic), but the array is still put in TLS, why ? because it is not needed to protect a TLS variable, unless that TLS variable is shared to another thread ? (nooooo ... )

@hominhquan Is there such as system that supports multithreading but not TLS? It seems like the latter is a prerequisite for most implementations of the former?

TLS is not a must-have feature to run multithreading code. The proof is BLIS has run well in either OpenMP or Pthread on C6x, or MPPA and some others before. On another hand, a sequential single-thread code can totally put a variable in TLS, and that code will compile and work on most CPU today. Most, but not all.

Supporting TLS on new architectures/OS/compilers is often a hazardous and hackish task, which implies many architecture-specific handling, e.g. zephyrproject-rtos/zephyr#22185

I certainly have no objection to disabling it if possible, but there is some useful functionality which it enables. Even if the library were architected to be free of global state, a globally-stateful interface would be the most natural for a lot of people, especially when they have to access BLIS through the BLAS interface for legacy reasons.

Totally agree and I have no problem (for legacy and historical reasons) with these global structures between the BLAS and BLIS API.

My rationale is: Today we are writing code that will run on who-knows-how-many different architectures in the next years/decades. I don't know if they will be extremely sophisticated, or contrastingly very simple and poorly-equipped for power reason and/or application-domain. Using TLS is one way to make our code less portable.

[fgvanzee]

@fgvanzee You are right to care about race-condition, but here I'm curious about write access to bli_l3_ind_oper_st[] being already protected (I would protect the read-access as well, or use lock-free atomic), but the array is still put in TLS, why ? because it is not needed to protect a TLS variable, unless that TLS variable is shared to another thread ? (nooooo ... )

IIRC, our collaborators at AMD once pointed out that TLS was desired to enforce some determinism on static variables such as bli_l3_ind_oper_st[] (the other example being error checking level I think?) because of shenanigans that could take place in application-level threading (rather than BLIS-level threading). It was certainly not a mainstream use case, but rather a situation where they wanted to prevent accidental misuse.

[devinamatthews]

@hominhquan @leekillough thanks for pointing out that TLS is more difficult to support than multithreading. If TLS were to be disabled but not multithreading, as a user would you consider it acceptable for thinks like number of threads to spawn, 1m enable/disabled, error checking level, etc. to be global (and possibly subject to race conditions)? Is it instead possible to fake the TLS behavior using the ptheads functions mentions by @leekillough ?

[hominhquan]

TLS was desired to enforce some determinism on static variables such as bli_l3_ind_oper_st[] (the other example being error checking level I think?) because of shenanigans that could take place in application-level threading (rather than BLIS-level threading)

OK, thanks for explanation @fgvanzee.

@hominhquan @leekillough thanks for pointing out that TLS is more difficult to support than multithreading. If TLS were to be disabled but not multithreading, as a user would you consider it acceptable for thinks like number of threads to spawn, 1m enable/disabled, error checking level, etc. to be global (and possibly subject to race conditions)?

@devinamatthews @fgvanzee Between (1) disable both TLS and multithreading for safety and (2) disable TLS but not multithreading at the risk of data-race. I would regrettingly choose the solution (1). In any case, functionality comes first. Debugging multithreading data-race is horribly a headache.

I still hope these runtime logic modifications can be reimplemented differently without recourse to TLS. At least it will be the first thing I do when if happens to me to rebase the latest BLIS version.

Is it instead possible to fake the TLS behavior using the ptheads functions mentions by @leekillough ?

IMHO, supporting explicit thread_local using pthread primitives mentioned by @leekillough is equivalent to using malloc/free or internal memory pool mechanism in BLIS, which is also probably what I would use to get rid of TLS.

[fgvanzee]

@devinamatthews @fgvanzee Between (1) disable both TLS and multithreading for safety and (2) disable TLS but not multithreading at the risk of data-race. I would regrettingly choose the solution (1). In any case, functionality comes first. Debugging multithreading data-race is horribly a headache.

In the abstract, I would agree. But I could also make the case that disabling TLS is a very "I know what I'm doing" kind of thing. The only people who would be doing so should be aware of the possible consequences for multithreading, especially if I include a warning in the configure --help text, which I am happy to do.

I still hope these runtime logic modifications can be reimplemented differently without recourse to TLS. At least it will be the first thing I do when if happens to me to rebase the latest BLIS version.

I'm always open to making BLIS more stateless, where possible. Maybe such hypothetical changes don't completely eliminate the applicability of TLS, but they may meaningfully reduce it.

[hominhquan]

In the abstract, I would agree. But I could also make the case that disabling TLS is a very "I know what I'm doing" kind of thing. The only people who would be doing so should be aware of the possible consequences for multithreading, especially if I include a warning in the configure --help text, which I am happy to do.

So OK to keep multithreading and add a warning message about possible data-race, and accordingly recommend the developer to try --disable-threading if (s)he encounters any correctness or deadlock issue. Such message can be added in both configure --disable-tls and the test runtime inside the macro #ifdef BLIS_DISABLE_TLS (at the beginning ?)

[fgvanzee]

@hominhquan I opted not to insert a compile-time warning since the warning would be emitted for every since file compilation (since everything goes through the one header, blis.h).

[hominhquan]

@hominhquan I opted not to insert a compile-time warning since the warning would be emitted for every since file compilation (since everything goes through the one header, blis.h).

Oh I think I was not clear enough about "test runtime message". I meant an additional printf message in test_libblis.c at the beginning of execution, where all parameters are printed to stdout for information, in libblis_test_output_params_struct I guess. Something like:

void libblis_test_output_params_struct( FILE* os, test_params_t* params )
{
    ...
    #ifdef BLIS_DISABLE_TLS
    libblis_test_fprintf_c( os, 
        "[WARNING] This BLIS library was compiled with TLS disabled. We consider you know what you are doing.\n"
        "[WARNING] Multi-threading race-condition, correctness issues and deadlock may occur.\n"
        "[WARNING] If this happens, please consider reconfiguring and recompiling with --disable-threading before any further deep debug.\n"
    );
    #endif // BLIS_DISABLE_TLS
    ...
}

[fgvanzee]

Oh I think I was not clear enough about "test runtime message". I meant an additional printf message in test_libblis.c at the beginning of execution, where all parameters are printed to stdout for information, in libblis_test_output_params_struct I guess.

@hominhquan Ohhh, yes. That would be fine. I'll work on it! Thanks for the clarification.

[fgvanzee]

Are we all okay now with this compromise? (Leave threading untouched when TLS is disabled, but warn of consequences in ./configure --help, in configure output, and in the testsuite informational header output.)

[devinamatthews]

I guess.

[hominhquan]

LGTM.