Skip to content

Stack overflow prevention on Windows #1529

@rorso-doctra

Description

@rorso-doctra

I'm currently debugging the CUPS system for native Windows.
I had several troubles running the test cases and ippfind and thus tried to configure various xxxx_DEBUG environment variables to get more insight about what's happening.

It turned out that now ippfind breaks even before the first debugging output with a stack-overflow. My current insight about this is as follows:

cups_globals_alloc() allocates and initializes a _cups_globals_t but does not register it into thread-local storage (via cupsThreadSetData(cups_globals_key, cg)) before calling DEBUG_printf(). The debug path (DEBUG_printf -> debug_thread_id -> _cupsGlobals) re-enters cups_globals_alloc() because thread-local data is still NULL, producing infinite recursion until the stack overflows.

I "fixed" this (with some serious help from GPT-5) by adding a

cupsThreadSetData(cups_globals_key, cg);

quite early in the code of cups_globals_alloc() before line 197.

This might be a crude fix to get rid of the symptom and, not universally applicable. But maybe someone more knowledgeable want to look at this:

cups/cups/globals.c

Lines 185 to 204 in 002cedc

/*
* Clear the global storage and set the default encryption and password
* callback values...
*/
cg->encryption = (http_encryption_t)-1;
cg->password_cb = (cups_password_cb2_t)_cupsGetPassword;
cg->trust_first = -1;
cg->any_root = -1;
cg->expired_certs = -1;
cg->validate_certs = -1;
#ifdef DEBUG
/*
* Friendly thread ID for debugging...
*/
cg->thread_id = ++ cups_global_index;
#endif /* DEBUG */

cups_globals_alloc(void)
{
  const char	*cups_userconfig = getenv("CUPS_USERCONFIG");
					// Location of user config files
  _cups_globals_t *cg = calloc(1, sizeof(_cups_globals_t));
					// Pointer to global data
#ifdef _WIN32
  HKEY		key;			// Registry key
  DWORD		size;			// Size of string
  const char	*userprofile = getenv("USERPROFILE");
					// User profile (home) directory
  char		userconfig[1024],	// User configuration directory
		*userptr;		// Pointer into user config
  static char	installdir[1024] = "",	// Install directory
		sysconfig[1024] = "";	// Server configuration directory
#endif // _WIN32


  if (!cg)
    return (NULL);

  // Clear the global storage and set the default encryption and password callback values...
  memset(cg, 0, sizeof(_cups_globals_t));
  cg->encryption     = (http_encryption_t)-1;
  cg->password_cb    = (cups_password_cb_t)_cupsGetPassword;
  cg->trust_first    = -1;
  cg->any_root       = -1;
  cg->expired_certs  = -1;
  cg->validate_certs = -1;

  //
  // Register the newly-allocated globals for this thread early.  This prevents
  // debug logging (which calls _cupsGlobals()) from recursing back into
  // cups_globals_alloc() before the TLS pointer is set and causing a stack
  // overflow.  On Windows this maps to TlsSetValue; on POSIX it maps to
  // pthread_setspecific.  This assumes the thread key has been created by the
  // caller path (DllMain on Windows or pthread_once on POSIX) before we are
  // here, which is the case in normal initialization flows.
  //
  cupsThreadSetData(cups_globals_key, cg);

#ifdef DEBUG
  // Friendly thread ID for debugging...
  cg->thread_id = ++ cups_global_index;
#endif // DEBUG

  // Then set directories as appropriate...
#ifdef _WIN32
  if (!installdir[0])
  {
...

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingplatform issueIssue is specific to an OS or desktop

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions