Wait for threads to exit after stopping model installer/downloader

[lstein]

Summary

This PR causes the model_installer and download_queue stop() methods to block until their worker threads have exited. Previously stop() returned immediately.

This fix addresses an issue in the model installer unit test, where the installer is created and torn down in a pytest fixture. Because a new installer was created immediately after the previous installer was torn down, it was possible for the old installer’s threads to intermix with the current ones, causing unpredictable results. This may fix the sporadic hangs and timeouts observed in the model installer unit test.

The downside of this is that the test_model_install and test_download unit tests now run noticeably slower than they used to due to waiting for the threads to exit. The alternative, of scoping the install fixture to the module or session level, is undesirable because we want each test function to start out with a clean install directory and registry.

Related Issues / Discussions

See discord https://discord.com/channels/1020123559063990373/1149513695567810630/1219777315148664882

QA Instructions

If this PR fixes the problem, we will no longer see sporadic timeouts during unit testing. It is a very hard bug to reproduce, however, so time will tell.

Merge Plan

Merge when approved.

Checklist

• The PR has a short but descriptive title
• Tests added / updated
• Documentation added / updated

[psychedelicious]

Working fine for me, let's fail forward on this.

I tidied up the log messages a bit.