Release 1.3.0
Creating a new minor release, because of the following major changes:
- The new threading model of the cache system, which allows multiple downloads (blocking foreground and background) to be done simultaneously.
- When cache system is enabled, the downloading thread yields as soon as there is enough data to reply to the FUSE read call.
- New pre-commit hooks to help developers to maintain a basic level of quality in each commit.
- New unit tests which test individual software components.
- New system integration tests which launch a HTTP server, then mount it for read testing. There are two variants:
- A short test which tests the filesystem against different edge case filenames.
- A long test which do multithread read on a 1GB file with different block sizes.
Here are the detailed changes:
Added
- Dynamically scale the number of background prefetch worker threads to match up to half of the maximum concurrent connection capacity (
CONFIG.max_connscapped atDEFAULT_NETWORK_MAX_CONNS), transitioning from a single hardcoded thread to improve performance when multiple concurrent active downloads are supported (2c39a7a). - Add a unit test
test_Cache_alloc_num_bg_workersto verify correct background worker allocation scaling under different network connection capacities (2c39a7a). - Implement a thread-safe manual reference-counting mechanism (
ActiveDownload_refandActiveDownload_unref) forActiveDownloadnodes to prevent Use-After-Free (UAF) vulnerabilities during concurrent read operations, background transfers, and cache teardown (9c34463). - Add a unit test
test_Cache_free_active_downloadsto verify that active download nodes are not prematurely freed while waiter threads are still active (9c34463). - Implement a robust shutdown handshake mechanism in
Cache_freeto coordinate safe cache teardown when reader threads are actively blocked on segment downloads, ensuring all threads unregister before mutexes and condition variables are destroyed (a01c9b1). - Add a unit test
test_Cache_free_active_downloads_with_waitersto exercise and verify proper thread synchronization and early-exit cleanup during cache shutdown (a01c9b1). - Add standardized GPLv3/OpenSSL copyright headers and detailed Doxygen file-level
\briefannotations to all C source and header files under thesrc/directory to improve legal compliance and maintain consistent documentation headers (f621da7, 666d53a). - Add
CURLOPT_PIPEWAITsetting to curl easy handles to optimize HTTP/2 multiplexing during concurrent file and directory transfers (4114977). - Support HTTP/3 (QUIC) protocol version negotiation with automatic and seamless fallback to HTTP/2 or HTTP/1.1 when unavailable (4114977).
- Implement shutdown memory leak detection in
DEBUGbuilds, recursively traversing the virtual filesystem at shutdown to tear down resources and verify that no memory leaks exist (30d7d10). - Add FUSE
.opendirand.releasedirhandlers to safely evict directories (LinkTables) from memory when they are no longer in use, using an atomic reference-counting strategy to prevent crashes in other subsystems (2fff24a). - Introduce a comprehensive unit test suite
test_memory_trackingto validate correct allocation, reallocation, and free behavior of debug allocator wrappers (9b38614). - Implement asynchronous early-return download support in cache mode to improve read performance during ongoing background downloads (0fb2d06).
- Introduce robust logging and error-checking wrappers for pthread condition variable APIs (
PTHREAD_COND_INIT,PTHREAD_COND_DESTROY,PTHREAD_COND_BROADCAST, andPTHREAD_COND_WAIT) (af0e263). - Support concurrent background segment downloads in cache mode by replacing the single active download tracker with a linked list of
ActiveDownloadstructs; addActiveDownload_find,ActiveDownload_add, andActiveDownload_removehelpers and implement double-checked locking inCache_read_segmentto prevent duplicate downloads under concurrent FUSE workers (7286314). - Add unit test for
ActiveDownload_findverifying lookup behavior on both empty and populated active download lists (0441ced). - Define a custom
sys_sem_ttype and emulate unnamed POSIX semaphores (sem_init,sem_destroy,sem_wait,sem_post, andsem_trywait) using pthread mutex and condition variables on Apple/macOS platforms, where unnamed POSIX semaphores are unsupported (af24b36). - Add zero-length file coverage to integration and unit test suites, including a dedicated test group verifying size, content, and SHA-256 digest in both direct and cache modes; add unit tests for negative-len, NULL
cf, and NULLcf->linkguards inCache_read(d02ac05).
Changed
- Transition from a single global condition variable (
dl_cond) to granular, block-specific condition variables integrated directly withinActiveDownloadnodes to eliminate redundant wakeups of unrelated FUSE waiter threads and reduce thread scheduling overhead (a40bd74). - Optimize the cache wait loop inside
Cache_read_segmentto check status in$O(1)$ time by introducing anunlinkedstate flag on theActiveDownloadstructure, avoiding expensive$O(N)$ list lookups on every loop iteration (42a4dd3). - Clean up and standardize
#includedirectives across all source files, headers, and test suites, using forward struct declarations to break circular dependency chains (94ced41). - Extract common cache test setup logic into dedicated helper functions (
setup_mock_link_tableandsetup_temp_cache_dir) insidetests/test_cache.c(2c39a7a). - Replace raw octal literals with symbolic permission constants (such as
S_IRWXU) in test directory creation calls to address SonarCloud security hotspot alerts (2c39a7a). - Update
README.mdto document the permanent cache thread-safety improvements, including granular signaling, manual reference counting, and the shutdown handshake mechanism (7eea0ea). - Update
USAGE.mdto document the--proxy-capathand--capathflags, update the default value of--max-connsto 6, and update internal commit reference links (1864cce, 7bcbcb2, 7d76285). - Lower the default maximum network connection count (
DEFAULT_NETWORK_MAX_CONNS) from10to6to align with the standard persistent connection limits agreed upon by all major modern web browsers (c049e6d). - Update
tests/test_config.cconfiguration initialization test to assertmax_connsagainst the configuration macro instead of a hardcoded value (c049e6d). - Optimize GitHub Actions workflows by separating short and long integration tests and removing redundant manual pre-commit hook stages to avoid duplicate, slow sequential CI execution (86b0e8e).
- Refactor synchronization and memory wrappers (mutex, semaphore, condvar, and allocator) to remove trailing name string parameters and clean up debug log statements (efb8c24).
- Remove all debug log statements (
log_printfandlprintfcalls under thedebuglog type) across FUSE, cache, and link modules to eliminate tracing overhead (efb8c24). - Eliminate unused helper variables (such as the static loop counter inside
LinkTable_new) that were only required by the removed debug prints, preventing compiler warnings (efb8c24). - Optimize memory tracking in debug allocator wrappers by replacing the
$O(N)$ linear linked list with a statically sized 8192-slot hash table with chaining (9b38614). - Optimize string sanitization loop in
sanitise_LinkTableby replacingstrlen()with a null-terminator check, eliminating$O(N^2)$ overhead (c8fbf51). - Refactor test suites into granular profiles (
unit_testfor C unit tests,integration_shortfor fast integration tests, andintegration_longfor intensive tests) and align documentation insrc/README.md(39963f7). - Revise pre-commit hook configurations to isolate builds, add debug-based verification (
meson-test-debug), restrict the automatic hook to fast unit tests, and provide manual pre-push hooks (all-testsandall-tests-clang) (39963f7, 76fa30b, 9ad044e, 89ac572, af4ee30). - Expand the GitHub Actions build matrix on Ubuntu and macOS to execute the test suite under both debug and release configurations in parallel (1554ed3).
- Upgrade GitHub Actions workflows to use the latest major runner and dependency actions version (68dd7bb, f819603).
- Add
--shortand--longflags togenerate_test_files.pyand isolate short and long integration test suites so that long cache tests no longer generate short fixture files redundantly; dynamically select the range-check probe file to match the active test mode (2225d0d). - Default
generate_test_files.pyto a no-op when no flag is given, removing the implicit--largealias;--allremains the canonical way to generate both the short fixture set and the 1 GB large file (3bb8737). - Refactor
struct Cacheto remove redundanttime,content_length, andfs_pathfields; updateMeta_read/Meta_writeto preserve the on-disk binary format using local variables, keepingstruct Linkas the single source of truth for core link attributes (63860bc). - Statically initialize
link_lockusingPTHREAD_MUTEX_INITIALIZER, removing redundant runtime initialization and resolvingEINVALcrashes in unit tests whereLinkSystem_initis bypassed (af24b36). - Extract repeated transfer-cleanup logic from
Link_downloadinto a single staticLink_download_finish_transferhelper, replacing three identical lock/clear/broadcast/reset blocks (12c6888). - Update
README.mdto describe high-performance concurrent and asynchronous cache downloads, and add a developer section on memory management and synchronization wrappers (3c0d145, 439264f). - Update
src/README.mdto reflect the split integration test stages present in the GitHub Actionsbuild.ymlconfiguration (439264f).
Fixed
- Resolve a critical Use-After-Free (UAF) deadlock vulnerability in the
Cache_freeteardown path by safely coordinating waiter exits via the new shutdown handshake and reference-counting mechanisms (a01c9b1). - Intercept size-zero allocations early in
REALLOC_wrapperby freeing the pointer and returningNULLto resolve a double-free on glibc and standardize behavior on BSD/macOS whererealloc(ptr, 0)returns a non-NULL sentinel pointer (1e1b11d, d6e3040). - Fix potential crashes by adding defensive
NULLchecks onthis_linkinCache_create(796fbb9). - Fix potential crashes by zero-initializing stack-allocated
TransferStructvariables (tsandheader) inLink_downloadandLink_download_full(1943f8e). - Fix directory eviction concurrency issues by holding
link_lockwhen freeing duplicate localLinkTabledirectories (29f51ff). - Resolve memory leak in
generate_md5sumby properly freeing the OpenSSL-allocatedmd5_digestviaOPENSSL_free(0d7a07e). - Resolve compiler warning and build issue on macOS CI by replacing GCC/Clang-specific
__attribute__((noreturn))with the C11 standard_Noreturnkeyword inlog.handutil.h(5ed8167). - Prevent duplicate background downloads by re-verifying segment existence under
w_lockaftersem_trywait()succeeds, closing the TOCTOU race where another thread could complete the download between the initialSeg_existcheck and thedl_lockacquisition (c11aad6). - Prevent null pointer dereference in
Cache_read_segmentby verifying thatad->ts->datais notNULLbefore performing the early-returnmemcpy(935b1e6). - Add strict defensive bounds checks in
Seg_existandSeg_setto reject negative or out-of-bounds offsets before dividing by block size; clamp requested read length inCache_readto remaining file size to prevent false-positive-EIOerrors near EOF; addNULLchecks forcf->linkand offset bounds checks inCache_read_segment(a527469). - Eliminate TOCTOU race conditions in
Cache_existandCache_deleteby callingunlinkdirectly and checking forENOENT, removing the redundantaccesscalls; replacestatwithfstatin tests to prevent CodeQL TOCTOU warnings (f911f7c). - Change
expected_segbcinMeta_writefromlongtooff_tto prevent silent truncation on 32-bit platforms before theINT_MAXcap; use asize_tintermediate inCache_createto avoid 32-bit overflow when computingcontent_length / blksz; add alen < 0guard inCache_readto prevent negative values wrapping to a largesize_t(96baffe). - Add a memory-only
Cachebypass shortcut inCache_openfor empty files and skip cache operations early infs_openfor zero-length files; treat stale or corrupt zero-byte cache metadata and data files as invalid inCache_existand trigger automatic cleanup; skip network activity early inLink_downloadfor zero-byte downloads to avoid invalid HTTP range headers (6c92d03). - Restructure
fs_readso that cache-enabled zero-length files (wherefi->fh == 0) return0immediately without falling through topath_download, avoiding a redundant path lookup and directory traversal (a491ade). - Cap
req_sizeinLink_downloadusing a subtraction-based bounds check to prevent potential integer overflow when the requested range extends beyond the remaining file content (a527469).