Skip to content

Release 1.3.0

Choose a tag to compare

@fangfufu fangfufu released this 27 May 12:21
· 65 commits to master since this release
106d56a

Creating a new minor release, because of the following major changes:

  • The new threading model of the cache system, which allows multiple downloads (blocking foreground and background) to be done simultaneously.
  • When cache system is enabled, the downloading thread yields as soon as there is enough data to reply to the FUSE read call.
  • New pre-commit hooks to help developers to maintain a basic level of quality in each commit.
  • New unit tests which test individual software components.
  • New system integration tests which launch a HTTP server, then mount it for read testing. There are two variants:
    • A short test which tests the filesystem against different edge case filenames.
    • A long test which do multithread read on a 1GB file with different block sizes.

Here are the detailed changes:

Added

  • Dynamically scale the number of background prefetch worker threads to match up to half of the maximum concurrent connection capacity (CONFIG.max_conns capped at DEFAULT_NETWORK_MAX_CONNS), transitioning from a single hardcoded thread to improve performance when multiple concurrent active downloads are supported (2c39a7a).
  • Add a unit test test_Cache_alloc_num_bg_workers to verify correct background worker allocation scaling under different network connection capacities (2c39a7a).
  • Implement a thread-safe manual reference-counting mechanism (ActiveDownload_ref and ActiveDownload_unref) for ActiveDownload nodes to prevent Use-After-Free (UAF) vulnerabilities during concurrent read operations, background transfers, and cache teardown (9c34463).
  • Add a unit test test_Cache_free_active_downloads to verify that active download nodes are not prematurely freed while waiter threads are still active (9c34463).
  • Implement a robust shutdown handshake mechanism in Cache_free to coordinate safe cache teardown when reader threads are actively blocked on segment downloads, ensuring all threads unregister before mutexes and condition variables are destroyed (a01c9b1).
  • Add a unit test test_Cache_free_active_downloads_with_waiters to exercise and verify proper thread synchronization and early-exit cleanup during cache shutdown (a01c9b1).
  • Add standardized GPLv3/OpenSSL copyright headers and detailed Doxygen file-level \brief annotations to all C source and header files under the src/ directory to improve legal compliance and maintain consistent documentation headers (f621da7, 666d53a).
  • Add CURLOPT_PIPEWAIT setting to curl easy handles to optimize HTTP/2 multiplexing during concurrent file and directory transfers (4114977).
  • Support HTTP/3 (QUIC) protocol version negotiation with automatic and seamless fallback to HTTP/2 or HTTP/1.1 when unavailable (4114977).
  • Implement shutdown memory leak detection in DEBUG builds, recursively traversing the virtual filesystem at shutdown to tear down resources and verify that no memory leaks exist (30d7d10).
  • Add FUSE .opendir and .releasedir handlers to safely evict directories (LinkTables) from memory when they are no longer in use, using an atomic reference-counting strategy to prevent crashes in other subsystems (2fff24a).
  • Introduce a comprehensive unit test suite test_memory_tracking to validate correct allocation, reallocation, and free behavior of debug allocator wrappers (9b38614).
  • Implement asynchronous early-return download support in cache mode to improve read performance during ongoing background downloads (0fb2d06).
  • Introduce robust logging and error-checking wrappers for pthread condition variable APIs (PTHREAD_COND_INIT, PTHREAD_COND_DESTROY, PTHREAD_COND_BROADCAST, and PTHREAD_COND_WAIT) (af0e263).
  • Support concurrent background segment downloads in cache mode by replacing the single active download tracker with a linked list of ActiveDownload structs; add ActiveDownload_find, ActiveDownload_add, and ActiveDownload_remove helpers and implement double-checked locking in Cache_read_segment to prevent duplicate downloads under concurrent FUSE workers (7286314).
  • Add unit test for ActiveDownload_find verifying lookup behavior on both empty and populated active download lists (0441ced).
  • Define a custom sys_sem_t type and emulate unnamed POSIX semaphores (sem_init, sem_destroy, sem_wait, sem_post, and sem_trywait) using pthread mutex and condition variables on Apple/macOS platforms, where unnamed POSIX semaphores are unsupported (af24b36).
  • Add zero-length file coverage to integration and unit test suites, including a dedicated test group verifying size, content, and SHA-256 digest in both direct and cache modes; add unit tests for negative-len, NULL cf, and NULL cf->link guards in Cache_read (d02ac05).

Changed

  • Transition from a single global condition variable (dl_cond) to granular, block-specific condition variables integrated directly within ActiveDownload nodes to eliminate redundant wakeups of unrelated FUSE waiter threads and reduce thread scheduling overhead (a40bd74).
  • Optimize the cache wait loop inside Cache_read_segment to check status in $O(1)$ time by introducing an unlinked state flag on the ActiveDownload structure, avoiding expensive $O(N)$ list lookups on every loop iteration (42a4dd3).
  • Clean up and standardize #include directives across all source files, headers, and test suites, using forward struct declarations to break circular dependency chains (94ced41).
  • Extract common cache test setup logic into dedicated helper functions (setup_mock_link_table and setup_temp_cache_dir) inside tests/test_cache.c (2c39a7a).
  • Replace raw octal literals with symbolic permission constants (such as S_IRWXU) in test directory creation calls to address SonarCloud security hotspot alerts (2c39a7a).
  • Update README.md to document the permanent cache thread-safety improvements, including granular signaling, manual reference counting, and the shutdown handshake mechanism (7eea0ea).
  • Update USAGE.md to document the --proxy-capath and --capath flags, update the default value of --max-conns to 6, and update internal commit reference links (1864cce, 7bcbcb2, 7d76285).
  • Lower the default maximum network connection count (DEFAULT_NETWORK_MAX_CONNS) from 10 to 6 to align with the standard persistent connection limits agreed upon by all major modern web browsers (c049e6d).
  • Update tests/test_config.c configuration initialization test to assert max_conns against the configuration macro instead of a hardcoded value (c049e6d).
  • Optimize GitHub Actions workflows by separating short and long integration tests and removing redundant manual pre-commit hook stages to avoid duplicate, slow sequential CI execution (86b0e8e).
  • Refactor synchronization and memory wrappers (mutex, semaphore, condvar, and allocator) to remove trailing name string parameters and clean up debug log statements (efb8c24).
  • Remove all debug log statements (log_printf and lprintf calls under the debug log type) across FUSE, cache, and link modules to eliminate tracing overhead (efb8c24).
  • Eliminate unused helper variables (such as the static loop counter inside LinkTable_new) that were only required by the removed debug prints, preventing compiler warnings (efb8c24).
  • Optimize memory tracking in debug allocator wrappers by replacing the $O(N)$ linear linked list with a statically sized 8192-slot hash table with chaining (9b38614).
  • Optimize string sanitization loop in sanitise_LinkTable by replacing strlen() with a null-terminator check, eliminating $O(N^2)$ overhead (c8fbf51).
  • Refactor test suites into granular profiles (unit_test for C unit tests, integration_short for fast integration tests, and integration_long for intensive tests) and align documentation in src/README.md (39963f7).
  • Revise pre-commit hook configurations to isolate builds, add debug-based verification (meson-test-debug), restrict the automatic hook to fast unit tests, and provide manual pre-push hooks (all-tests and all-tests-clang) (39963f7, 76fa30b, 9ad044e, 89ac572, af4ee30).
  • Expand the GitHub Actions build matrix on Ubuntu and macOS to execute the test suite under both debug and release configurations in parallel (1554ed3).
  • Upgrade GitHub Actions workflows to use the latest major runner and dependency actions version (68dd7bb, f819603).
  • Add --short and --long flags to generate_test_files.py and isolate short and long integration test suites so that long cache tests no longer generate short fixture files redundantly; dynamically select the range-check probe file to match the active test mode (2225d0d).
  • Default generate_test_files.py to a no-op when no flag is given, removing the implicit --large alias; --all remains the canonical way to generate both the short fixture set and the 1 GB large file (3bb8737).
  • Refactor struct Cache to remove redundant time, content_length, and fs_path fields; update Meta_read/Meta_write to preserve the on-disk binary format using local variables, keeping struct Link as the single source of truth for core link attributes (63860bc).
  • Statically initialize link_lock using PTHREAD_MUTEX_INITIALIZER, removing redundant runtime initialization and resolving EINVAL crashes in unit tests where LinkSystem_init is bypassed (af24b36).
  • Extract repeated transfer-cleanup logic from Link_download into a single static Link_download_finish_transfer helper, replacing three identical lock/clear/broadcast/reset blocks (12c6888).
  • Update README.md to describe high-performance concurrent and asynchronous cache downloads, and add a developer section on memory management and synchronization wrappers (3c0d145, 439264f).
  • Update src/README.md to reflect the split integration test stages present in the GitHub Actions build.yml configuration (439264f).

Fixed

  • Resolve a critical Use-After-Free (UAF) deadlock vulnerability in the Cache_free teardown path by safely coordinating waiter exits via the new shutdown handshake and reference-counting mechanisms (a01c9b1).
  • Intercept size-zero allocations early in REALLOC_wrapper by freeing the pointer and returning NULL to resolve a double-free on glibc and standardize behavior on BSD/macOS where realloc(ptr, 0) returns a non-NULL sentinel pointer (1e1b11d, d6e3040).
  • Fix potential crashes by adding defensive NULL checks on this_link in Cache_create (796fbb9).
  • Fix potential crashes by zero-initializing stack-allocated TransferStruct variables (ts and header) in Link_download and Link_download_full (1943f8e).
  • Fix directory eviction concurrency issues by holding link_lock when freeing duplicate local LinkTable directories (29f51ff).
  • Resolve memory leak in generate_md5sum by properly freeing the OpenSSL-allocated md5_digest via OPENSSL_free (0d7a07e).
  • Resolve compiler warning and build issue on macOS CI by replacing GCC/Clang-specific __attribute__((noreturn)) with the C11 standard _Noreturn keyword in log.h and util.h (5ed8167).
  • Prevent duplicate background downloads by re-verifying segment existence under w_lock after sem_trywait() succeeds, closing the TOCTOU race where another thread could complete the download between the initial Seg_exist check and the dl_lock acquisition (c11aad6).
  • Prevent null pointer dereference in Cache_read_segment by verifying that ad->ts->data is not NULL before performing the early-return memcpy (935b1e6).
  • Add strict defensive bounds checks in Seg_exist and Seg_set to reject negative or out-of-bounds offsets before dividing by block size; clamp requested read length in Cache_read to remaining file size to prevent false-positive -EIO errors near EOF; add NULL checks for cf->link and offset bounds checks in Cache_read_segment (a527469).
  • Eliminate TOCTOU race conditions in Cache_exist and Cache_delete by calling unlink directly and checking for ENOENT, removing the redundant access calls; replace stat with fstat in tests to prevent CodeQL TOCTOU warnings (f911f7c).
  • Change expected_segbc in Meta_write from long to off_t to prevent silent truncation on 32-bit platforms before the INT_MAX cap; use a size_t intermediate in Cache_create to avoid 32-bit overflow when computing content_length / blksz; add a len < 0 guard in Cache_read to prevent negative values wrapping to a large size_t (96baffe).
  • Add a memory-only Cache bypass shortcut in Cache_open for empty files and skip cache operations early in fs_open for zero-length files; treat stale or corrupt zero-byte cache metadata and data files as invalid in Cache_exist and trigger automatic cleanup; skip network activity early in Link_download for zero-byte downloads to avoid invalid HTTP range headers (6c92d03).
  • Restructure fs_read so that cache-enabled zero-length files (where fi->fh == 0) return 0 immediately without falling through to path_download, avoiding a redundant path lookup and directory traversal (a491ade).
  • Cap req_size in Link_download using a subtraction-based bounds check to prevent potential integer overflow when the requested range extends beyond the remaining file content (a527469).