Releases: fangfufu/httpdirfs
Releases · fangfufu/httpdirfs
Release list
1.3.3
Added
- ci: Add a dedicated CodeQL scanning job targeting GitHub Actions workflows while streamlining the C/C++ analysis job (d8e07c0, c02c0d1).
Changed
- ci: Update pre-commit hook revisions for
pre-commit-hooks,codespell,mdformat, andyamlfmt(8cf9d4c). - test: Update unity test framework subproject wrap file to track the latest master branch commit (2b4c765).
- Update and expand
USAGE.mdto comprehensively document raw help options and FUSE/module flags, linking to Git commit0855c0a(2bc5306). - ci: Replace legacy Prettier hooks with
yamlfmtandmdformat(with GFM and alerts plugins) to format all Markdown and YAML files repo-wide, removing obsolete formatting bypass tags (a7d2931). - ci: Restrict automated pull request pre-commit hook runs to branches targeting the
masterbranch (30dac3e). - ci: Modernize pre-commit configuration readability with clean multi-line block literal scripts and structure the
PKG_CONFIG_PATHdefinition inbuild.ymlincrementally (4c8f561). - doc: Fix grammar regarding parsing options in
README.md(3746293).
Fixed
1.3.2
Added
- Add opt-in
--external-linksfeature to parse absolute cross-origin directory
links from HTML listings and expose them as local files and subdirectories,
featuring first-wins deduplication and same-origin scoping of headers and
credentials (edce0ca). - Add persistent caching support for external links, mapping and sanitizing
cache paths dynamically to prevent directory traversal (855e134). - Introduce comprehensive unit and integration tests verifying external links
detection, filename sanitization, same-origin credential isolation, cache
mapping, and robust error handling (edce0ca, 855e134). - Add
--cache-min-sizeand--cache-max-sizecommand-line flags and
configuration options to dynamically enforce file size thresholds for
persistent caching (1547e39). - Introduce unit and integration tests verifying caching thresholds, including
boundary conditions and bypass behavior (1547e39).
Changed
- Update macOS build job in GitHub Actions workflow to only run unit tests and
exclude integration tests, which are unsupported due to the lack of
FUSE/macFUSE in the GitHub Actions runner environment (93bb8b5). - Optimize macOS CI workflow by removing redundant packages (
openssl,curl,
andpkg-config) from the brew installation step, as these are pre-installed
in the runner environment (bd3503c). - Suppress zero-length file checks in the long integration test suite,
preventing redundant skip messages when running integration tests under the
long profile since zero-length files are only generated under the short
profile (e8c6e16). - Rewrite and expand
USAGE.mdinto a comprehensive reference manual detailing
all 35 command-line flags, option syntax, security considerations, and
defaults (37bb031). - Implement complete dynamic global state reset in
tearDown()in
tests/test_link.cto prevent configuration and root link table mutations
from leaking between unit tests (1a13d41). - Improve integration test compatibility with BSD and macOS systems by replacing
GNU-specificfind -print -quitoptions with portable POSIX pipelines using
head -n 1(47d79de).
Fixed
- Fix origin validation by canonicalizing default ports (HTTP/80 and HTTPS/443)
inis_cross_origin()checks, ensuring same-origin URLs utilizing default
ports are not misclassified as cross-origin (7ca2f2a). - Fix argument parsing and help output when parsing
--helpor--version
options internally (860453a). - Harden argument parsing error handling by terminating early with
exit(EXIT_FAILURE)instead of returning non-zero error codes from
parse_arg_list(8d13093).
1.3.1
Added
- Add comprehensive unit and integration tests for duplicate URL handling,
including unit tests forlinknames_equal,hash_str,LinkHashSet
operations, andLinkTable_parse_html(4534563). - Extend the system integration test HTTP server to inject duplicate URLs and
directories, and implement mounted directory integration tests to verify
robust FUSE duplicate deduplication (4534563). - Expose and document internal
LinkHashSetoperations and comparison helpers
inlink.husing Doxygen (4534563).
Changed
- Optimize duplicate link detection during HTML parsing by replacing the
$O(N^2)$ linear scan inHTML_to_LinkTablewith a custom open-addressing hash
set (LinkHashSet) using linear probing and a djb2-basedhash_strfunction (26c0c5c). - Refine HTTP version negotiation behavior in
Link_to_curlto gracefully fall
back to HTTP/2 (CURL_HTTP_VERSION_2_0) if negotiation for HTTP/3
(CURL_HTTP_VERSION_3) is unsupported or fails (5de0dd6).
Fixed
- Fix formatting in error log outputs by adding missing newlines (
\n) to
multiplelprintf(error, ...)calls logging curl easy setup/info errors in
src/link.candsrc/network.c(2f5cf96).
Release 1.3.0
Creating a new minor release, because of the following major changes:
- The new threading model of the cache system, which allows multiple downloads (blocking foreground and background) to be done simultaneously.
- When cache system is enabled, the downloading thread yields as soon as there is enough data to reply to the FUSE read call.
- New pre-commit hooks to help developers to maintain a basic level of quality in each commit.
- New unit tests which test individual software components.
- New system integration tests which launch a HTTP server, then mount it for read testing. There are two variants:
- A short test which tests the filesystem against different edge case filenames.
- A long test which do multithread read on a 1GB file with different block sizes.
Here are the detailed changes:
Added
- Dynamically scale the number of background prefetch worker threads to match up to half of the maximum concurrent connection capacity (
CONFIG.max_connscapped atDEFAULT_NETWORK_MAX_CONNS), transitioning from a single hardcoded thread to improve performance when multiple concurrent active downloads are supported (2c39a7a). - Add a unit test
test_Cache_alloc_num_bg_workersto verify correct background worker allocation scaling under different network connection capacities (2c39a7a). - Implement a thread-safe manual reference-counting mechanism (
ActiveDownload_refandActiveDownload_unref) forActiveDownloadnodes to prevent Use-After-Free (UAF) vulnerabilities during concurrent read operations, background transfers, and cache teardown (9c34463). - Add a unit test
test_Cache_free_active_downloadsto verify that active download nodes are not prematurely freed while waiter threads are still active (9c34463). - Implement a robust shutdown handshake mechanism in
Cache_freeto coordinate safe cache teardown when reader threads are actively blocked on segment downloads, ensuring all threads unregister before mutexes and condition variables are destroyed (a01c9b1). - Add a unit test
test_Cache_free_active_downloads_with_waitersto exercise and verify proper thread synchronization and early-exit cleanup during cache shutdown (a01c9b1). - Add standardized GPLv3/OpenSSL copyright headers and detailed Doxygen file-level
\briefannotations to all C source and header files under thesrc/directory to improve legal compliance and maintain consistent documentation headers (f621da7, 666d53a). - Add
CURLOPT_PIPEWAITsetting to curl easy handles to optimize HTTP/2 multiplexing during concurrent file and directory transfers (4114977). - Support HTTP/3 (QUIC) protocol version negotiation with automatic and seamless fallback to HTTP/2 or HTTP/1.1 when unavailable (4114977).
- Implement shutdown memory leak detection in
DEBUGbuilds, recursively traversing the virtual filesystem at shutdown to tear down resources and verify that no memory leaks exist (30d7d10). - Add FUSE
.opendirand.releasedirhandlers to safely evict directories (LinkTables) from memory when they are no longer in use, using an atomic reference-counting strategy to prevent crashes in other subsystems (2fff24a). - Introduce a comprehensive unit test suite
test_memory_trackingto validate correct allocation, reallocation, and free behavior of debug allocator wrappers (9b38614). - Implement asynchronous early-return download support in cache mode to improve read performance during ongoing background downloads (0fb2d06).
- Introduce robust logging and error-checking wrappers for pthread condition variable APIs (
PTHREAD_COND_INIT,PTHREAD_COND_DESTROY,PTHREAD_COND_BROADCAST, andPTHREAD_COND_WAIT) (af0e263). - Support concurrent background segment downloads in cache mode by replacing the single active download tracker with a linked list of
ActiveDownloadstructs; addActiveDownload_find,ActiveDownload_add, andActiveDownload_removehelpers and implement double-checked locking inCache_read_segmentto prevent duplicate downloads under concurrent FUSE workers (7286314). - Add unit test for
ActiveDownload_findverifying lookup behavior on both empty and populated active download lists (0441ced). - Define a custom
sys_sem_ttype and emulate unnamed POSIX semaphores (sem_init,sem_destroy,sem_wait,sem_post, andsem_trywait) using pthread mutex and condition variables on Apple/macOS platforms, where unnamed POSIX semaphores are unsupported (af24b36). - Add zero-length file coverage to integration and unit test suites, including a dedicated test group verifying size, content, and SHA-256 digest in both direct and cache modes; add unit tests for negative-len, NULL
cf, and NULLcf->linkguards inCache_read(d02ac05).
Changed
- Transition from a single global condition variable (
dl_cond) to granular, block-specific condition variables integrated directly withinActiveDownloadnodes to eliminate redundant wakeups of unrelated FUSE waiter threads and reduce thread scheduling overhead (a40bd74). - Optimize the cache wait loop inside
Cache_read_segmentto check status in$O(1)$ time by introducing anunlinkedstate flag on theActiveDownloadstructure, avoiding expensive$O(N)$ list lookups on every loop iteration (42a4dd3). - Clean up and standardize
#includedirectives across all source files, headers, and test suites, using forward struct declarations to break circular dependency chains (94ced41). - Extract common cache test setup logic into dedicated helper functions (
setup_mock_link_tableandsetup_temp_cache_dir) insidetests/test_cache.c(2c39a7a). - Replace raw octal literals with symbolic permission constants (such as
S_IRWXU) in test directory creation calls to address SonarCloud security hotspot alerts (2c39a7a). - Update
README.mdto document the permanent cache thread-safety improvements, including granular signaling, manual reference counting, and the shutdown handshake mechanism (7eea0ea). - Update
USAGE.mdto document the--proxy-capathand--capathflags, update the default value of--max-connsto 6, and update internal commit reference links (1864cce, 7bcbcb2, 7d76285). - Lower the default maximum network connection count (
DEFAULT_NETWORK_MAX_CONNS) from10to6to align with the standard persistent connection limits agreed upon by all major modern web browsers (c049e6d). - Update
tests/test_config.cconfiguration initialization test to assertmax_connsagainst the configuration macro instead of a hardcoded value (c049e6d). - Optimize GitHub Actions workflows by separating short and long integration tests and removing redundant manual pre-commit hook stages to avoid duplicate, slow sequential CI execution (86b0e8e).
- Refactor synchronization and memory wrappers (mutex, semaphore, condvar, and allocator) to remove trailing name string parameters and clean up debug log statements (efb8c24).
- Remove all debug log statements (
log_printfandlprintfcalls under thedebuglog type) across FUSE, cache, and link modules to eliminate tracing overhead (efb8c24). - Eliminate unused helper variables (such as the static loop counter inside
LinkTable_new) that were only required by the removed debug prints, preventing compiler warnings (efb8c24). - Optimize memory tracking in debug allocator wrappers by replacing the
$O(N)$ linear linked list with a statically sized 8192-slot hash table with chaining (9b38614). - Optimize string sanitization loop in
sanitise_LinkTableby replacingstrlen()with a null-terminator check, eliminating$O(N^2)$ overhead (c8fbf51). - Refactor test suites into granular profiles (
unit_testfor C unit tests,integration_shortfor fast integration tests, andintegration_longfor intensive tests) and align documentation insrc/README.md(39963f7). - Revise pre-commit hook configuratio...
Release 1.2.11
Added
- Add support for
--capathand--proxy-capathconfiguration options (da9b0e9). - Support
--config=pathto specify a configuration file (db65bb5). - Add core unit testing suite using the Unity framework (e50d690).
- Add system integration test suite (89e5027).
- Add a debug memory allocation tracker and hardened memory allocations (e86afe8).
Changed
- Optimize
Cache_openby caching file size locally and avoiding redundant system calls toData_size(10e5208). - Exclude integration tests from default Meson test runs, executing them only in CI or when explicitly requested (f13559b, 91a8295).
- Refactor fatal logging infrastructure (0753fc4).
- Refactor and enhance the
FREEmacro and fix format specifiers (ef80e80). - Replace custom limit macros with standard
limits.hconstants (f9d4542). - Run cache tests with multiple block sizes and recreate cache directory between iterations (6f889ac, 9fe774a).
- Install clang in build and pre-commit workflow environments, and upload Meson test logs as build artifacts (d6dd5d8, 4edc45b).
- Update documentation in
src/README.md(unit/integration test guides) andUSAGE.md(4d67613, bde2dfa).
Fixed
- Resolve concurrent cache initialization race in
fs_open(64e8ea3). - Correct off-by-one error in cache segment count calculation (4cba404).
- Prevent unbound
MOUNT_DIRerror in integration test cleanup (12b8cdd). - Check for backtrace in libc before searching for
execinfoin build configuration (ded25f9). - Reset
proxy_cafilecorrectly inConfig_init(6a88d4d). - Handle
Link_downloaderrors in cache mode (23ec785).
Version 1.2.10
Release 1.2.9
Added
- Fail early if mountpoint or FUSE module is missing (5c53ad3).
Fixed
- Improve link initialization error reporting (8911119) (#207).
- Return the correct failure exit code (5dd032a) (#208).
- Resolve mountpoint to absolute path to prevent unmount failure on termination for relative mountpoints (3a53c40) (#177).
- Removed useless
NOLINTtags (c8abdec).
Changed
Release 1.2.8
What's Changed
- fix: Allow links starting with an underscore by @HitoriSensei in #180
- Pass the --no-info option when calling help2man by @jscott0 in #191
- Fix const‑correctness for strchr/strrchr by @owentrigueros in #199
- Avoid polling if response is already there by @Arusekk in #201
- Avoid corruption on cache write errors by @Arusekk in #200
- Fix issue #190: Implement robust retry logic for downloads by @fangfufu in #203
- Fix hang and crash on 404 directory access (Issue #95) by @fangfufu in #204
- Fix #193: Parse links to intermediate subdirectories by @fangfufu in #205
- Fix argument parsing for fstab and mount.fuse compatibility (Issues #100, #102) by @fangfufu in #206
New Contributors
- @HitoriSensei made their first contribution in #180
- @jscott0 made their first contribution in #191
- @owentrigueros made their first contribution in #199
- @Arusekk made their first contribution in #201
Full Changelog: 1.2.7...1.2.8
1.2.7 - Fixed FreeBSD compilation
1.2.6
What's Changed
Fixed
- The refreshed LinkTable is now saved. (#141)
- Only one LinkTable of the same directory is created when the cache mode is enabled. (#140)
- Cache mode now works correctly with escaped URL. (#138)
- Fixed
--user-agentoption (#159) - Allow leading
./segments in links (#125) - Fix segfault due to missing LinkTable allocation (#155)
- Fix Segmentation fault if HOME not set (#162)
- Handle sites that use absolute links and sites that require the final slash (#121)
Added
- Add
--http-headeroption (#157) - Add
--cache-clearoption (#111) - Add
--refresh-timeoutto set refresh timeout for directory contents (#114)
Changed
- Improved LinkTable caching. LinkTable invalidation is now purely based on timeout. (#147)
- Replaced the GNU Autotools based build system with Meson. (#149)
- Transitioned from using libfuse 2.x to to 3.x. (#116)
- Updated OpenSSL MD5 checksum generation API usage. (#143)
New Contributors
- @jikamens made their first contribution in #121
- @bikemike made their first contribution in #114
- @chrysn made their first contribution in #125
- @michich made their first contribution in #155
- @EyitopeIO made their first contribution in #161
Full Changelog: 1.2.5...1.2.6