v1.4.0

This is the fifth major release of JuiceFS, and is an LTS version as well that will be maintained for 24 months.
Since v1.3.0 it has 520 commits from 57 contributors, thanks to @jiefenghuang @zhijian-pro @Xuyuchao-juice @zxh326 @YunhuiChen @polyrabbit @solracsf @chenjie4255 @CaitinChen @yuhr123 @davies @Looka149 @201341 @zhoucheng361 @cxljs @tangyoupeng @wzhallright @Light-City @JoyLiuc @mdolr @simpleqin @qingketsing @timfeirg @prOOrc @eakman-datadog @mt5225 @sw-cho @T-TRz879 @wsq77-ws @Niennienzz @vyalamar @lynus @zhitaoli-6 @VerTiGoEtrex @mwkang @lesterMing @asyqm @kyungwan-nam @ndeepak-baseten @zexee @hegdeadithyak @anthonyeleven @kungf @CodingPoeta @rifeplight @Joffref @mcassaniti @nautaa @juchaosong @krmeda-msft @yangliping @leoliulei @cuiweixie @qingzhongli @CodeRusher @sjp00556 @mzygQAQ

Highlights

• Support storage tiers with configurable storage classes (#6761,#6769,#6809,#6805,#6803,#6854,#6820,#6824,#6816,#6807,#6786,#6817,#6799,#6831,#6787,#6785,#6795,#6782,#6930,#7074,#7099,#7209)
• Support user and group quota management (#6330,#6369,#6386,#6396,#6361,#6766,#6789,#6813,#6511,#6381,#6771,#6776,#6743,#6751,#6737,#6826,#6788,#6698,#6808,#6480,#6933,#6991,#7011,#7080,#7171,#7168,#7104)
• Support checkpoint for resumable sync (#6755,#6852,#6905,#6877,#6892,#6872,#6870,#6839,#6950,#6958,#6959,#7046,#7076,#7071)
• Support metadata operation changelog (#6777,#6942,#6951,#6956,#6979,#7023,#7029)

New

• Support hadoop kerberos (#6445, #6644, #6970)
• windows: improved stability and compatibility (#6662,#6651,#6645,#6643,#6642,#6641,#6634,#6547,#6907,#6976,#7033)
• cmd/rmr: support batch unlink (#6553,#6581,#6618,#6537,#6457,#6564,#6603,#6609,#6610,#6647)
• cmd/clone: support batch clone (#6656,#6695,#6828,#6779,#6650,#6922,#6715,#6938)
• meta/redis: support client-side caching (#6495,#6857,#7015,#7021)
• meta: support trash realtime statistics (#6818,#6838,#6899,#6901)
• cache: support LRU cache (#6271)
• cmd/sync: support encryption/decryption (#6806,#6834,#6955,#6947,#7092)
• cmd/sync: support global traffic control (#6626)
• Support SM encryption for format and data at rest (#6419, #6414)
• Support end-to-end context cancellation (#6312, #6516, #6382, #6383, #6377, #6379, #6567, #6565, #6598, #6582, #6397, #6499, #6524, #6590, #6410, #6477, #6927)
• object: support cifs/smb (#6368, #6712, #6702, #6949)
• object: support native Storj object storage via storj.io/uplink (#6763,#6929,#6996,#7081)
• vfs: add support to skip trash if s attribute is set (#6359)
• cmd/fsck: support check data by path (#6475)
• cmd/fsck: add repair-dir-mode option to specify directories mode (#6561)
• cmd/mount: add max-downloads option to limit concurrent get for object storage (#6472)
• cmd/mount: add writeback-threshold-size option to filter blocks for writeback (#6447)
• cmd/mount: add fuse option ReadDirPlusAuto for automatic readdir optimization (#6299)
• cmd/mount: add network-interfaces option for IP discovery configuration (#6392)
• cmd/mount: add hide-internal option (#6307,#6911)
• cmd/mount: add META_PASSWORD_FILE env to read metadata engine password from file (#6438)
• cmd/gateway: add --bucket-name option for using custom bucket names (#6741)
• cmd: add mountpoint option to improve symlink resolution for absolute paths (#6228)
• sdk/java: support multiple subdir for hadoop-sdk (#6608)
• sdk/java: add max-downloads configuration option (#6679)
• sdk/java: add startTime option for nnbench (#6757)
• sdk/java: add support for sending metrics via Prometheus remote_write protocol (#6296)
• object: allow unordered list in fsck/backup/gc/destroy commands (#6389)
• object: add support for enabling/disabling default CRC checksum via query parameters (#6239,#6232)
• object/jfs: support upload-part stream API (#7123, #7124, #7163)
• object/azure: add managed identity authentication support (#6424)
• metrics: add metadata background jobs metrics (#6544, #6660)
• metrics: add subdir mount configuration metric (#6592)

Changed

Important

• meta/backup: delete backups older than two years (#6484)

• meta: commit new chunks in write order (#7016)
• meta: speed up background trash cleanup job (#6602)
• meta/badger: improve reliability and resource management (#6725)
• meta/kv: do lookup in a single transaction (#7039,#7047)
• meta: avoid creating massive kv tombstones for quota keys that were never set (#7142)
• meta: replace bulk doReaddir with batched DirHandler in cloneEntry to reduce memory pressure on large directories (#6469)
• meta/sql: batch chunk_ref updates to reduce SQL round-trips (#7018)
• meta/sql: change setting.Value from varchar(4096) to text (#7152)
• fs: return sorted entries when sort-dir is true (#6442, #6468)
• cache: speed up scanning (#6549, #6528, #6557, #6994)
• cache/stage: reduce the priority of the stage cache by introducing a cooldown period (#6435)
• cache: use partial reads when cache is disabled to halve read amplification (#6364)
• chunk: scale prefetcher pending buffer with worker count (#7059)
• fs: add permission check for BatchDeleteEntries (#7083)
• fuse: return immediately if readdir failed in ReadDirPlus (#7062)
• cmd/mount: warn when both readdir-cache and skip-dir-mtime are enabled as they may conflict (#6209)
• cmd/mount: warn when negative-entry-cache is used on kernels below 5.11 to avoid concurrent mkdir failures (#6302)
• cmd/mount: improve mount process kill and diagnostics (#6998, #7034)
• cmd/mount: check explicit cancel before duration (#7061)
• cmd/sync: limit cached directory entries number per level to reduce memory usage (#6652)
• cmd/sync: reduce the number of concurrent large files (#6341,#6372)
• cmd/sync: set umask in sync cluster mode to preserve correct file permissions (#6400)
• cmd/sync: only retry temporary failures (#6132)
• cmd/sync: skip non-existent paths in files-from list instead of failing (#6339)
• cmd/sync: skip empty keys to prevent errors from blank entries (#6193)
• cmd/sync: rename sync temporary files to ".jfs.xxx.tmp.xxx" (#6665)
• cmd/sync: warn and skip non-regular files during entry listing (#6954)
• cmd/sync: enhance symlink handling (#7030) (#7121)
• cmd/sync: improve sync log (#6623) (#7116)
• cmd/sync: improve usage description for update flag (#7169)
• cmd/gateway: improve performance for batch deletion of objects (#6605,#7078)
• cmd/gateway: reduce memory for list objects (#6476)
• cmd/gateway: add ETag verification in multi-part upload completion for data integrity (#6485)
• cmd/gateway: add descriptive error messages for bucket name validation failures (#6773)
• cmd/restore: improve restore progress labels and conflict description (#6888)
• cmd/gc: allow listed objects without size and mtime (#7063)
• cmd: use %q for CLI-derived strings in logs (#6804)
• cmd: add adaptive sleep time to read response (#6526)
• object/oss: support oss private link endpoint (#6686)
• object/oss: support switching signature version v1 (#6631)
• object/s3: limit the maximum number for list operations to 1000 (#6367)
• object/bos: remove redundant data validation from Put interface, already covered by SDK (#6904)
• object/wasb: support SAS token authentication via --session-token (#6926)
• object/ceph: add env JFS_OBJECT_NO_ORDER for faster listing (#7119, #7122)
• object: add API to check whether a storage is supported (#6559) (#7115)
• object: add object.Sys() interface (#6825) (#7118)
• object: add context parameter to Head method in DefaultObjectStorage (#7140)
• sdk/java: exclude gspt from libjfs.so to prevent Java process argv truncation on Linux (#6726)
• sdk: support fully closing filesystem on delete (#6939)
• utils: export LogHandle (#7005) (#7120)
• build: support make debug mode in makefile (#6573)
• build: reduce size of lite version (#7017)
• monitor: merge grafana dashboards with node_label variable (#6492)

Bug Fixes

• fuse: disable readdir-cache for special inodes like .trash to avoid showing stale entries (#6690)
• vfs: remove O_TMPFILE file directly when closed (#6464)
• meta/redis: fix potential deadlock when rename triggers checkTrash with nested transactions (#6558)
• meta/redis: close standalone client after cluster mode detection (#6948)
• meta/redis: fix duplicative prefix and loop counter (#7154)
• meta/kv: fix potential data corruption issues caused by clock diff and TiKV gc (#6217)
• meta/kv: avoid repeatedly setting the same key which can lead to excessive memory usage in CopyFileRange (#6449)
• meta/sql: fix tx_isolation setting error for MySQL <= 5.7 (#6768)
• meta/sql: avoid premature delfile marker removal (#7093)
• meta/sql: retry transaction for serialization failure (#7128, #7180)
• meta/badger: run simpleTxn as read-only transaction (#7173)
• meta: fix format decrypt error when uuid changed (#6542)
• meta: fix stale cache of the overwritten file in rename (#6893)
• meta: fix GetDirStat returning stale inodes count before flush (#6886)
• meta: fix group check when setting the setgid bit (#6943)
• meta: fix escaping when the original string contains spaces (#7159)
• meta: fix quota flush race (#7181)
• meta: fix progress bar goroutine leak (#7224,#7225)
• meta: fix openfiles cleanup goroutine leak on Shutdown (#7208)
• meta/kv: fix entries skipped by readdir during concurrent deletion (#7226)
• cache: fix disk cache full checking logic (#6675)
• cache/stage: fix duplicate upload of staged blocks (#6697)
• cache/stage: fix potential EIO when reading stage blocks (#6388)
• cache/stage: fix free stage blocks check when disk full (#7056)
• cache: fix cache-items option description and log (#6252)
• cmd/mount: add nofail to fstab mount options to improve mount reliability (#6280)
• cmd/mount: fix fstab option parsing that dropped parameter values like attr-cache=1 to attr-cache (#6420)
• cmd/mount: disable shared metrics reporting when no-bgjob is set (#6530)
• cmd/mount: fix FUSE fd leaking into child process (#7162, #7161)
• cmd/sync: fix failed object count (#6533)
• cmd/sync: fix metrics for excluded and extra objects (#6863)
• cmd/sync: fix lost count in summary log (#6928)
• cmd/sync: keep files-from symlink dirs as links (#7050)
• cmd/sync: fix --limit budget and extra-dst handling, clean up checkpoint on DeleteDst (#7195)
• cmd/sync: avoid print the secret key in the log (#7191)
• cmd/gateway: fix infinite loop in config handling (#7037)
• cmd/gateway: prevent panic on invalid uploadID (#7051)
• cmd/gateway: head file/ should always fail (#7064)
• cmd/config: fix bug where --bucket is treated as local path when --storage is not explicitly set (#6710)
• cmd/fsck: fix repair to set directory permissions to 0755
• cmd/info: fix tail empty slice in output (#6832)
• cmd/destroy: allow destroy with --force to ignore missing bucket (#6767)
• cmd/gateway: fix incorrect object name assignment in getObjectInfoNoFSLock (#6493)
• cmd/objbench: fix incorrect mtime check log message in list results (#6490)
• cmd: use a dedicated ServeMux to avoid exposing pprof/metrics (#7214)
• sdk/python: load platform-specific libjfs extension (#6967)
• sdk: close file handle in jfs_chown to prevent fd leak (#6965)
• sdk: fix close session handling (#6977)
• sdk/java: fix flags when reload (#7082)
• sdk/java: fix resource leaks, ancestor lookup, and null checks (#6964,#6961,#6960)
• sdk/java: convert delegation token issue/max timestamps to milliseconds in getDelegationToken (#7113)
• sdk/python: fix tests folder being packaged (#7067)
• sdk: fix accesslog dead loop (#7065)
• object: fix client dial delay and prefer ipv6 (#6753)
• object/bos: fix endpoint format for bucket (#6746)
• object/s3: use default credential chain in autoS3Region when accessKey is empty (#6754)
• object/azure: handle nil ArchiveStatus to prevent panic (#6843)
• object/s3: pass session token to region auto-detection for STS support (#6946)
• object/minio: fix MinIO object storage info printing to remove duplicated bucket name in path (#6310)
• object: decode S3 object listing responses based on the encoding type returned by server (#6532)
• object/oss: avoid passing empty token (#6446) (#7114)
• object/ceph: remove the check of JFS_NO_CHECK_OBJECT_STORAGE env (#6742) (#7117)
• object: nil-guard pointer fields in S3-family responses (#7133)
• object: fix S3 configuration with default checksum options (#7172)
• log: fix log in SelfCheck bandwidth warning (#7057)
• log: fix typo 'boudary' in WriteAt error message (#7058)
• log: fix compactChunk error log (#7198)

Deps

• deps: upgrade pgx/v5 from 5.7.3 to 5.9.2 (#6963)
• deps: upgrade ibm-cos-sdk-go from v1.12.1 to v1.13.0 (#6851)
• deps: upgrade qingstor sdk from v4.4.0 to v4.4.1 (#6855)
• deps: upgrade bunny sdk from v0.0.10 to v1.0.0 (#6848)
• deps: upgrade bce sdk from v0.9.221 to v0.9.263 (#6847)
• deps: upgrade azure sdk from v1.3.1 to v1.6.4 (#6842)
• deps: upgrade tos sdk from v2.7.8 to v2.9.0 (#6840)
• deps: upgrade cos sdk from v0.7.63 to v0.7.73 (#6837)
• deps: upgrade oss sdk from v1.2.1 to v1.4.1 (#6836)
• deps: update go-fuse to fix flock (#6778)
• deps: bump gmsm to v0.41.1 (#6775)
• deps: update go version to 1.25 (#6303, #6680, #6862)
• deps: upgrade aws-sdk-go-v2 from v1.36.1 to v1.41.5 (#6883)
• deps: upgrade dgraph-io/badger to v4.9.1 (#6879)
• deps: upgrade go-sql-driver/mysql to v1.9.3 (#6878)
• deps: upgrade redis sdk from v9.16.0 to v9.18.0 (#6871)
• deps: upgrade sftp sdk from v1.13.5 to v1.13.10 (#6868)
• deps: upgrade qiniu sdk from v7.25.2 to v7.26.8 (#6861)