0.41.0

Multi-Storage Client (MSC)

Breaking Changes

The default POSIX profile has been renamed from default to __filesystem__. If your code or configuration explicitly sets profile="default", please update it to profile="__filesystem__" (or remove the profile parameter entirely). Users who do not explicitly specify a profile, or who are not using the reserved POSIX profile, are not affected.

New Features

• Added multi-backend configuration support, enabling datasets distributed across multiple storage backends to be accessed through a single unified profile. See configuration reference for usage details.
• Added size-based batching to msc sync operations for improved performance and resource utilization.
• Added automatic ulimit increase for high-concurrency operations to prevent file descriptor exhaustion.
• Added GCS credentials provider support for Rust client with comprehensive refactoring.
• Implemented async metrics collection to significantly reduce OpenTelemetry overhead in telemetry operations.

Bug Fixes

• Fixed sync_from ignoring preserve_source_attributes parameter when using source_files argument.
• Fixed directory creation race conditions on distributed filesystems by implementing proper handling of concurrent mkdir operations.

Multi-Storage File System (MSFS)

New Features

• Added Prometheus/Grafana integration with dashboard support for monitoring file system operations.

Bug Fixes

• Fixed cache ttl_check_interval validation to ensure it is always positive.
• Fixed unlocked cache prune call that could cause race conditions.
• Fixed inode eviction to properly include clean cache lines.
• Fixed missing inode.touch() calls in file system operations.
• Fixed Grafana dashboard job selector for metrics.