feat(vfs): add VFS decorators and S3/database backends#21
Merged
Conversation
- new project SquidStd.Vfs.S3 (net10.0, PublishNuget=true, Minio 7.0.0) - S3FileSystemOptions: Aws (AwsConfigEntry) + Bucket, mirrors S3StorageOptions shape - S3FileSystem: IVirtualFileSystem + IDisposable over MinIO client - lazy EnsureBucketAsync (BucketExistsArgs / MakeBucketArgs) with SemaphoreSlim - ExistsAsync via StatObjectArgs, ObjectNotFoundException → false - ReadAllBytesAsync via GetObjectArgs + callback stream, ObjectNotFoundException → null - OpenReadAsync: same buffering, ObjectNotFoundException → FileNotFoundException - WriteAllBytesAsync via PutObjectArgs with MemoryStream + WithObjectSize - OpenWriteAsync returns DeferredWriteStream flushing through WriteAllBytesAsync - DeleteAsync: stat-then-remove, returns whether object existed - ListAsync: ListObjectsEnumAsync with WithRecursive(true), yields VfsEntry(Key, Size, LastModifiedDateTime) - CreateClient mirrors S3StorageService (URI parsing, WithSSL, optional WithRegion) - Dispose last, Interlocked-guarded - RegisterS3FileSystemExtensions: extension(IContainer) block using RegisterDelegate<IVirtualFileSystem> directly (no SquidStd.Vfs dep) - integration tests: MinioCollection fixture reused, 9 tests all green (Docker available locally) - slnx + test csproj updated to include new project
- New project SquidStd.Vfs.Database targeting net10.0 with FreeSql 3.5.310
- VfsFileEntity: BaseEntity subclass with Path/Content/Size; unique index on
Path via FreeSql class-level [Index("uq_vfs_file_path", "Path", true)]
- DatabaseFileSystem: IVirtualFileSystem backed by IDataAccess<VfsFileEntity>;
query-then-write upsert in WriteAllBytesAsync; StartsWith predicate for
ListAsync prefix filtering; DeferredWriteStream for OpenWriteAsync
- RegisterDatabaseFileSystemExtensions: extension(IContainer) DryIoc helper
- Integration tests (SQLite, no Docker): write-read round-trip, upsert
same-path deduplication, ExistsAsync, DeleteAsync, ListAsync with/without
prefix — all 7 pass
- Added SquidStd.Vfs.Database project reference to SquidStd.Tests
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| var uri = new Uri(options.Aws.ServiceUrl!, UriKind.Absolute); | ||
| var endpoint = uri.IsDefaultPort ? uri.Host : $"{uri.Host}:{uri.Port}"; | ||
|
|
||
| var minio = new MinioClient() |
|
|
||
| public async Task InitializeAsync() | ||
| { | ||
| _dbPath = Path.Combine(Path.GetTempPath(), "squidstd-vfsdb-" + Guid.NewGuid().ToString("N") + ".db"); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
New backends and composable decorators for
SquidStd.Vfs, all implementing the unchangedIVirtualFileSystem.Decorators (zero new dependencies, in
SquidStd.Vfs)ReadOnlyFileSystem— delegates reads, rejects every mutation.ScopedFileSystem— roots an inner filesystem at a path prefix (chroot-like); list paths are scope-relative, with a boundary guard so a sibling scope sharing a name prefix (tenant10vstenant1) can't leak.OverlayFileSystem— reads overlay-first then base; writes/deletes affect the overlay only; list is the union (overlay shadows base).CachingFileSystem— read-through cache: reads prefer the remote and refresh the cache; on a transport failure they fall back to the (possibly stale) cache; writes are write-through (remote then cache) and fail when the remote is unreachable.Backends (new adapter projects)
SquidStd.Vfs.S3— S3-compatible object storage (AWS S3, MinIO, Cloudflare R2, Backblaze B2) via the MinIO SDK, mirroring the existingS3StorageService. Path = object key.SquidStd.Vfs.Database— files stored as rows viaSquidStd.Database/ FreeSql (path-keyed, unique index, upsert).Headline composition — offline resilience
The seam was verified end-to-end:
S3FileSystemmaps not-found → null/false but lets MinIO transport errors propagate;CachingFileSystemcatches only transport exceptions and falls back, while a genuine not-found passes through without serving stale cache.Shared
DeferredWriteStream(inSquidStd.Vfs.Abstractions) backsOpenWriteAsyncfor the deferred backends (flush-once on dispose).Notes
SquidStd.Vfs(only*.Abstractions+ Core + their SDK).DatabaseFileSystemupsert is read-then-write (single-writer assumption), with a unique index as a hard DB-level guard.Test plan
SquidStd.Vfs.S3— 9 integration tests against a real MinIO container (Testcontainers)SquidStd.Vfs.Database— 7 integration tests against SQLite (FreeSql)