Virtual file system

Demis Bellot edited this page Oct 25, 2016 · 25 revisions

  1. Getting Started

    1. Creating your first project
      1. Create Service from scratch
    2. Your first webservice explained
    3. Example Projects Overview
    4. Learning Resources
  2. Designing APIs

    1. ServiceStack API Design
    2. Designing a REST-ful service with ServiceStack
      1. Simple Customer REST Example
    3. How to design a Message-Based API
    4. Software complexity and role of DTOs
  3. Reference

    1. Order of Operations
    2. The IoC container
    3. Configuration and AppSettings
    4. Metadata page
    5. Rest, SOAP & default endpoints
    6. SOAP support
    7. Routing
    8. Service return types
    9. Customize HTTP Responses
    10. Customize JSON Responses
    11. Plugins
    12. Validation
    13. Error Handling
    14. Security
    15. Debugging
    16. JavaScript Client Library (ss-utils.js)
  4. Clients

    1. Overview
    2. C#/.NET client
      1. .NET Core Clients
    3. Add ServiceStack Reference
      1. C# Add Reference
      2. F# Add Reference
      3. VB.NET Add Reference
      4. Swift Add Reference
      5. Java Add Reference
    4. Silverlight client
    5. JavaScript client
      1. Add TypeScript Reference
    6. Dart Client
    7. MQ Clients
  5. Formats

    1. Overview
    2. JSON/JSV and XML
    3. HTML5 Report Format
    4. CSV Format
    5. MessagePack Format
    6. ProtoBuf Format
  6. View Engines

    1. Razor & Markdown Razor
    2. Markdown Razor
  7. Hosts

    1. IIS
    2. Self-hosting
    3. Messaging
    4. Mono
  8. Security

    1. Authentication
    2. Sessions
    3. Restricting Services
    4. Encrypted Messaging
  9. Advanced

    1. Configuration options
    2. Access HTTP specific features in services
    3. Logging
    4. Serialization/deserialization
    5. Request/response filters
    6. Filter attributes
    7. Concurrency Model
    8. Built-in profiling
    9. Form Hijacking Prevention
    10. Auto-Mapping
    11. HTTP Utils
    12. Dump Utils
    13. Virtual File System
    14. Config API
    15. Physical Project Structure
    16. Modularizing Services
    17. MVC Integration
    18. ServiceStack Integration
    19. Embedded Native Desktop Apps
    20. Auto Batched Requests
    21. Versioning
    22. Multitenancy
  10. Caching

    1. Caching Providers
    2. HTTP Caching
      1. CacheResponse Attribute
      2. Cache Aware Clients
  11. Auto Query

    1. Overview
    2. Why Not OData
    3. AutoQuery RDBMS
    4. AutoQuery Data
      1. AutoQuery Memory
      2. AutoQuery Service
      3. AutoQuery DynamoDB
  12. Server Events

    1. Overview
    2. JavaScript Client
    3. C# Server Events Client
    4. Redis Server Events
  13. Service Gateway

    1. Overview
    2. Service Discovery
  14. Encrypted Messaging

    1. Overview
    2. Encrypted Client
  15. Plugins

    1. Auto Query
    2. Server Sent Events
    3. Swagger API
    4. Postman
    5. Request logger
    6. Sitemaps
    7. Cancellable Requests
    8. CorsFeature
  16. Tests

    1. Testing
    2. HowTo write unit/integration tests
  17. ServiceStackVS

    1. Install ServiceStackVS
    2. Add ServiceStack Reference
    3. TypeScript React Template
      1. React, Redux Chat App
    4. AngularJS App Template
    5. React Desktop Apps
  18. Other Languages

    1. FSharp
      1. Add ServiceStack Reference
    2. VB.NET
      1. Add ServiceStack Reference
    3. Swift
      1. Swift Add Reference
    4. Java
      1. Add ServiceStack Reference
      2. Android Studio & IntelliJ
      3. Eclipse
  19. Amazon Web Services

    1. ServiceStack.Aws
    2. PocoDynamo
    3. AWS Live Demos
    4. Getting Started with AWS
  20. Deployment

    1. Deploy Multiple Sites to single AWS Instance
      1. Simple Deployments to AWS with WebDeploy
    2. Advanced Deployments with OctopusDeploy
  21. Install 3rd Party Products

    1. Redis on Windows
    2. RabbitMQ on Windows
  22. Use Cases

    1. Single Page Apps
      1. HTML, CSS and JS Minifiers
    2. Azure
      1. Connecting to Azure Redis via SSL
    3. Logging
    4. Bundling and Minification
    5. NHibernate
  23. Performance

    1. Real world performance
  24. Other Products

    1. ServiceStack.Redis
    2. ServiceStack.OrmLite
    3. ServiceStack.Text
  25. Future

    1. Roadmap
Clone this wiki locally

This page has moved to docs.servicestack.net/virtual-file-system


In order to access physical files in view engines from multiple sources, ServiceStack includes its own pluggable virtual file system API that lets it support multiple filesystem backends.

The virtual file system (VFS) is what allows ServiceStack to support view engines in a standard ASP.NET websites (e.g. serving directories from the root directory) as well in self-hosting stand-alone HttpListener websites and Windows Services serving from the output /bin directory as well as embedded resources inside .dlls, in memory filesystems populated at runtime, remote datastores like AWS S3 or any combination of either.

Embedded Resources

To enable ServiceStack to serve embedded resources in your Website's compiled .dll Assembly you'll need to register either an Assembly or a Type in an Assembly that contains embedded resources, e.g:

SetConfig(new HostConfig {
   EmbeddedResourceSources = { typeof(TypeInDllWithEmbeddedResources).Assembly },
   EmbeddedResourceBaseTypes = { typeof(TypeInDllWithEmbeddedResources) } 
});

By default ServiceStack automatically includes the Assembly where your AppHost is defined which since it's typically the same top-level assembly where all your Website assets are maintained, no configuration is required to serve any embedded resources which are accessible from the same path as it's defined in your VS.NET project. E.g. if you have an embedded resource in your project at /dir/file.js it would be available from the same path where ServiceStack is mounted, e.g http://localhost:1337/dir/file.js.

ServiceStack.Gap

See the ServiceStack.Gap project for different examples of how to create single .exe ILMerged applications with Embedded Resources and Compiled Razor Views.

Registering additional Virtual Path Providers

An easy way to register additional VirtualPath Providers in ServiceStack is to override the GetVirtualFileSources() method in your AppHost where you can add, remove, or re-order existing providers to change their priority. E.g. we can use this to provide an elegant solution for minifying static .html, .css and .js resources by simply pre-loading a new InMemory Virtual FileSystem with minified versions of existing files and giving the Memory FS a higher precedence so any matching requests serve up the minified version first with:

public override List<IVirtualPathProvider> GetVirtualFileSources()
{
    var existingProviders = base.GetVirtualFileSources();
    var memFs = new InMemoryVirtualPathProvider(this);

    //Get FileSystem Provider
    var fs = existingProviders.First(x => x is FileSystemVirtualPathProvider);

    //Process all .html files:
    foreach (var file in fs.GetAllMatchingFiles("*.html"))
    {
        var contents = Minifiers.HtmlAdvanced.Compress(file.ReadAllText());
        memFs.WriteFile(file.VirtualPath, contents);
    }

    //Process all .css files:
    foreach (var file in fs.GetAllMatchingFiles("*.css")
        .Where(file => !file.VirtualPath.EndsWith(".min.css")))
    {
        var contents = Minifiers.Css.Compress(file.ReadAllText());
        memFs.WriteFile(file.VirtualPath, contents);
    }

    //Process all .js files
    foreach (var file in fs.GetAllMatchingFiles("*.js")
        .Where(file => !file.VirtualPath.EndsWith(".min.js")))
    {
        try
        {
            var js = file.ReadAllText();
            var contents = Minifiers.JavaScript.Compress(js);
            memFs.WriteFile(file.VirtualPath, contents);
        }
        catch (Exception ex)
        {
            //Report any errors in StartUpErrors collection on ?debug=requestinfo
            base.OnStartupException(new Exception(
                "JSMin Error in {0}: {1}".Fmt(file.VirtualPath, ex.Message)));
        }
    }

    //Give new Memory FS highest priority
    existingProviders.Insert(0, memFs);
    return existingProviders;
}

Using a different Virtual Path Provider

You can also globally replace the VFS used by setting it in your AppHost, e.g. If you only want to use an InMemory File System:

base.VirtualPathProvider = new InMemoryVirtualPathProvider(this);

Fine-grained control on which VFS to use can also be specified on any Plugins requiring access to the FileSystem like ServiceStack's built-in HTML ViewEngines, here's how you could override the VFS used in ServiceStack's Razors support:

Plugins.Add(new RazorFormat { 
  VirtualPathProvider = new InMemoryVirtualPathProvider(this) 
});

Changing Physical File Path

You can change the physical root path from where ServiceStack serves your files from by changing Config.WebHostPhysicalPath, e.g. the current directory for self-hosts is where the .exe is run from, during development this is typically \bin\Release. You can change the self-host to serve files from your project folder with:

SetConfig(new HostConfig {
    WebHostPhysicalPath = "~/".MapProjectPath()
});

Where string.MapProjectPath() is just an extension method that goes back 2 directories ~\..\.. - resolving the project folder from the Debug/Release bin folders.

Overriding Embedded Resources with Static Files

The VFS supports multiple file source locations where you can override embedded files by including your own custom files in the same location as the embedded files. We can see how this works by overriding the built-in templates used in metadata pages:

Updating HTML and Metadata Page Templates

The HTML templates for the metadata pages are maintained as embedded html template resources.

The VFS lets you replace built-in ServiceStack templates with your own by simply copying the metadata or HtmlFormat Template files you want to customize and placing them in your Website Directory at:

/Templates/HtmlFormat.html        // The auto HtmlFormat template
/Templates/IndexOperations.html   // The /metadata template
/Templates/OperationControl.html  // Individual operation template

Which you can customize locally that ServiceStack will pick up and use instead.

Writable Virtual File System

The Virtual File System extended IVirtualFiles interface extends the read-only IVirtualPathProvider interface to offer a read/write API:

public interface IVirtualFiles : IVirtualPathProvider
{
  void WriteFile(string filePath, string textContents);
  void WriteFile(string filePath, Stream stream);
  void WriteFiles(IEnumerable<IVirtualFile> files,Func<IVirtualFile,string> toPath=null);
  void AppendFile(string filePath, string textContents);
  void AppendFile(string filePath, Stream stream);
  void DeleteFile(string filePath);
  void DeleteFiles(IEnumerable<string> filePaths);
  void DeleteFolder(string dirPath);
}

Folders are implicitly created when writing a file to folders that don't exist

The new IVirtualFiles API is available in local FileSystem, In Memory and S3 Virtual path providers:

  • FileSystemVirtualPathProvider
  • InMemoryVirtualPathProvider
  • S3VirtualPathProvider

All IVirtualFiles providers share the same VirtualPathProviderTests ensuring a consistent behavior where it's now possible to swap between different file storage backends with simple configuration as seen in the Imgur and REST Files examples.

VirtualFiles vs VirtualFileSources

As typically when saving uploaded files you'd only want files written to a single explicit File Storage provider, ServiceStack keeps a distinction between the existing read-only Virtual File Sources it uses internally whenever a static file is requested and the new IVirtualFiles which is maintained in a separate VirtualFiles property on IAppHost and Service base class for easy accessibility:

public class IAppHost
{
    // Read/Write Virtual FileSystem. Defaults to Local FileSystem.
    IVirtualFiles VirtualFiles { get; set; }

    // Cascading file sources, inc. Embedded Resources, File System, In Memory, S3.
    IVirtualPathProvider VirtualFileSources { get; set; }
}

public class Service : IService //ServiceStack's convenient concrete base class
{
    //...
    public IVirtualFiles VirtualFiles { get; set; }
    public IVirtualPathProvider VirtualFileSources { get; }
}

Internally ServiceStack only uses VirtualFileSources itself to serve static file requests. The new IVirtualFiles is a clean abstraction your Services can bind to when saving uploaded files which can be easily substituted when you want to change file storage backends. If not specified, VirtualFiles defaults to your local filesystem at your host project's root directory.

Examples

Implementing a new Virtual File System

The VFS is designed to be implementation agnostic so can be changed to use any file repository, e.g. it could easily be made to support a Redis, RDBMS, embedded Sqlite or other NoSQL back-ends.

Like most of ServiceStack's substitutable API's, the interfaces for the VFS lives in the ServiceStack.Interfaces.dll under the ServiceStack.IO namespace.

These are the API's that are needed to be implemented in order to create a new VFS:

public interface IVirtualPathProvider
{
    IVirtualDirectory RootDirectory { get; }
    string VirtualPathSeparator { get; }
    string RealPathSeparator { get; }

    string CombineVirtualPath(string basePath, string relativePath);

    bool FileExists(string virtualPath);
    bool DirectoryExists(string virtualPath);

    IVirtualFile GetFile(string virtualPath);
    string GetFileHash(string virtualPath);
    string GetFileHash(IVirtualFile virtualFile);

    IVirtualDirectory GetDirectory(string virtualPath);

    IEnumerable<IVirtualFile> GetAllMatchingFiles(
        string globPattern, int maxDepth = Int32.MaxValue);

    bool IsSharedFile(IVirtualFile virtualFile);
    bool IsViewFile(IVirtualFile virtualFile);
}

public interface IVirtualNode
{
    IVirtualDirectory Directory { get; }
    string Name { get; }
    string VirtualPath { get; }
    string RealPath { get; }
    bool IsDirectory { get; }
    DateTime LastModified { get; }
}

public interface IVirtualFile : IVirtualNode
{
    IVirtualPathProvider VirtualPathProvider { get; }
    string Extension { get; }
    string GetFileHash();
    Stream OpenRead();
    StreamReader OpenText();
    string ReadAllText();
}

public interface IVirtualDirectory : IVirtualNode, IEnumerable<IVirtualNode>
{
    bool IsRoot { get; }
    IVirtualDirectory ParentDirectory { get; }

    IEnumerable<IVirtualFile> Files { get; }
    IEnumerable<IVirtualDirectory> Directories { get; }

    IVirtualFile GetFile(string virtualPath);
    IVirtualFile GetFile(Stack<string> virtualPath);

    IVirtualDirectory GetDirectory(string virtualPath);
    IVirtualDirectory GetDirectory(Stack<string> virtualPath);

    IEnumerable<IVirtualFile> GetAllMatchingFiles(
        string globPattern, int maxDepth = Int32.MaxValue);
}