Allow users to change the location of PreValue Sources json files

[CantSeeSharp]

Hi all, it was suggested to me by an admin on the Discord server to write a feature request for the following.

Allow the location of PreValue Sources JSON files to be changed.

Would make sense to add a configuration option to specify the location of the JSON files - https://our.umbraco.com/documentation/add-ons/umbracoforms/developer/Configuration/

My justification for this change is as follows

1. You will no longer have to manually copy the JSON files for every deployment
2. As more and more Umbraco sites are becoming load-balanced, this would be a step in the right direction for Umbraco Forms to be used in a load-balanced environment as you would be able to set a network share as your file store to share across multiple web servers.

I know in the past other people have run into this issue - https://our.umbraco.com/forum/using-umbraco-and-getting-started/102260-umbraco-forms-prevalue-sources-not-copied-across-load-balancer

[NikRimington]

It would be great if the blob storage provider for Umbraco came with a Forms provider as well so all of the Forms files can be stored in blob storage. I'm pretty sure this used to be possible in v7/v8 with the old provider.

This caught us by surprise and caused our client to lose a load of prevalue sources they had set up when we did a deployment as we assumed (maybe wrongly) that the files were stored in the database as that is how Forms is now configured by default.

Not being able to control where these files are stored easily is quite problematic as they are essentially "content" and I wouldn't expect them to get lost with a deployment.

[jamesrichardbrett]

We are seeing this for v9 sites where items should be stored in the database. I've contacted Umbraco support who say if it's v9 then this is a bug.

[CantSeeSharp]

We are seeing this for v9 sites where items should be stored in the database. I've contacted Umbraco support who say if it's v9 then this is a bug.

Using v9 here too

[AndyButland]

Just want to check I'm not getting confused with the issue here... the JSON definitions for prevalue sources are stored in the database for Forms 9+ (and Forms 8 if configured to store form definitions in the database).

I think the issue being reported here is with the text files used as the source of the values, when the particular prevalue source that uses a text file to read it's values is used. Do I have that correct please?

[NikRimington]

From my perspective yes that is correct. In v7 and v8, when you used the Azure blob file system provider in your solution those files could be stored in blob storage, but it appears the new blob storage provider doesn't work for Forms files.

It does pose a question of where should the files that are being stored as sources for pre-values be stored because deployments could override them.

[CantSeeSharp]

Hi,

Sorry if I haven't explained myself well but this is still an issue.

The front end rendering of the Prevalue sources relies on the JSON files stored under "...\umbraco\Data\UmbracoForms\PreValueTextFiles"

If for some reason these JSON files are missing then the whole page breaks like in the below image

[Myster]

We also have this issue. in our CI-CD system (using octopus deploy) a deployment will cause these files to be lost. all other files we wish to keep can be configured to be stored somewhere that is persistent. but our deployment process replaces all other website assets, and these pre-values are swept away along with the old version of the website.
UmbracoForms moved all the other form configuration into the DB but not the list of pre-values, it seems to me that if someone uploads a list of prevalues in a text file those should be stored in the DB.

[ronaldbarendse]

Let's first clear up some of the confusion: the JSON files that contain all Forms definitions (including the prevalue source configuration) can already be moved into the database since version 8.5 by enabling a setting (see documentation) and this is the default/only option in version 9.0 and up.

The text files stored by the Get value from textfile prevalue source however still saves the files to the physical filesystem, which might be an issue when running a load-balanced setup or when your CI/CD system removes the App_Data\UmbracoForms\Data (v8) or umbraco\Data\UmbracoForms (v9+) folder (which you should be able to exclude though).

We're currently looking into how to make it easier to only move the files of the text prevalue source into a different file system without having to implement your own IPreValueTextFileStorage from scratch.

---

For now, you can move the Forms saved data location as a workaround (which shouldn't include any JSON files with Forms definitions if you're storing them in the database). Please note that moving this to the media filesystem will make them publicly available (on an obscure URL, but still).

In v8 you can add the following composer and move any existing files from App_Data\UmbracoForms\Data into your media folder or another custom physical folder:

using Umbraco.Core;
using Umbraco.Core.Composing;
using Umbraco.Core.IO;
using Umbraco.Forms.Data.FileSystem;

public class FormsFileSystemForSavedDataComposer : IComposer
{
    public void Compose(Composition composition)
        // Move all saved Forms data to media file system
        => composition.RegisterUniqueFor<IFileSystem, FormsFileSystemForSavedData>(factory => factory.GetInstance<IMediaFileSystem>());
        // Move all saved Forms data to custom physical file system
        // composition.RegisterUniqueFor<IFileSystem, FormsFileSystemForSavedData>(_ => new PhysicalFileSystem("~/App_Data/UmbracoForms/Data/"));
}

And in v9+ you can do the same (but move any existing files from umbraco\Data\UmbracoForms) using:

using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Umbraco.Cms.Core.Composing;
using Umbraco.Cms.Core.DependencyInjection;
using Umbraco.Cms.Core.Hosting;
using Umbraco.Cms.Core.IO;
using Umbraco.Extensions;
using Umbraco.Forms.Data.FileSystem;

public class FormsSavedDataFileSystemComposer : IComposer
{
    public void Compose(IUmbracoBuilder builder)
        // Move all saved Forms data to media file system
        => builder.SetFormsSavedDataFileSystem(factory => new FormsFileSystemForSavedData(factory.GetRequiredService<MediaFileManager>().FileSystem));
        // Move all saved Forms data to custom physical file system
        // builder.SetFormsSavedDataFileSystem(factory =>
        // {
        //     IIOHelper ioHelper = factory.GetRequiredService<IIOHelper>();
        //     IHostingEnvironment hostingEnvironment = factory.GetRequiredService<IHostingEnvironment>();
        //     ILogger<PhysicalFileSystem> logger = factory.GetRequiredService<ILogger<PhysicalFileSystem>>();
        //
        //     var path = "~/umbraco/Data/UmbracoForms/";
        //     var rootPath = hostingEnvironment.MapPathContentRoot(path);
        //     var rootUrl = hostingEnvironment.ToAbsolute(path);
        //
        //     return new FormsFileSystemForSavedData(new PhysicalFileSystem(ioHelper, hostingEnvironment, logger, rootPath, rootUrl));
        // });
}

[ronaldbarendse]

In addition to the above: for Umbraco 8 you can easily move all Forms data (including the prevalue text files) to Azure Blob Storage by using UmbracoFileSystemProviders.Azure.Forms. Because Forms 9+ stores all definitions in the database, Umbraco.StorageProviders didn't provide a similar implementation (and creating/maintaining an additional package is probably not justified if it's only for prevalue text files).

We'll make sure to release new (patch/minor) versions of 8, 9 and 10 that will provide a public IPreValueTextFileStorage base implementation, so you can easily switch to a different file system.

[ronaldbarendse]

Besides the above workarounds, we've incorporated the feedback in the latest (upcoming) releases: Forms 8.13.4, 9.5.4 and 10.2.0 will expose a public PreValueTextFileSystemStorage class to make changing just the prevalue source text file location a lot easier.

The documentation PR is currently pending on the release of these versions, but you can already view the new IPreValueTextFileStorage page here.

We haven't added a setting for this, as you still have to manually move existing files to the new location and we:

1. Don't want to make it too easy to move the files to the media filesystem and be publicly accessible;
2. Can't configure Azure Blob Storage without adding new, tightly coupled dependencies (or creating new supporting packages).