Improved StorageFile APIs

[jonwis]

Proposal: Improved file-access APIs for UWP and AppContainer

Summary

Provide an updated version of StorageFile that fixes performance and usability issues in the current Windows.Storage.Storage* set of types. Support apps that think in terms of file paths & handles instead of StorageFile objects.

Rationale

The current Windows.Storage.Storage* types are designed to give programmatic access to regular files, shell namespace items, and other Windows entities backed by files. A single abstraction over all those spaces is interesting but comes with high costs - namely performance and impedence-mismatch with existing application code. Project Reunion should clearly explain why StorageFile is important and - where possible - provide lower-level access to the user's files.

• The BroadFileAccess capability is "scary" to customers but required even for limited UWP scenarios.
• Many libraries and helpers take file paths instead of objects (even instead of handles)
• Apps observe serious performance costs related to using StorageFile's built in functionality
• Low-level code can access the ...FromApp API variants, but higher-level code (JS, C#) interop for them is very limited.

Scope

Capability	Priority
App has direct access to any files the user has given it using low-level APIs	Must
Feature exposes an object-oriented view of ...FromApp platform APIs	Must
APIs in this feature are synchronous after user permission is acquired	Must
Feature provides standard IStream/ISequentialStream APIs for these files	Should
App can convert between existing W.S.Storage* and types in this functionality	Should
UWPs can access Shell Namespace types directly with user approval	Should
APIs taking or returning StorageFile should return/accept these objects instead	Should
App can prompt the user for broader access without requiring an app restart	Could
App can access user data without user intervention	Won't

Open Questions

Should UWPs be given access to the IShell* family of APIs?

As these APIs are not truly universal on all editions of Windows, if an app were to program against them directly the app would not work on editions like HoloLens or Xbox.

Should Reunion warn the user about apps using this capability?

Some platforms periodically remind users that "App X is using feature Y, are you OK with that?" How should Reunion make the user aware that a "low rights" application is using capabilities?

[MarkIngramUK]

For applications that are already compatible with sandboxing (i.e. only read from %APPDATA% or %PROGRAMDATA%, and only access files explicitly chosen by a user) I don't think a continuous reminder is warranted. This could perhaps be controlled via some sandbox capabilities options that are chosen at development time (similar to the macOS model).

[eklipse2k8]

A good test of this feature would be, I can create an SQLite database in the user's documents folder. I shouldn't need permission to do this if it's a new file. This is such a common scenario that the sandbox and the politics around it have never tried to solve for. An app should never need "broadFileAccess" but they should also make it so users can easily open documents and see their saved file right there.

[soumyamahunt]

A good test of this feature would be, I can create an SQLite database in the user's documents folder.

Why not create in AppData folder itself? A lot of users have complained apps creating random folders in their library folders like Documents, Music etc. introduces clutter to their workspaces and they have more trouble in finding their Work folders. Also, this kind of behavior from developers is what prompted MS to limit storage access in first place.

[benstevens48]

It’s really great that this is being looked into, and that it’s the first issue here. UWP file performance and permissions is the thing I’ve most wanted to see an improvement in since 2015! I imagine that any permissions changes will be difficult to deliver via NuGet, so hopefully something can happen on this as soon as possible in order to ensure we don’t have to wait too long before the users have the necessary OS updates.

Below I’ve put together a collection of scenarios related to file system performance and permissions that are problematic in UWP. These are things I have encountered while building my photo-viewing app. (Maybe it’s a bit selfish of me to concentrate on these problems, but hopefully they cover a wide range of use cases). If this post is too long, feel free to move it to a separate issue. Also, I am very happy to discuss any of these issues/ideas further. The scenarios themselves come first, followed by the problems with them in UWP and some possible solutions. I hope that this project will make time to think more creatively about how to deal with some of the file permissions issues. Some of my suggestions below are just initial ideas, so don’t judge them too harshly. It’s such an important subject that I think it’s worth spending a lot of time thinking about.

The scenarios

0. The user does not necessarily keep all their images in their Pictures library

I put this in as point 0 as it influences all the others. A key principle of my app is to allow users to organise their photos how they like and not to force them into any patterns. Therefore the app cannot assume that all the user’s images are in their Pictures library, and indeed it’s designed for viewing all images, even for web designers etc who are probably not keeping images in there. Also, I’m going to assume that broadFileSystem access is not used, because that is perhaps too unrestricted and also the current user experience for it is poor.

1. User selects a folder containing a large number of files (can include sub-folders) to view in the app

My app has quite a simple concept of allowing users to select files they want it to display. They can do this by selecting a folder, optionally including subfolders, or selecting individual files etc. They can make multiple selections. It is not uncommon for a folder selected to contain 10,000+ image files. The app needs to maintain a list of the selected files (including the order they were selected in etc), therefore it needs to be able to list files in folders, and the quicker and less resource-intensive this is, the better. Obviously the app needs to be able to access the contents of these files alter on, but at this stage a path and file name is all that’s needed.

2. The app needs to be able to watch for files added/removed/modified in the folders that the user has selected

My app has the option to auto-refresh the selected files to match the contents of the folders that the user selected to view. This can include modifications to files as well as files added/removed/renamed.

3. The app makes heavy use of StorageFile.GetBasicPropertiesAsync StorageItemContentProperties. RetrievePropertiesAsync to get properties such as DateTaken and Orientation

This should ideally be fast. Also, it expects to be able to read properties simultaneously.

4. The user can select files/folders via drag and drop, and expects to be able to edit these files

This is self-explanatory.

5. The app offers a ‘Duplicate file’ function

This is self-explanatory. There is a button in the app that allows the user to duplicate the file they are currently viewing in the app. (The file is one of those selected in scenario 1). This should place a copy of the file in the same directory as that file, with the name suffixed by ‘ Copy’ or similar. It should be a one-click operation as it’s a convenience function for the user (e.g. to make a copy to edit).

6. The app offers a button to load the other files in the same folder as the file that the user is currently viewing

The user may have selected an individual file to view in scenario 1. The app would like to show a button that would load the remaining files from the same folder in this case, ideally via a 1-click operation.

7. The app maintains a ‘quick access’ or ‘favourites’ list of files. Generally, clicking on a file in this list should also load the neighbors, as launching from File Explorer does.

Users who make heavy use of this list may add hundreds or even thousands of files to it.

8. The app would like to offer a feature that allows users to save their session to a file, e.g. to create a slideshow that they can bring up straight away

Such as slideshow may consist of thousands of images, which were originally selected in multiple different ways. The saved data file would store the file paths of these images, and when the data file is loaded it needs to be able to load the associated images.

9. The user can load files by double-clicking on a file in file explorer. This should load the neighboring files.

The user needs to be able to do the same things with these files that they can do with files selected another way. An important feature of this is that the neighboring files are ordered in the same way as in File Explorer. The neighboring files list should be resilient to problems with the file system indexer.

The challenges faced in a UWP app

Scenario 1

The challenge here is simply that listing the StorageItems in a StorageFolder is incredibly slow and resource intensive compared to using the standard .Net/Win32 API to list the file paths. A quick test showed that in .Net it takes about 0.003ms per file, whereas with UWP StorageItems it takes about 2ms per file, which is around 700 times slower. Also, looking at the Runtime Broker, UWP uses about 70kB memory per file, which is a huge cost when loading 10,000+ files, whereas .Net uses around 0.1kB per file (that’s a very rough estimate).

Scenario 2

UWP has the StorageFileQueryResult.ContentsChanged method for watching for file system changes. However, it has absolutely no configuration options for what sort of changes to watch for, so it will fire on any change, even something like last access time I think, and include subfolders. (I agree that if you have no configuration options that firing on any change is the best option). It also does not tell you what changed. Combined with the speed issues in scenario 2, this makes for a doubly bad experience. If you look at the .Net docs of file system watching, they give an example of watching for changes in “C:\”, which made me laugh. In UWP, assuming that you managed to get permission for this folder, in order to determine what changed, you would first have to enumerate all the contents, which would literally take several days, then on a change you would have to enumerate all the contents again, taking several days, and in fact several weeks if you also want to check the date modified to detect file modifications, and look at the difference. Also, at one stage, I tried adding a separate watching for each of 600 sub-folders that had been added to the app (in order to avoid having to query all the files for changes) and basically it ground to a halt – it seems to create a new thread for each watcher which is not necessarily very efficient.

Scenario 3

I added this because I’m not aware of a way to do this easily in .Net without writing interop code. Also, for any alternative to StorageFile in UWP, it needs to be possible to get these properties ideally without the overhead of creating a StorageFile. Also, there are some issues with these in UWP. Firstly, if you call any of the Get..PropertiesAsync methods apart from RetrievePropertiesAsync simultaneously on the same StorageFile instance, they will fail, even though they are read-only as far as I can see so have no reason to. Secondly, fetching System.Size and System.DateModified using RetrievePropertiesAsync after the first time does not return an up-to-date result. Thirdly, there can be issues with scenario 9, which I’ll mention later. Finally, using SorageFile.GetBasicPropertiesAsync to get the file modified data is incredibly slow compared to .Net. in .Net it takes about 0.05ms per file. In UWP, it takes about 5ms per file, which is 100 times slower, and also adds about an extra 60kB per file (if you keep the StorageFile), compared to about 0.05kB per file in .Net.

Scenario 4

Currently in UWP if a user drags and drop a file/folder from File Explorer into the app, the file/folder is read-only, so can’t be edited, unless the app has access to the file path via some other means.

Scenario 5

This duplicate button is not possible in UWP if the user opened the file individually or by double-clicking in File Explorer, unless the file is in the Pictures library (assuming the Pictures library capability). Worse, the app cannot even open the save file dialog to the current folder, since there is no way to set the starting folder of the save file dialog.

Scenario 6

This is a similar problem to scenario 5. If the user opened the file individually this is impossible, unless the file is in the Pictures library.

Scenario 7

The only general way to save access to files is via the FutureAccessList. However, this is beset with problems. It has quite a small limit of 1000 files which is easily exceeded. The developer has to write code to handle the case when this happens, which is hampered by the fact that any of the APIs surround the FutureAccessList are quite slow, typically taking several milliseconds, and hence such a function has to be carefully debounced so as not to impact performance of the app. Furthermore, the app has to maintain a list of the tokens it is using and ensure this is in sync with the list, which is awkward. It would be easier if the list could be accessed via file path, with a use count. This could be stored in the metadata currently but for some reason in order to obtain the metadata for a token you have to enumerate the whole list. I find that whenever I add a feature to my app that requires this list, it takes me a couple of days to get the logic right, and even then it’s full of annoying compromises. Compare this with a .Net app where you can simply forget about the whole thing and spend zero days on it.

Scenario 8

Due to the fact that the slideshow list would be stored in an external file which could then be deleted without the knowledge of the app, and also because this could exceed 1000 files, the FutureAccessList cannot be used for this purpose, so it is impossible unless the files are in the Pictures library.

Scenario 9

There are currently various issues with the neighboring files query, which I believe is related to the way it uses the file system indexer. Firstly, it is quite common for the file system indexer to stop working properly. On my old PC this happened after every major Windows update. In this case, the neighboring files query would contain no files or be missing some files. I found disabling and re-enabling file content indexing fixed the problem. Also, if you use RetrievePropertiesAsync to retrieve properties for files loaded in this way, you won’t be able to access some properties. Also, some properties like title will be limited to a certain number of characters (e.g. around 250 for System.Title). Another problem is that the allowed file extensions does not always include all media files .e.g. for some reason .webm video files are not included in the neighboring files list (as of 1909).

A few suggestions for solutions

Scenarios 1-3 are performance-related, and in the case of scenario 2 the APIs are simply incomplete for UWP and need to be completed. The performance issues should be covered by the suggestions in your proposal. In my opinion in this case it is important that all APIs involving StorageFile/Folder have versions which accept the lightweight alternative. This should include an alternative for RetrievePropertiesAsync. Also, once you have permission to access a file for the current app session, you should be able to access it via any means, unlike currently where StorageFile.GetFileFromPathAsync(myAlreadySelectedStorageFile.Path) will fail. (Memory usage should be taken into account if it becomes a problem, but hopefully it shouldn’t).

Scenario 4 is a simple matter of change the default behavior to read-write, and I think most people agree with this.

Scenario 9 could be covert by enabling the app to indicate that it doesn’t want the neighboring files query to use the file system indexer, or something like that. It could also be solved by just allowing access to the folder, as long as it was possible to get the current sort order. I know this is an increase in access rights, but it would also solve some other problems which is why I mention it. The issue of some media files being omitted could be solved by Windows having a list of such allowed extensions that could be updated independently of a feature update so we do not have to wait a year for such a simple change (assuming the list was then updated to add files such as .webm).

Scenarios 5-8 are more tricky. Basically all of these require more permissive file access or additional capabilities. I have sketched out a few ideas, some of which you might consider outrageous, but I think this is a very important area that needs exploring.

• Suggestion 1 – If the user double-clicks on a file in File Explorer which launches the app, allow the app full access to that folder. This would solve some of the issues with 5 and 7 and 9 (it is very common for the user to open files in this way). Alternatively, if technically possible, allow read-write access to that folder including creating new files but only for Pictures file extensions (if the app has the Pictures library capability).
• Suggestion 2 – A bit out on a limb. Add an ‘exendedPicturesLibrary’ (and videos etc) capability which would allow read-write access to all standard folders for the users, but would be limited to picture files. This would solve most of the issues for my particular app but perhaps is not so generally applicable. Also, the app may save some settings files etc in a non-image format, so maybe it should include other file types specific to the app – it could even be required that these extensions should be specially named (e.g. start with x- or something, to ensure the app does not gain access to other files).
• Suggestion 3 – Allow the app to prompt for access to files on a file-by file basis. The problem with broadFileSystemAccess, in my opinion, is that at the moment the app prompts for it, the user will assume the app is trying to access all their files, rather than just the ones they want it to, and that will put them off. It makes sense for the app for be able to tell the system ‘I want to access this list of file and folder paths’ and then the system can prompt the user (if the app does not already have access), and say ‘Do you want to give the app access to these files’. This dialog should ensure the list is displayed in an easy to read way that is not overwhelming even if the app prompts for multiple files. There could also be a checkbox ‘Always allow this app to access all my files’ which would be the same as broad file system access, except that now it would not seem like the app was unjustifiably tying to access all files. Ideally also the system should then automatically remember that the app was given access to those files so it does not have to prompt the next time. It might be difficult to maintain a large list however, so the system could simply maintain the most recent few thousand or something. This would address most of the file system issues and remove a lot of the need for the FutureAccessList, although it would still be necessary to add files to this to guarantee future access. It should not be necessary for the user to go into settings and restart the app to achieve any of this.
• Suggestion 4 – I’m not really sure this would work. In order to support scenarios like 8, I had this idea that once the app has access to a file, it can ask the system for a future access token, and the file access permission is then stored in the token itself. For example, this could be achieved with signing using the path of the storage item, the app’s identity and a secret system key, and maybe it could include a date stamp as well. However, it might be difficult to revoke access in this case. I guess it would be best used in conjunction with suggestion 3.
• Suggestion 5 – This is a really simple suggestion that would cost noting to implement but would make using broadFileSystemAccess where needed much less of a concern for me as a developer. In the app’s listing on the Store, if you use a capability like broadFileSystemAccess, then it says something like ‘This app can access all your files’. This makes it sound like by installing the app, it can access all your files. However, it requires an additional prompt. So why not say ‘This app can access all your files if you agree to an additional dialog’ or something like that? That would be must less off-putting.

Finally, my thoughts on periodic warnings for users about apps using capabilities - personally I find this kind of notification annoying, so I would be against it. Better to make it easy for a user to investigate this, without giving them an intrusive notification. Maybe it could go so far as something like the using your location icon in the task bar, although even that is potentially annoying if there are too many.

Thanks for your patience in reading that!

[eklipse2k8]

Why not create in AppData folder itself?

Sorry @soumyamahunt, but this is a limitation. What if the user wants to copy the file to a USB drive or put it in their OneDrive storage? I'm forced to build a file manager UI so they can "export" the document, when the file system provides all of this functionality already.

AppData is for caches and app metadata, but not user documents that they want to access, move around or do as they please with. AppData should be considered off limits to users mucking around with anyway.

[soumyamahunt]

Sorry @soumyamahunt, but this is a limitation. What if the user wants to copy the file to a USB drive or put it in their OneDrive storage? I'm forced to build a file manager UI so they can "export" the document, when the file system provides all of this functionality already.

AppData is for caches and app metadata, but not user documents that they want to access, move around or do as they please with. AppData should be considered off limits to users mucking around with anyway.

You don't have to build a file manager UI, user can go to the AppData folder of your application and copy it. If you are insistent in creating folder/file anywhere other than that you can ask user to create a folder with file picker and then add the folder/file in FutureAccessList. Automatically creating folder anywhere is huge no for me.

[MarkIvanDev]

These are the limitations I see in UWP file-access APIs besides the ones that are pointed out above

1. The broadFileSystemAccess capability limitations

• StorageLibrary APIs. You still cannot get the user's libraries when calling StorageLibrary.GetLibraryAsync even though technically, Libraries are part of the file system that is available to the user. Since the docs are saying that the app should not declare the capabilities for accessing the libraries (documentsLibrary, musicLibrary, picturesLibrary, videosLibrary) if it already uses the broadFileSystemAccess capability, declaring broad file system access should also grant access to the user's libraries through the StorageLibrary APIs

• Removable Storage Access. Although accessing removable storage requires another capability (removableStorage), broadFileSystemAccess should help lessen the restrictions. As of now, the app still needs to declare the file type associations just to access those specific file types in the user's removable storage devices.

• UNC folders. As with the previous point, accessing UNC folders needs a combination of capabilities (privateNetworkClientServer, internetClientServer, enterpriseAuthentication) to access them. Combining broadFileSystemAccess should also remove the file type restriction.

2. Low-level ...FromApp API although helps with performance is still severely limited

• Copy, Delete, Replace, Move APIs does not expose a way to monitor progress. CopyFileEx, although available to UWP, still only accepts a file path to work. If only it has a variant that accepts a file handle, a workaround can be made. Or better yet, expose a FromApp api that can monitor progress.

---

To answer the Open Questions

1. Should UWPs be given access to the IShell* family of APIs?

Since these APIs are not universally available, I think UWP should not be given access to these. It would add an added complexity of checking whether the device that the app runs on supports these or not. IMHO, the Windows.Storage is already a major improvement over the Storage APIs available in previous implementations. It just needs to be updated to have more APIs that will better align with the design of UWP

2. Should Reunion warn the user about apps using this capability?

UWP apps already has a robust way of requesting capabilities (i.e. Microphone, Camera) from the user that does not require a restart. I think this design should be implemented when requesting access for ANY capability. As it stands, changing access to the File System requires an app restart that users can mistake for a crash and does not provide a good look for the apps that use these kind of capabilities.

[eklipse2k8]

You don't have to build a file manager UI, user can go to the AppData folder of your application and copy it. If you are insistent in creating folder/file anywhere other than that you can ask user to create a folder with file picker and then add the folder/file in FutureAccessList. Automatically creating folder anywhere is huge no for me.

The problem is, AppData is hidden and I don't believe it's even easy to traverse because the final document would be behind your package folder that could be anything. If anything, it's there, but it's very unfriendly to users. That's why we're writing our software right?

The scenario I see is this,

• App creates a temporary document in AppData until the user chooses to save it somewhere on their system. You shouldn't need any kind of broadFileSystem access to do that. If they have write permission in the location of their choice, and they tell the app to put the document there, then that's it!
• Now, SQLite likes to create a journaling file, and potentially a backup file incase of a bad save. The SQLite that's bundled with Windows winsqlite3.dll will want to write those files next to the actual database file.
• This is the same behavior of Microsoft Office apps. They all get to save backups right next to the actual document file.

This is not about letting an app write junk all over the place, this is about a document based application creating a good user experience.