VFS emulation

[petmac]

No description provided.

[VelocityRa]

Is there anything preventing us from upgrading the project to C++17? Because std::filesystem would certainly make things cleaner and easier.
Edit: This comment was made before this issue's purpose was changed.

[petmac]

I didn't try. If it works on Clang, MSVC and GCC I'd be happy to go to C++17.

[motoharu-gosuto]

Here is more general discussion about filesystem design.
Overall description of filesystem is given here: https://wiki.henkaku.xyz/vita/File_Management
If we want to implement filesystem properly we need to consider the following modules:

• SceAppUtil - responsible for mount functions.
• SceAppMgr - responsible for lots of mount functions.
• SceIofilemgr - abstract layer that provides i/o operations.
• SceLibFios2 - i/o scheduling, caching layer. It is extensively used by games. Check this log from frangarcj
  https://pastebin.com/Zcpa56ye
• ScePfsMgr - this is kernel only module responsible for pfs decryption and encryption.
• SceSdstor - this is kernel only module responsible for reading physical devices.

SceAppUtil
Details on this module are here: need to add this on wiki
As already said contains multiple mount functions and interacts with SceAppMgr.

SceAppMgr
Details on this module are here: https://wiki.henkaku.xyz/vita/SceAppMgr
As already said contains lot of mount functions and interacts with abstract i/o layer SceIofilemgr.

SceIofilemgr
Details on this module are here: https://wiki.henkaku.xyz/vita/SceIofilemgr
This module is an abstract i/o layer and it serves multiple purposes.

• It has mount/unmount functionality that maintains list of mount points internally. They are not exposed to outside world.
• It has number of standard i/o functions like open/close read/write.

SceIofilemgr is implemented as a VFS and I would suggest to take the same approach.
Details on VFS are here: https://wiki.henkaku.xyz/vita/VFS_Implementation

The idea is to have mount/unmount functions that create vfs object. vfs object should have open/close functions - thats the minimum. These functions should return vfs_node object. Or some kind of descriptor. vfs_node object should have read/write functions - thats the minimum.
These objects should be stored internally in SceIofilemgr. There should be a possibility to have a tree of vfs_nodes so that each node corresponds to one layer of filesystem.

vfs and vfs_node should be the interface. There should be 2 implementations which correlate with ScePfsMgr and SceSdstor. Both of these modules are kernel level modules. We do not have kernel level, so we will have to create those from scratch.

ScePfsMgr
I have already reversed PFS. All the code is located here: https://github.com/motoharu-gosuto/psvpfstools
At the moment I am working on API. My idea is to reuse this as a library in Vita3K.
PFS data is handled in pages of size 0x400. We need to take this into account when implementing read/write operations. They should allow random access to data of arbitrary length. However under the hood data will be read in pages.

SceSdstor
This module is an intermediate layer for physical i/o operations that are specific for the device.
In reality it routes i/o operations to SceSdif (sd card), SceMsif (memory card), SceUsbMass (usb stick).
We do not need these complications, however we have to have some node that is capable of reading host filesystem. Either streams or FILE* should do. This should be encapsulated into vfs and vfs_node.
This layer should provide ability to random access data of arbitary length.

ScePfsMgr and SceSdstor interaction
The idea behind vfs and vfs_nodes is that they are stored as a tree.
Upon calling open - you aquire top level vfs (vfs_pfs). Under the hood it also calls vfs_phys. You get new vfs_node back. Upon calling read - it calls vfs_node_pfs. Under the hood it calls vfs_node_phys. It returns raw data from phys layer which is then decrypted on pfs layer.

SceLibFios2
This is the layer that sits on top of SceIofilemgr and it should not interact with anything else related to i/o. SceIofilemgr should be abstract enough so that SceLibFios2 can perform the following tasks:

• Schedule i/o requests - you can queue i/o operations, you can prioritize i/o operations, can have delayed (scheduled for specific time) async i/o operations and i/o operations can be parallelized
• Overlays - this allows to remap i/o operations to different location. good example is patches or localization.
• Caching - caching layer for i/o operations. They are split into chunks. Chunks can be cached in RAM. Performance can increase.
• Dearchiving - transparent filter for i/o operations that dearchives your data. I assume this is used for psarc files.

[motoharu-gosuto]

I continue thinking about filesystem. How it is implemented in vita and what we should implement on our end. I want to talk about mount points. From what I understand 2 main types of mount points exist. I would call them physical and virtual.
Physical mount points correspond to real partitions on the storage. For example ux0: gro0: os0: and so on. You can find list of partitions here: https://wiki.henkaku.xyz/vita/Partitions

These partitions are usually mounted with https://wiki.henkaku.xyz/vita/SceIofilemgr#sceIoMountForDriver And additional info about mount points is here: https://wiki.henkaku.xyz/vita/SceIofilemgr#Mount_Points

Now about virtual mount points.
I am assuming that Iofilemgr function is not supposed to be used by games.
All physical partitions that are required should be mounted prior to running the game.
Game should only use virtual mount points (they are also called drives). Mounting of this drives is the responsibility of AppUtil and more widely AppMgr under the hood.
For example such functions exist:
sceAppUtilExtVideoMount - mounts video0: drive
sceAppUtilSaveDataMount - mounts savedata1: drive
sceAppUtilAddcontMount - mounts addcont1: drive
sceAppUtilPhotoMount - mounts photo0: drive
sceAppUtilMusicMount - mounts music0: drive
These drives (when mounted) get mapped to physical locations.
For example addcont1: can be mapped on ux0:addcont/TITLEID

Also I am assuming that depending on type of data - drive can be mounted normally or pfs mounted.
There is even a special function for that in Iofilemgr:
https://wiki.henkaku.xyz/vita/SceIofilemgr#scePfsMgrVfsMountForDriver(edited)
However I am still not quite clear how pfs stuff integrates in overall structure.

My initial guess is that system does not check which of the mounts to do. This is assumed by design.
For example When you call https://wiki.henkaku.xyz/vita/SceAppMgr#sceAppMgrGameDataMountForDriver
under the hood it calls
https://wiki.henkaku.xyz/vita/ScePfsMgr#mount
which calls https://wiki.henkaku.xyz/vita/SceIofilemgr#scePfsMgrVfsMountForDriver

I am assuming that if you call https://wiki.henkaku.xyz/vita/SceAppMgr#sceAppMgrPhotoMountForDriver
under the hood it will call different mount function (normal one). Not sure though.

There is also additional layer in the middle managed by AppMgr. We might not need it but I still want to mention it. Not quite sure how it works at the moment though.
When virtual drive is mounted - appmgr generates yet another mount point (which is a temporary identifier that can be used instead of TITLEID. I think so but it is not confirmed).
It consists of prefix like "td", "PD", "ms", "lm" followed by randomly generated string of integers.

This behavior is implemented in single function that is responsible for all mounting and is used by all mount and unmount functions in AppMgr. I still think that it is important to understand how it works because it glues virtual mount points (drives) to physical locations. I am reversing this function here:
https://github.com/motoharu-gosuto/psvcmd56/blob/master/src/CMD56Reversed/SceAppMgr.cpp#L912

get_random_path - up above is the one that is part of it - responsible for generating temporary mount points.

[pent0]

It's added, PFS is translated to normal filesystem on installation. All operations are non PFS