Add DLC refresh + restore license from Sqlite DB

[VitaSmith]

DLC from ux0:addcont/ is now installed during refresh, provided there exists an sce_sys/package/
directory containing: body.bin, head.bin, inst.bin stat.bin, tail.bin, temp.bin and work.bin.
Apart from work.bin, all of these files can be be restored from the official PKG downloads, as
demonstrated by pkg_dec.

Moreover, this patch adds the ability to restore one's RIFs from an ux0:/license/licenses.db
SQLite database with the schema described hereafter (the RIF column being the uncompressed
512-byte binary license):

CREATE TABLE Licenses (
CONTENT_ID TEXT NOT NULL UNIQUE,
RIF BLOB NOT NULL,
PRIMARY KEY(CONTENT_ID)
);

With both these feature and provided that one's personal licenses have been backed up in
ux0:/license/licenses.db, Vita users can conveniently restore or reinstall their legally
owned content, straight from Sony's PKG files.

Additional notes:

• Using zRIFs instead of RIFs would not make much sense, as, even if one were to own the ~30,000 titles and dlc that are available for the PS Vita, the SQlite DB would barely require 20 MB of disk space...
• Also, we use SQLite over any other form because it is designed to allow for fast queries and it is natively supported on the Vita. Plus, GUI apps like the one from http://sqlitebrowser.org/ make it very easy to manipulate from Windows/Mac/Linux.
• Please, please, please, don't be tempted to redesign the RIF lookup to use text or CSV, instead of SQLite: user-friendly tools to create/manipulate ux0:/license/licenses.db are coming, and this is what DBs are designed for. Switching to anything else than a SQLite will penalize users in the long run...
• Finally, a sample database (that does NOT contain any keys or private data) can be found at https://github.com/VitaSmith/VitaShell/tree/VitaSmith/db

[VitaSmith]

Just an additional note that the hardcoded MAX_LIST_SIZE in refresh.c, which is currently set to 256 is probably way too small. If you apply this pull request, it will probably need to be increased to 8192 or higher, as it's not difficult to imagine users with loads of DLC hitting that limit.

Or we could realloc() the list (which I considered but wanted to avoid in order to keep this initial proposal simple).

[mmozeiko]

Please, please, please, don't be tempted to redesign the RIF lookup to use text or CSV, instead of SQLite: user-friendly tools to create/manipulate ux0:/license/licenses.db are coming, and this is what DBs are designed for. Switching to anything else than a SQLite will penalize users in the long run...

I see one issue with this - users are required to learn new software. Which is not required for restoring their main game from NoNpDrm dump.

If you keep only work.bin support, you will get exactly same behavior as it is for main app refresh right now. Basically - 1) you dump your game, 2) you get your rif files, 3) rename to work.bin for safekeeping/backup, 4) when you need to restore, just copy them to device & refresh. Easy-peasy.

[mmozeiko]

It seems that there is an issue when many DLCs for one game are refreshed. I did following:

• copied 45 dlcs for same game, all have correct work.bin
• did refresh
• only 1 dlc got picked up (first folder that was created)
• after next refresh - 0 items

If instead I do following:

• copied 1 dlc to device
• do refresh - 1 new item gets installed
• copy 1 new dlc to device (for same game as before)
• do refresh - 1 new item gets installed
• repeat previous process as many times as you want.

It looks like promoter starts to ignore all other potential DLCs which exists inside addcont/TITLEID folder when it installs first one there.

Not sure the best way to solve this, but it seems that first moving all new DLCs out of addcont (to some ux0:/temp/whatever folder) and then calling promoter one by one on each folder would solve this. That's pretty much what is happening with installation already now.

[VitaSmith]

@mmozeiko, I think you missed the point I was making about the reason why the use of SQLite should not be dismissed early:

1. Noone is asking people to learn about new software like SQLite browser. The applications that create/manipulate the DB are coming. For instance, I am planning to add a new refresh feature to VitaShell that parses the license folder and automatically backs up all the licenses found on a user's Vita to licenses.db (and will create the db if it's not there). I'll probably also create Windows/Linux/Mac application (hopefully with UI) that does the same from a mounted drive as well as probably import licenses from zRIFs.
   Please understand that I was not even implying that Vita end users should ever have to learn how to use SQLite browser to modify their DB. This is only for early adopters, who want to play with the DB before the user-friendly tools are added that automate this task.

2. I don't think you quite see the point of dissociating the licence restoration from PKG restoration. There is a very logical reason for doing so, which I hope @TheOfficialFloW can understand. People who use their Vita with content they legally own should be able to back up and restore their games/dlcs using the fastest way possible (like through extracting archives created from .pkg by your pkg2zip along with sd2vita) without having to worry about their licenses, and this is the first step to achieving that. I have to stress out that, as far as I'm concerned licenses.db is the most important feature of this proposal, though it may require a bit of vision to see why that is the case. So I hope people will not simply shoot it down on account that they do not yet see how much easier it is actually going to make things for Vita users in the long run.

As to your the issue you report with multi DLC, I will check it out. It will probably be a bit of time before I can do so though... Obviously, I also need to point out that this is still an early and experimental feature, which I'm pushing out in the open so that people can test it and bugs/issues can be addressed. I know that promote is very picky about what's going on with the addcont/titleid folder. This is the reason why I had to go through creating a list of content to refresh so that promote would stop reporting SCE_ERROR_ERRNO_EBUSY while that folder was opened for parsing content...

[TheOfficialFloW]

If it's working fine, I will merge it.

[VitaSmith]

@TheOfficialFloW, thanks, but please don't forget to at least increase MAX_LIST_SIZE.

I would also suggest you publish a pre-release so that people can test and confirm that it also works for them, before doing an official release, but it's your call.

[VitaSmith]

Disregard, I missed the if and thought you were about to merge it... For now, this needs more testing, so please hold on on merging.

[TheOfficialFloW]

Can you please commit the change for the increasement? Compile it and share it on vitahacks or somewhere else. Unfortunately I don't have time to do it right now.

[VitaSmith]

I will do that.

[mmozeiko]

I can confirm that this now works very good - all newly found DLCs are refreshed correctly.

[VitaSmith]

Thanks!