[Suggestion] Reduce amount of extracted files #12141
Comments
|
These things are already available as ZIP files (though, perhaps not in the exact structure that you want). It would surely just be a matter of using the existing code that extracts ROMS to extract those things, but I've never been successful at using zlib to do anything, so the person to do it isn't me. I do have a Windows version of RetroArch for testing purposes (I run it in WINE). The .7z file extracts all the contents to the same folder (so that would be easy to transport; just move the entire folder). That seems different for the installer, which extracts the binary to %ProgramFiles% and the support files to %AppData%. BTW, there are currently three groups of shaders files: Cg, GLSL, and GLSLang. I have no idea why the Cg shaders are still around (even the Cg website advises to use GLSL or (on Windows) HLSL). GLSLang seems to be libretro's attempt to create a shader language that can be used with multiple graphics APIs (OpenGL 3.2+, Metal, DirectX, etc), seemingly only keeping GLSL around for mobile devices (where OpenGLES can only use GLSL shaders). Anyway, all three seem to be bundled with the Windows version of RetroArch, but I don't think they all should be in the same ZIP file. Here on Linux, Cg shaders are useless outside of the legacy X windowing system; in both DRM and Wayland, they don't work. Anyway, most of these things are optional. In fact, the reason I started using rgui as the menu driver was it was the only one that didn't complain about missing assets. |
Thank you for your answer. The folders are available as .zip files (like when we update them in RA Online Updater) but unfortunately, they can't be put as this in the file structure of retroarch :/ As you said, RA has so many useless things included in the main archive ! I think Retroarch should be minimalist regarding amount of files, and you should download more files if you want. Just like cheats, which currently has 23.000 files, imagine if it was included by default ! |
|
We will have to think of ways we can tackle this. Having to write a ton of files on an SSD for every update can definitely contribute to a lot of wear and tear, so that is an immediate concern. |
I thought so. In the meantime, it would be great to remove all useless files from the standard retroarch pack. For instance XMB has as said before, that has way too much files. This files or features could be downloaded separately maybe, or included but as a zip to extract before using. |
|
@twinaphex I checked all files of a fresh Retroarch extraction (1.9.1 windows) and here is a list of secondary or useless files that could either be removed or included as a zip file to extract :
If you add everything up you have almost 9000 files ! Retroarch 1.9.1 raw = 10500 files Literally ten times less files. I thought about a "Retroarch basic" or "lite" version that you can add removing all these files (and more) or rethink the way assets are downloaded, not providing and assets.zip with so much files, but an assets.zip with the minimum required fiels for each menu driver, and then a themes.zip with all other assets and skins of each menu driver. Some options could also be removed or hidden by default to make it more user friendly (Achievements, Images, Music, Latency, Logging, and all these things that the common user don't use at all) |
|
Update : I wanted to add that there are many useless folder containing system files that just confuse the casual user, namely : bearer All these files (DLL on windows) should be in Retroarch root folder (and btw it'd be more convenient if all system dll files in root folder were in only one folder, like "libraries") |
|
I think those all come from Qt and can't really be moved or omitted, unfortunately. The easiest thing, I think, if you don't want to see all the clutter is just make a shortcut/launcher one directory level up. |
Really ? I don't really understand why qgenericbearer.dll has to be in "bearer" folder and wouldn't be recognized in "libraries" folder for example. |
|
@twinaphex Any hope to see RA become lighter anytime soon ? It has many updates these weeks but it come with the same crazy amount of files. |
|
Updating RA on my 3DS is painful, via FTP is almost impossible and having to physically move the microSD is super annoying oh and RA is so insanely "fragmented" that Universal Updater just doesn't work and those devs are struggling to find some workaround. I really hope this issue gets seriously picked up, it's time for RA to go on a stringent file diet... |
|
I approve this idea for sure! Every time I update or move my RA folder - my word it takes ages! AND if you're using a medium with a large allocation chunk size (such as the default large ExFat SD Card format) the RA folder can take a massive amount of wasted space from small files. Also, how about the ability to put whole systems library as a singe (uncompressed) zip (with the actual 1 zip 1 rom zips inside) so that it they may be transferred as one large file? |
|
The xmb folder can be ditched completely for the 3DS release. rgui is plenty as default and removing xmb significantly reduces the number of files to copy like 4519 less, it's bonkers |
This has been addressed now today, should be implemented in nightlies and should be ready for the next stable. |
Pushed this change through now for the non-WiiU platforms as well, should start being stripped out of nightlies (and the next stable) from now on. |
Removed this too for non-Switch platforms |
While your main point still stands, we did change things so that the folders/files for other platforms are removed now before the package is built (i.e. Linux version no longer has autoconfig/dinput and autoconfig/xinput, and likewise for Windows with autoconfig/udev and autoconfig/x, just to cite one example). |
|
Some more optimizations have taken place on the infrastructure level:
|
No love for the 3DS? Copying thousands of files to a microsd is really frustrating and even more so through ftp. I mentioned in a previous comment that the xmb folder can be completely removed from the release and made optional |
|
And then if the user selects XMB, all assets would be missing and they'd complain about that. Would anyone miss it if XMB were removed from the 3Ds build? It probably doesn't have the RAM headroom anyway and people are probably better off with RGUI (or Ozone/MaterialUI, although they would eat some RAM too) |
Can it be downloaded at that point? It's 4500+ files alone it would make quite the dent |
|
I think XMB issue should be addressed somehow. |
|
The issue needs more thought and more work than you guys might realize. So just be appreciative of the current reductions for now and maybe some solutions will emerge in the next upcoming weeks |
|
For sure its already a great improvement |
|
This is great work and already will speed up copying the folder over a network and reduce SSD wear! Well done Libretro Team! |
|
@LibretroAdmin Oh, I just remembered something about XMB. Actually there are additional themes installed by default, but if you only include the "monochrome" theme it'd be only 700+ files and you can get rid of 4500+ files I don't think many users would complain about having to download extra XMB themes. |
|
OK we don't have a real code solution yet for the extra icon packs but since Android already only bundles the 'monochrome' icon set as is, I've decided to take the plunge and only have the 'monochrome' icons for 3DS/Vita as well in the install package. So this should cut down on the thousands of files you guys are referring to. Code solution can hopefully come later where it will become more modular to install custom sets without it all being so hardcoded. EDIT: OK, this has been done. Will apply it for WiiU and Switch as well. |
I still think xmb could be axed entirely (I see you mention a potential code option for the future) but this is a pretty good compromise. |
|
@LibretroAdmin Are these changes planned for next update for Windows and Linux ? Because nothing changed in v1.11.0 or nightly - but it looks like it has for other system. |
Unfortunately this may have mildly broken 3DS Retroarch's GLUI menu driver (#14544), although I can't prove that it used [the now removed] Ozone assets without further testing. I was going to suggest the 'more modular' download option as part of this: Rather than downloading all themes at once, a submenu could open with a list of the themes (and perhaps a short description and preview thumbnail) so users can download asset packs individually. |
|
The prospect of using an atlas / spritesheet for PNG assets, to save on I/O calls, was proposed in libretro/retroarch-assets#435 . Ozoneozone/png/icons ozone png spritesheet ozone png spritesheet (compressed) Monochromexmb/monochome/png monochome png spritesheet monochome png spritesheet (compressed) As Ozone and Monochrome are grayscale icon sets, they will see greater savings compared to full-colour icon sets such as FlatUI/FlatUX. |
|
It would be great to remove Shaders on all platforms that support them, on Android for example there are currently +3500 shaders that I and many don't even use this and for those who want to use there is already an option to download, Removing these files not only saves space, but also shortens the installation and extraction time of the files. |
|
One quick idea - The assets for individual cores could be moved into the core packages themselves - if you've not installed an N64 core, why would you need the N64 GUI assets? These are the downsides I see though: Some of that could be solved with scripting maybe, but it doesn't sound worth the trouble. |
|
Any updates on this? Whenever I install a new version of RetroArch it extracts the shaders that I had previously deleted If not remove from base.apk at least give an option to disable shader extraction, as I said before there is already an option to download the shaders whenever you want |
|
You can use the F-Droid version, which doesn't come with anything except for the bare minimum assets needed for the GLUI menu. |
I always want to keep it up to date and I saw that F-droid only has an old update, and also I use XMB menu instead of GLUI, for me only the shaders are a problem because having to keep deleting about 4000 files with each update is annoying, the rest of the files I don't see a problem with being in the base.apk |
|
fdroid has the latest stable: You can still use XMB, you'll just need to download the assets. You can choose not to download the shaders.
That's the thing, though: you want the XMB assets but you don't want shaders. Other people want shaders but don't want the XMB assets. This is clearly a never-win situation for us. |
It would be nice to at least update monthly to keep up with the nightly version :)
I see, thanks for the answer, I will try to use the F-droid version and install the XMB menu |
There's thousand of files when we extract retroarch, which can be a pain when copying the folder elsewhere or even during the extraction.
It'd be great if files could be packed in .zip files according to categories.
Example of big folders that could be zipped :
Shaders.zip (around 4000 files)
Cheats per hardware (Nintendo - Nintendo DS.zip, containing all .cht inside)
Assets folder, with each driver, like xmb.zip.
Overlays, thumbnails... Anyway, any folder with a huge amount of files.
Files would be extracted inside retroarch each time it's needed. I can't estimate the feasibilty of such thing, but having so much much files can be really annoying.
Retroarch 1.9.0 Windows x64
The text was updated successfully, but these errors were encountered: