Option to extract single file from archive

[fraganator]

Add an option to extract only a single file from a very large archive.

Things to consider:

• Best UX to handle this - Extract All checkbox when selecting individual file? Global emulator / platform setting?
• Required cache size would only be that of the file, not the entire archive. Don't clear a 20GB cache to play a 1MB game.
• Similarly if only single file, probably don't need to cache it depending on min cache size setting

See this post initial report:
https://forums.launchbox-app.com/topic/35010-archive-cache-manager/?do=findComment&comment=405266

[nixxou]

This one would be a gamechanger for me.

[fraganator]

It's the next feature on my todo list 👍 I'll hopefully have something pushed to the dev branch for testing by the weekend.

[fraganator]

Hi @nixxou - I've added single file extraction, available as an option in the config window as "Smart Extract" (default is enabled). It will extract a single file from an archive when:

1. A file has been selected in "Select ROM In Archive" menu
2. All file types in the archive are the same

The assumption here is that when all the files in an archive are the same type, the archive probably contains multiple versions of the game, and only a single file is required to extract. If the file types differ, it's assumed all files are required to extract (e.g. cue+bin).

I've attached plugin v2.12 beta 1 for you to test. It also includes the second part of the idea mentioned in #19, where you can select the emulator in the file selection window.

Could you let me know if the new single file extraction works for your archives?

ArchiveCacheManager.v2.12-beta1.zip

[nixxou]

Thank you, i will try that (maybe not before monday, my wife gonna kill me if i spend my time on computer on week end)

[nixxou]

Just a side note, i think your "Smart Extract" is a genius idea.
But if that's not the case right now, you should make some tweaks :

• Ignore txt and nfo extensions.
• regroup extensions of the same type, like for N64 : "z64", "v64", "n64"
  (It's not so much an issue for me because in 2021 i've got all my roms in the correct format, z64, but with snes it's more problematic with a mix of .smc and .sfc in my 7z)
  You should take all this extension : gb, gbc, gba, agb, nes, fds, smc, sfc, n64, z64, v64, ndd, md, smd, gen, iso, chd, rvn, gg, gcm, 32x, bin and treat them like it's a .bin file.
  So if there is a cue on the archive, it will not make smart extract, but if you got romA.smc, romb.sfc, it will do the smart extract.

[nixxou]

I'm not great with C# and github, but i was thinking about something like that :

        public static bool GetExtractSingleFile()
        {
            List<string> fileList = new List<string>();
            var soloExt = new List<string> { ".gb", ".gbc", ".gba", ".agb", ".nes", ".fds", ".smc", ".sfc", ".n64", ".z64", ".v64", ".ndd", ".md", ".smd", ".gen", ".iso", ".chd", ".rvn", ".gg", ".gcm", ".32x", ".bin" };

            if (mGameCacheData.ExtractSingleFile == null)
            {
                mGameCacheData.ExtractSingleFile = false;
                if (Config.SmartExtract && !string.IsNullOrEmpty(mGame.SelectedFile))
                {
                    string extension = Path.GetExtension(mGame.SelectedFile);
                    fileList = Zip.GetFileList(mGame.ArchivePath, "*" + extension, true).ToList();
                    if (fileList.Count() == 0)
                    {
                        mGameCacheData.ExtractSingleFile = true;
                        Logger.Log(string.Format("Smart Extraction enabled for file \"{0}\".", mGame.SelectedFile));
                    }
                    else
                    {
                        mGameCacheData.ExtractSingleFile = true;
                        bool allowSoloExt = false;
                        if (soloExt.Contains(extension.ToLower())) allowSoloExt = true;
                        foreach (string fl in fileList)
                        {
                            string extFl = Path.GetExtension(fl).ToLower();
                            if (extFl != ".nfo" && extFl != ".txt")
                            {
                                if(allowSoloExt == false || soloExt.Contains(extFl)==false)
                                { 
                                    mGameCacheData.ExtractSingleFile = false;
                                }
                                else
                                {
                                    Logger.Log(string.Format("Smart Extraction enabled for file \"{0}\".", mGame.SelectedFile));
                                }
                            }   
                        }
                    }
                }
            }

            return (bool)mGameCacheData.ExtractSingleFile;
        }

[fraganator]

Thanks very much for the feedback, ideas, and code snippets. Much appreciated!

* Ignore txt and nfo extensions.

Great idea. I'll add an ignore list, with txt, nfo, and maybe xml and dat too. It'd make sense for this list to be configurable, but I'll hardcode it for now.

  You should take all this extension : gb, gbc, gba, agb, nes, fds, smc, sfc, n64, z64, v64, ndd, md, smd, gen, iso, chd, rvn, gg, gcm, 32x, bin and treat them like it's a .bin file.
  So if there is a cue on the archive, it will not make smart extract, but if you got romA.smc, romb.sfc, it will do the smart extract.

It might also make sense to include a 'must extract' extension list including cue, gdi, toc, mds, and so on. Any file extensions found that match this list will always extract the whole archive.

With all of the extension checks, I wonder how much can be done by 7z itself with a specially crafted command line, containing combinations of files to include and exclude. I'll see what can be done 🤔

[nixxou]

It might also make sense to include a 'must extract' extension list including cue, gdi, toc, mds, and so on. Any file extensions found that match this list will always extract the whole archive.

I don't think that's the best move. In some case, you have some rom file that run in standalone except if they are bundled with extra file (like audio CD file for SNES MSU1 roms, or texture files for some N64 games). An example :

That's why i like your approach of "Smart Extract" checking if all file share the same extensions.
It's just that it would be better if all file know to be able to run solo are treated like they share the same extension. So you can mix for exemple sfc and smc roms from snes, but if you got unknown ext file like a pcm file, it will extract the whole thing.

Also, on a side note, i'm wondering if, instead of parsing the 7z file a second time using 7z.exe, the whole logic part could be done using the data from the fileListBox ? (except if you plan to add future feature like search/filter that will alter the content of the fileListBox)

Edit : I misread your comment, i thought you were talking about list of ext that does not need extract, but it's the other way around with ext that need extract. In that case, is that still relevant with your system ? in a 7Z, a cue file will always be bundle with a bin, and since you got two different extensions and cue is not on the "solo ext" list, it will do the whole extract, right ?

[fraganator]

Thanks again for your comments and insights, I think this feature is close to complete. v2.12 beta 2 is ready to try:
ArchiveCacheManager.v2.12-beta2.zip

[nixxou]

I didn't try it yet, just took a look on the dev branch. Seems nice.
There is two things that come in mind, but that may be stupid or irrelevant :
1- Maybe the StandaloneFileList should be editable on the option page ?
2- In case we have a 7z file with a zip/7z/rar inside, can we add the zip to the StandaloneFileList AND only if we have a file that match a FileName Priority inside this sub-zip file, make the plugin treat it like a SmartExtract and launch the emulator with the file designed by the fileName Priority.

The use case for the 2 that come in my mind is bundling multiples SNES MSU1 roms into a 7z file. (Or N64 games with .htc extra texture packs, stuff like that)

[nixxou]

I made some test with the dev branch from April 16, i didn't try the last commit, so maybe it's fixed now, but i found an issue :

If there is no selected game in game-index.ini and you launch from a direct click within the launchbox UI (not using the select rom in Archive), it will decompress the whole archive to cache even if smart extract is active and the archive is filled with standalone file.

[fraganator]

If there is no selected game in game-index.ini and you launch from a direct click within the launchbox UI (not using the select rom in Archive), it will decompress the whole archive to cache even if smart extract is active and the archive is filled with standalone file.

I was a little unsure whether to do the smart extract when a single file hadn't been selected. Now that the smart extract option can be set for individual emulator\platforms, it's probably safe to remove the check for an individual file. So the only factor determining smart extract is the file extensions, but I think that's OK. I'll implement the fix later tonight.

[nixxou]

On that topic, i'm wondering something about the ListFileArchive function, if i understand your code well enought, it will execute a 7z l for each priority extension (at least until it find a match), right ?
Let says for my n64 set, my priority list look like that :
[France]*Virtual Console*[!*,[France]*GameCube*[!*,[France]*[Rev 3]*[!*,[France]*[Rev 2]*[!*,[France]*[Rev 1]*[!*,[France]*[!*,[Fr]*Virtual Console*[!*,[Fr]*GameCube*[!*,[Fr]*[Rev 3]*[!*,[Fr]*[Rev 2]*[!*,[Fr]*[Rev 1]*[!*,[Fr]*[!*,[Europe]*Virtual Console*[!*,[Europe]*GameCube*[!*,[Europe]*[Rev 3]*[!*,[Europe]*[Rev 2]*[!*,[Europe]*[Rev 1]*[!*,[Europe]*[!*,[USA]*[!*,[USA]*Virtual Console*[!*,[USA]*GameCube*[!*,[USA]*[Rev 3]*[!*,[USA]*[Rev 2]*[!*,[USA]*[Rev 1]*[!*,[USA]*[!*,*[!]*,*[!!]*,z64,n64,v64

It will take a lot of time, peeking into the 7z file. I wonder if the wildcard check should be done on the plugin side instead of 7z.
Get the full list of file from 7z only once, and check the priority/wildcard ourself.
I tried to implement it, i'm not sure if my code is reliable, it's untested (and since my coding skill are meh, you should double check) :

`
public static List ListFileArchive()
{
string stdout = string.Empty;
string selectedFilePath = string.Empty;
List fileList = new List();

        if (!LaunchInfo.Game.SelectedFile.Equals(string.Empty))
        {
            if (LaunchInfo.Extractor.List(LaunchInfo.GetArchivePath(), LaunchInfo.Game.SelectedFile.ToSingleArray()).Length > 0)
            {
                fileList.Add(LaunchInfo.Game.SelectedFile);
                Logger.Log(string.Format("Selected individual file from archive \"{0}\".", LaunchInfo.Game.SelectedFile));
                return fileList;
            }
        }

        List<string> prioritySections = new List<string>();
        prioritySections.Add(Config.EmulatorPlatformKey(LaunchInfo.Game.Emulator, LaunchInfo.Game.Platform));
        prioritySections.Add(Config.EmulatorPlatformKey("All", "All"));

        List<string> fileList_All = new List<string>();
        fileList_All = LaunchInfo.Extractor.List(LaunchInfo.GetArchivePath()).ToList();

        foreach (var prioritySection in prioritySections)
        {
            try
            {
                string[] extensionPriority = Config.GetFilenamePriority(prioritySection).Split(new string[] { "," }, StringSplitOptions.RemoveEmptyEntries);

                // Search the extensions in priority order
                foreach (string extension in extensionPriority)
                {

                    foreach (string file in fileList_All)
                    {
                        if (Wildcard.Match(file.ToLower(), string.Format("*{0}", extension.ToLower().Trim())))
                        {
                            fileList.Add(file);
                        }
                    }
                    if (fileList.Count > 0)
                    {
                        Logger.Log(string.Format("ListFileArchive : Using filename priority \"{0}\".", extension.Trim()));
                        return fileList;
                    }
                }
            }
            catch (KeyNotFoundException)
            {

            }
        }

        //A little extra, not sure of my code, to add metadata file in the end of the fileList, so if no priority is set or match, avoid to try to launch a txt file
        List<string> metadataList = Utils.SplitExtensions(Config.MetadataExtensions).ToList();
        List<string> fileList_lowpriority = new List<string>();
        foreach (string extension in metadataList)
        {
            foreach (string file in fileList_All)
            {
                if (Wildcard.Match(file.ToLower(), string.Format("*{0}", extension.ToLower().Trim())))
                {
                    fileList_lowpriority.Add(file);
                }
            }
        }
        if (fileList_lowpriority.Count > 0)
        {
            fileList = fileList_All.Except(fileList_lowpriority).ToList();
            fileList.AddRange(fileList_lowpriority);
            return fileList;
        }
       return fileList_All;

        
    }

`
As for the wildcard class, i use the code found here : https://stackoverflow.com/a/65839522

Edit : I make a test on one of my archive (not a big one), i went down from 1808,2772ms. to 387,024ms.
So yeah, it's not that big of a deal

[fraganator]

Edit : I make a test on one of my archive (not a big one), i went down from 1808,2772ms. to 387,024ms.
So yeah, it's not that big of a deal

Thanks for testing that out. I ran some tests here and found repeated calls to 7z.exe l take roughly 20ms each to complete. So it does add up if there are many priorities like in your example. I've raised issue #24 to look at improving performance.

[fraganator]

Implemented in v2.12. Issue in extracting entire archive when individual file wasn't previously selected tracked in #27.