Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upBACKENDS: Add zip filesystem node #536
Conversation
This exposes a zip archive as a file system according to the ScummVM common API. The benefit of this is that generic code can use compressed zip archives transparently without having to special case for it.
|
@skristiansson : Thanks for this code. It looks reasonable, but zip file archive support has historically been controversial in the project. See: http://www.scummvm.org/faq/#6_1 You should probably send an e-mail to scummvm-devel mailing list for discussion, but we may end up limiting this to the WP8.1 backend. |
|
Yes, I have understood that this is controversial and there was some discussion on IRC already about it. Apart from the political concerns, there are a couple of technical issues as well. Anyway, I'll send a mail to the mailing list to start the discussion there. |
|
Thanks. I did see the discussion on IRC in the logs after I commented, and saw you were already aware of that FAQ point. Mea Culpa. The FAQ covers the reasons. I think the single file / piracy issue is probably spurious. This was more due to the technical problems and lack of need for this. I would suggest a different option. Rather than using ZIP, even in an uncompressed mode, it would be better to support ISO or a similar virtual disk image format: That is intrinsically designed for random, rather than sequential access, is standard and users dumping an iso of their game CDs is not hard. For floppy games, the user can either generate a disk image using an number of FOSS programs for this or use Nero etc. or you could support IMG (raw floppy dumps) as well. Doing this only requires one or two file extensions in a directory to be registered... so removes the issue you are seeing. |
|
Sure, that sounds fine to me. |
Hades32
commented
Dec 25, 2014
|
You might also consider using uncompressed zip files. These should be simpler than isos and it should be also possible to read individual files. |
|
If you have to add support for single file game data archives, then the only option which could be accepted by ScummVM Team is that you provide a utility to create such archives and they will not be standard zips or isos. E.g. simple concatenator similar to tar or Mac bundles in SCUMM engine could be an option. |
Hades32
commented
Dec 25, 2014
|
Sorry I overlooked this. Still my opinion as a user and from a technical point of view is that a format like ZIP or TAR would be best. Everything else would be too cumbersome for the user and creating a custom format sounds like reinventing the wheel... Even MS Office is using ZIPs with a certain directory structure these days... I'd also pose the question of it really must be part of the core of if you could not just put it in the WP8.1 port and di as you please... ----- Ursprüngliche Nachricht ----- @Hades32 : Thanks, but see my earlier comment i.e. "Rather than using ZIP, even in an uncompressed mode, it would be better to support ISO or a similar virtual disk image format". |
|
@sev- Hades32 raises a good point.. couldn't such a new filesystem class be compiled in for just Windows Phone? That would limit the potential for piracy. Although a pirate could in theory manually compile a Windows (or other popular system) ScummVM executable with it hacked enabled, anyone so experienced could just as easily do it with one of the other existing filesystems, such as with Arj files. |
|
Sigh. As I mentioned earlier, I am strictly against anything which could potentially affect our pristine reputation of a project not supporting piracy. All known torrents have .zip files as standard means of distributing ScummVM-related games, and although I am calling ScummVM "a MAME of adventure games", I wouldn't want to make life of pirates any easier, e.g. grab the torrent and simply run your games even while seeding, the thing which belongs to MAME these days My suggestion is thus to reuse existing solutions, particularly those for N64. It uses RO FS without compression: http://romfs.sourceforge.net/ Relevant ScummVM code could be refactored from backends/fs/n64 directory. What do you think? |
|
Well, ANY solution that would allow people to pack files into archives would lead to similar issues. If we don't support zip files, but add support for other format, then people will start using that other format instead to pack their game files. One important issue here is seeking within ZIP files: it can potentially be quite slow, especially in large files. So, to sum up: I believe that whatever format is used to pack files into archives could potentially be abused. Thus, if we do allow such functionality, I would personally prefer a standard format, rather than an exotic one, otherwise we would just piss people off by forcing them to use exotic tools. |
|
(a) suggested format is not exotic, romfs is part of any modern Linux kernel. (b) non-compressed file format makes it pointless for pirates, e.g. file size will not be lessened (c) overhead for non-compressed format is negligible (d) you missed my point of us continuing our anti-piracy course (e) our userbase has no problems with using scummvm-tools for recompressing their audio files, so why using same tools for "preparing game to be transferred to your Windows-based phone" could be a problem? After all similar actions are required for at least DC, GC and N64 and I heard zero complains in this regard. |
|
Although there is good support for romfs in Linux kernels, Windows support feels a bit... lacking. There's only one page, with Windows driver examples, here: So, to me, support for romfs under Windows isn't good enough. However, if we could provide some sort of tool to package game files into this format, then it could work. A similar format would be an uncompressed ZIP file, which wouldn't introduce additional overhead, but would have good support in all operating systems. |
|
And what does Windows driver have to do with ScummVM? @bluegr please think another time. In ScummVM romfs is used by N64, no OS at all. And my final word and veto: NO ZIP SUPPORT as means of repackaging the game data. Closing this pull request. |
|
@sev- : While I agree with your decision, though mainly for the technical reasons I listed earlier and user support reasons (Getting the users to generate uncompressed zip is likely to generate various bug reports of the type "I generated the zip file, but it isn't reading it... Did you generate it uncompressed? Errm.."). However, @bluegr 's point is quite clear. Most users who are using a Windows Phone are likely to be using Windows as their desktop operating system and thus will need to generate the romfs package from the original datafiles using Windows. Since there are not standard utilities for this on Windows or built-in support for this filesystem, we will need to get them to use a tool from scummvm-tools ... which should be fine. |
|
@skristiansson : I suggest you look at implementing this as @sev- sugggested using romfs to create a single datafile archive. See the code here for the scummvm side filesystem code used by N64 port: The N64 port uses the genromfs tool to generate the datafile archives: Not sure if there is a Windows port of this tool... |
|
Hmm... There has been previously using Cygwin, but it is very outdated: |
|
The latest release of this tool is release 0.5.2 (2007-06-26) : http://romfs.sourceforge.net/files/ Looking over the code, It is probably not very hard to build this for Windows using the MinGW toolchain... I would suggest that as a first step, prior to any work on incorporating this into the scummvm-tools as a tool for both N64 and WP8.1 |
|
Ok, romfs it is. I'm on Linux, so lack of windows tool to generate the romfs isn't a problem for me personally, |
|
@skristiansson : Agreed. My main argument for ISO was to make it easy for non-technical users to generate, but a simple tool, especially if integrated into scummvm-tools would be fine. Yes, I would suggest using a standard reference implementation first to avoid issues and get the scummvm code side working. Since scummvm-tools is licensed as GPL-2 and so is genromfs, it should be possible to integrate the code with minor modifications, rather than needing to write from scratch. Good luck on this... |
|
How about the simplest solution of all? Adding .scummvm or something like that as a file suffix to ALL the gamefiles, and handling the addition of that in the backend? That would mean only one file-extension to register, where the backend can do the rest. The tedious process of renaming all files from i.e. monster.sou -> monster.sou.scummvm could easily be made into a tool for endusers. Less overall technicality, and less changes that would be technically problematic for us. |
Hades32
commented
Jan 16, 2015
|
Another alternative would be to use the user's OneDrive. After you download a folder into your local app data directory you can access every file regardless of the file extension. ----- Ursprüngliche Nachricht ----- How about the simplest solution of all? Adding .scummvm or something like that as a file suffix to ALL the gamefiles, and handling the addition of that in the backend? That would mean only one file-extension to register, where the backend can do the rest. The tedious process of renaming all files from i.e. monster.sou -> monster.sou.scummvm could easily be made into a tool for endusers. Less overall technicality, and less changes that would be technically problematic for us. |
Hades32
commented
Jan 29, 2015
|
There might be an alternative way via folder pickers: |
|
Both OneDrive and file picker suggestions are worth investigating (especially the onedrive support, the file picker suggestion has a slight problem that you have to the view 'native gui', not sure how easily achievable that is under the current circumstances). Meanwhile I've got a proof-of-concept romfs implementation done in my windows phone 8.1 branch: It works fairly well, a lot of games I've tried with it works fine, but there's some problems with broken sword (more specifically, seems to be the libvorbis file seek callbacks that make it act up) that I need to investigate. |
skristiansson commentedNov 25, 2014
This exposes a zip archive as a file system according to
the ScummVM common API.
The benefit of this is that generic code can use compressed
zip archives transparently without having to special case for
it.
To elaborate a bit more on why I'm doing this - it's a part of the Windows Phone 8.1 port I'm working on
and there's certain restrictions what an app is allowed to gain access to on the file system.
In particular, to be able to read files from the SD-card, the application needs to register a set of
file extensions that it is allowed to read.
With the wide plethora of different types of files that can exist in the game data (to not mention files
without extension), the most viable option I saw was to add support of containing game data in an archive.
To give an idea how the zip-fs backend is thought to be used, I pushed a proof-of-concept commit
that make the posix backend use it here:
skristiansson@f808db9
I realize that pulling in code without current users might be undesirable, but this was self-contained
enough from the rest of the wp 8.1 port, so I thought I'd take a chance to get some input on this while
working on the rest.