Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Packed games support #1085

Closed
fdelapena opened this issue Jan 11, 2017 · 20 comments · Fixed by #2455
Closed

Packed games support #1085

fdelapena opened this issue Jan 11, 2017 · 20 comments · Fixed by #2455
Assignees
Labels
Enhancement FileFinder Patch available UX For issues affecting the user experience, such annoyances, counter-intuitive or ugly design
Milestone

Comments

@fdelapena
Copy link
Contributor

This feature would allow to read game data inside from zip files and other formats. This feature may be useful for systems with slow file copy, such Android via MTP.

Supporting several game installer or self extracting executable types, with the purpose of direct download and not requiring a Windows machine may be useful, too.

This would vastly improve the user experience for the Android and other non-Windows ports.

@Ghabry
Copy link
Member

Ghabry commented Jan 11, 2017

limited EXE (Setups and SFX) support is a good point, didn't think about this yet.

Adding FileFinder because thats our File IO wrapper.

@carstene1ns
Copy link
Member

There are quite a few libraries for archives, some also have a good license
(for example the 7z SDK is in the public domain)

One good, portable candidate (can even be modular, when built into the static ports):
libarchive - http://www.libarchive.org/ - BSD licensed, lots of supported formats

@ChristianBreitwieser
Copy link
Member

ChristianBreitwieser commented Jan 29, 2017

There are many different points in the player and liblcf where a File is accessed. Sometimes by C++ fstreams (Which can be overloaded to redirect reading to an archive library) and sometimes C FILE * References with stdio.h file api - which makes the whole thing complicated, before one can think of compression (fread, fwrite, fopen, usw are not easy to 'override' ).

So maybe it would be a good idea to replace/abstract the C Style File accesses in a first pass?

Some of the used libraries (lippng,lipmpg123,libsndfile,libvorbis,tremor) support custom implemented fileio functions
which can be used to access a c++ stream inside them (I dont' know about wildmidi and xmp yet).

A Quick scan over all Files which use a FileAPI:

FileFinder.cpp/h - Check

  • Abstracts PathHandling, Directories and retrieving c stdio.h FILE references as well as c++ fstreams

async_handler.cpp - Check

  • Uses c++ fstream to open json files (<- I don't get the purpose of this one, maybe related to emscripten?)

game_system.cpp - Check

  • Uses c++ fstream to parse Ineluki .link files (MP3 patch) (Stream is retrieved by filefinder)

output.cpp - Check

  • Uses c++ fstream to open a ostream for screenshots

audio_decoder.cpp/h - Check

  • Uses c FILE* for it's opening function, and the static create function

audio_resampler.cpp/h - Check

  • Inherits from audio_decoder

decoder_fmmidi.cpp/h - Check

  • Inherits from audio_decoder
  • Uses external library fmmidi to load midi files (FILE)*

decoder_libsndfile.cpp/h - Check

  • Inherits from audio_decoder
  • Uses external library libsndfile to load wav files (FILE)* <- But an abstraction for c file io api is present.

decoder_mpg123.cpp/h - Check

  • Inherits from audio_decoder
  • Uses external library libsndfile to load mp3 files (FILE)* <- But an abstraction for c file io api is present.

decoder_oggvorbis.cpp/h - Check

  • Inherits from audio_decoder
  • Uses external library libvorbis to load ogg files (FILE)*

decoder_wav.cpp/h - Check

  • Inherits from audio_decoder
  • Uses external library ??? to load wav files (FILE)*

decoder_wildmidi.cpp/h - Waiting for a pull request on wildmidi to get accepted

  • Inherits from audio_decoder
  • Uses external library wildmidi to load midi files (FILE)*

decoder_xmp.cpp/h - Check

  • Inherits from audio_decoder
  • Uses external library ??? to load mod files (FILE)*

3ds_decoder.cpp/h - Rebased on ghabrys branch where this file is nonexistent

  • Inherits from audio_decoder

audio_sdl.cpp - Check

  • Uses audio_decoder, opens FILE References for Sounds and BGM with filefinder

audio_3ds.cpp - Check

  • Uses audio_decoder, opens FILE References for Sounds and BGM with filefinder

audio_psp2.cpp - Check

  • Uses audio_decoder, opens FILE References for Sounds and BGM with filefinder

audio_al.cpp

  • Uses audio_decoder, opens FILE References for Sounds and BGM with filefinder

audio_secache.cpp - Check

  • Uses audio_decoder, opens FILE References for Sounds and BGM with filefinder

Bitmap.cpp - Check

  • Uses c FILE References for opening Image files

image_bmp.cpp/h - Check

  • loads bmp files with c FILE References (own implementation?)

image_png.cpp/h - Check

  • loads png files with c FILE References
  • Uses external library libpng to load pngfiles (FILE)* <- But an abstraction for c file io api is present.

image_xyz.cpp/h - Check

  • loads xyz file with c FILE References (own implementation?)

main_data.cpp - These acesses are specific for the system - untouched

  • Uses c FILE References in PSP2 Port and 3DS Port

liblcf

ini.cpp - Check

  • loads ini file with c FILE Reference

reader_lcf.h - Check

  • Uses c FILE Reference

reader_xml.h - Check

  • Uses c FILE Reference

writer_lcf.cpp/h - Check

  • Uses c FILE Reference

writer_xml.cpp/h - Check

  • Uses c FILE Reference

@Ghabry
Copy link
Member

Ghabry commented Jan 29, 2017

I planned to do the same but it looks like compared to me you have more time for this so I happily move the task to you.
My plan was to write something like PhysFS (whose development is stalled, so I thought writing something on my own isn't that hard) to abstract IO on files, zip, whatever away but besides some structs and function names theres no code at all, yet. ^^'

@carstene1ns
Copy link
Member

However, both projects accept "good" patches, so writing an abstraction API for them would be possible.
For example the virtual file stuff used by libsndfile or callbacks like libvorbisfile uses seem sensible.

@Ghabry
Copy link
Member

Ghabry commented Jan 29, 2017

yes async_handler.cpp is emscripten only so you can ignore this.

decoder_wav is own implementation for PCM WAV. Because libsndfile parses the whole file during load and thats slow on some platforms.

output also open a stream to a logfile

image_bmp and image_xyz are own implementations

@ChristianBreitwieser
Copy link
Member

Thanks a lot for the info,
I started a branch for each of liblcf and Player to replace the C IO with iostreams.
(And the fstreams with more abstract i and ostreams).

(remove_c_file_dependencies in ChristianBreitwieser/Player@be3615c and ChristianBreitwieser/liblcf@2850b48)

When the first replacement pass is finished every request to open a file should either go to one of the following three functions of filefinder (I added the lower two to optimize later stream implementations on read only/ write only files)

/**
 * Creates stream from UTF-8 file name.
 *
 * @param name UTF-8 string file name.
 * @param m stream mode.
 * @return NULL if open failed.
 */
std::shared_ptr<std::iostream> openUTF8(const std::string& name, std::ios_base::openmode m);

/**
* Creates stream from UTF-8 file name.
*
* @param name UTF-8 string file name.
* @param m stream mode.
* @return NULL if open failed.
*/
std::shared_ptr<std::istream> openUTF8Input(const std::string& name, std::ios_base::openmode m);

/**
* Creates stream from UTF-8 file name.
*
* @param name UTF-8 string file name.
* @param m stream mode.
* @return NULL if open failed.
*/
std::shared_ptr<std::ostream> openUTF8Output(const std::string& name, std::ios_base::openmode m);

After that i guess it's cleaner to make a pull request for these changes.

And then start to implement a Filesystem abstraction behind them in a seperate pull request?

@Ghabry
Copy link
Member

Ghabry commented Jan 30, 2017

Cool, you even thought about using the wide string API under Windows for utf16 support 👍 .

When this API was invented EasyRPG was still using C++03. Afaik you can get rid of the std::shared_ptr<> wrapping by returning via a std::move because streams are movable.

@Ghabry
Copy link
Member

Ghabry commented Jan 30, 2017

Forgot to mention: For FmMidi you can provide a custom fgetc function: Ghabry@0f716b1#diff-71a10a4568320825d9e72ab2d22df5c4R42

@ChristianBreitwieser
Copy link
Member

Leaving this here as a marker for testing:

fseek behaviour differs significantly from std::istream.seekg.
fseek resets end of file state.
seekg doesn't even work if end of file is set.

TO REMARK: ensure that functions which seek after reaching the end of file (like the seek functions in the various Audio Decoders) reset the eof state before seeking.

(TODO: Find RPG Game with as many different sound formats as possible)
(TODO: Integration Tests for AudioDecoders)

@Ghabry
Copy link
Member

Ghabry commented Jan 31, 2017

RPG Games with Music:
Midi: Our TestGame-2000.
MP3: Ib, Monigote Fantasy
MOD: AE - El últimos días de Ezra
OGG: Not supported by RPG Maker 2000, simply convert some files. The game Oneshot has a heavily patched RPG_RT and ships with OGG files. But the game has a runtime check to detect normal RPG_RT, so will not run.
WAV: Yume2kki and usually sound effects usually use this.
ADPCM WAV: Ara Fell
24bit WAV: A few WAV files of Pom gets Wifi

To test WavDecoder you need WANT_FASTWAV defined. Only normal WAVs go through it, ADPCM still goes through libsndfile. Note that theres a bug: 24bit wavs break when FASTWAV is on.

For MIDI you have to test WildMidi and FmMidi.

All other formats are only supported by one lib.

@Ghabry
Copy link
Member

Ghabry commented Feb 4, 2017

About your openal question: Dont worry too much about it. That code wasnt maintained in years and nobody is currently using it. I have a half finished branch somewhere which adds audiodecoder Ghabry@59fa5d7

@ChristianBreitwieser
Copy link
Member

ChristianBreitwieser commented Feb 4, 2017

So heres a first proof of concept filesystem abstraction behind filefinder
ChristianBreitwieser/Player@c4e4028

It's basically nothing more than moving all filesystem reliant code out of filefinder into a filesystem class,
which is currently only specialized for os filesystem accesses.

My next step is to figure out how mounting of different filesystems (os,zip,physfs,etc) could be realised without bugging the player about it. In the current POC filefinder has just an internal reference to the os filesystem. But ideally i want paths which resemble something like this after the refactor:

"os:/whatever/windows/fonts/path/font.ttf" <- Mounted automatically on start
"rtp:/system/system.png" <- Mounted when the rtp is found (may be inside an archive file?)
"game:/RPG_RT.ldb" <- Mounted when gamebrowser selects a folder/zip/sfx/whateverGameArchive or when Player is started in a game dir
"ressources:/wildmidi/wildmidi.cfg" <- Mounted when ressource directory/archive/resourcefs is found
"save:/some_save" <- Mounted after configured save path is read (or on start^^)

PS: It's not buildable without adding filesystem files to the makefile/vcproj and using my branch of liblcf.
(Because I dont't want to commit my all inclusive makefile with dynamically linked libxmp+wildmidi^^)

easyrpg_fs

Please tell me below where the flaws of the concept are! ;)

@Ghabry
Copy link
Member

Ghabry commented Feb 5, 2017

At first thanks for the work you invested to abstract the IO in the whole player.

I like that FileFinder (FF) and Filesystem (FS) are splitted now, making it look a bit more maintanable.
Recursion is handled by FF, not by FS, good.
I don't think PhysFS is needed anymore because you do the same now in that code. ^^

There are no external dependencies (except for RTP and Output::Debug) in the FileFinder code, which allows it to be used by the Android game browser via JNI 👍.

The UML looks mostly well designed for me (but you should have reduced the amounts of arrow overlaps ;))

I was wondering how FF interfaces with the Filesystems but it implements the FS-interface, so this is transparent to the FF, very cool!

It would be enough to support "xxx:/" mounts, not C:/x/mountpoint/, mounts.

MountFolder could be renamed to Mount and MountFile removed and mount should be smart enough to detect stuff like "MountFolder("game", "/data/data/YumeNikki.apk/resources/game") which initializes a ZipFS for YumeNikki.vpk and uses a path prefix of "/resources/game/" and mounts this as "game:/".
More legal Mount invocations: Mount("game", "/games/Yume Nikki") <- Mount a normal os folder as "game:/".
Mount("game", "/games/Yume Nikki.zip") <- Mount a ZipFS with path prefix "/" as "game:/").
Not legal: Mount("game", "/games.zip/Yume Nikki.zip")

Which mountpoint does the startup directory (for GameBrowser case) get? game:/ wouldn't be useful here because it's overwritten when the real game starts.
When "Mount" function becomes smart as described the GameBrowser could just throw any file it finds in the directory (after maybe checking file ext :D) at the Mount command (with mount point "gamebrowser_tmp", then scan the folder/file content and umount).

Maybe a "IsWritable" function would be useful to determine if a path is suitable for savegames/logfiles.

Do you still plan to use "save:/" as the general writable path? Currelty EasyRPG writes besides saves also logfiles and screenshots in there.

For zip you need a way to set the Encoding because many zip files are not using utf-8.

rtp:/ would be limited to only one RTP except if you get overlayfs working :D. Not important yet but the RTP redirection logic could be moved from FF to RtpFS which simply forwards to the real FS (OsFS or ZipFS) and does filename redirection in between. For this you would need a 2nd mount command that excepts the FS-Interface you want as an argument.

For resources:/ I'm not really sure about the use. Usually the wildmidi file will be part of a directory of game:/ or os:/ and can just use this path.

Do you think it makes sense to polish this and create a PR already? This way the code is at least already in the repo if you are going MIA again ^^.

@ChristianBreitwieser
Copy link
Member

ChristianBreitwieser commented Feb 5, 2017

Yep i planned to make a pull request for the first part as soon as your generic audio branch is in the code base (because i based on that branch.) Also this will give me enough time to do documentation and cleanup. (And adding the IsWritable which is a really good point)

I would include the conversion to istream/ostream as well as the initial filesystem class + os filesystem in the pull request. But before this, pull requests for liblcf and tools have to be made, as i need a stream interface in liblcf as well.

ZIPFs and MountFS is currently in developement and will need some time, mainly because i want to test it locally extensivly (also regarding performance).

Hmm i don't know what MIA means but i can imagine^^
(I will certainly do so between 10.2 and 19.2 as i'm abroad this time without my laptop ;)).

The rest of this comment are some random thoughts about the supported compression:

  • zlib contains DEFLATE algorithm which can be used for .gz and .zip archives - will be implemented first, should lead to a speedup on file reads.
  • lzma (http://7-zip.org/sdk.html) would be useful for .7z , compression is better, more cpu usage
  • rar - i haven't found a c++ lib for this yet - maybe have to port parts of the unarchiver

Seeking (as most audio decoders do) will be a a problem with compressed streams, forward seeking can be implemented by reading and discarding data, backward seeking is another thing.

Maybe it's useful to think of a standard archive structure/format for easyrpg form compressed games and creating a tool which packages to it. (Which could also be made for android for users without a pc)
Reasons:

  • Predetermined encoding can be included and speeds up loading
  • Code stays maintainable
  • The extra tool could use libraries which aren't lgpl. So maybe more archive formats can be supported.

And a random rant:
ZIP Format is a hell of a legacy format - optimized for multiple disk archives^^ (https://users.cs.jmu.edu/buchhofp/forensics/formats/pkzip.html)

@Ghabry
Copy link
Member

Ghabry commented Aug 7, 2017

@fdelapena fdelapena added the UX For issues affecting the user experience, such annoyances, counter-intuitive or ugly design label Dec 11, 2017
@KevinX8
Copy link

KevinX8 commented May 21, 2018

Bumpity bump, this is essential for the switch version now that it has been discovered that the switch has poor Japanese file support (ironically)

@Ghabry Ghabry removed this from the 0.7.0 milestone May 21, 2018
@Ghabry Ghabry added this to the 0.6.0 milestone May 21, 2018
@Ghabry
Copy link
Member

Ghabry commented May 21, 2018

This is actually work in progress since a while, we just don't keep the issues up-to-date because we discuss most of this stuff in our IRC channel but I won't give any estimates.

Here is a demo video: https://track8.mixtape.moe/ggotjx.webm
Though such videos are never showing the complete truth, the Player still crashes for some configurations and we must ensure that it works on all platforms we target (9).

Also saving doesn't work at all and fixing this is out of scope for this issue. Because the virtual filesystem by its own is already very complex (and we have a serious lack of developers) we decided to exclude savegame support and solve it afterwards as part of #666 because portable "Create Directory" and "Get Config/Save Directory" code is another huge can of worms...

@Ghabry Ghabry modified the milestones: 0.6.x, 0.6.0 (likely) Dec 10, 2018
@Ghabry
Copy link
Member

Ghabry commented Jan 7, 2019

Can we agree on that this issue counts as resolved when the filesystem-branch (VFS) is merged which gives ZIP support?
For any additional formats I would prefer new issues because this is already highly complex.

@Ghabry Ghabry modified the milestones: 0.6.0, 0.6.x Feb 12, 2019
@Ghabry
Copy link
Member

Ghabry commented Feb 12, 2019

and again the VFS-branch is moved to one release later. Sorry, it is by now ~80% finished :).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancement FileFinder Patch available UX For issues affecting the user experience, such annoyances, counter-intuitive or ugly design
Development

Successfully merging a pull request may close this issue.

5 participants