Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Codechange: refactor File I/O slot functions to be object oriented and pass references instead of indices #9039

Merged
merged 7 commits into from May 8, 2021

Conversation

@rubidium42
Copy link
Contributor

@rubidium42 rubidium42 commented Apr 14, 2021

Motivation / Problem

Technically the idea comes from #8870, although the bug part of it has been solved. Looking into it, it became clear to me that the code could definitely use some help. Especially when there are global bool arrays that use the number of a slot a file was loaded in, which makes it look like something that was added on later. Similarly the GRF Container version is passed around in many places, even though it is something that is part of the file so if you have a reference to that which add a second parameter?

Functionally there should be no change at all, except that the number of fopens/fcloses should be smaller now as the caching checks for the same file already being opened and reusing that instead of the old logic that closed that file and then reopened it.

Description

A RandomAccessFile class is created with the functionality of the old Fio functions, i.e. read a byte, word and dword, an seeking around in the file with some caching. Contrary to the Fio functions this caches 512 bytes per file, instead switching the "used" file and re-reading those 512 bytes the next time something is read.
This class is used for Sound and Music.

A SpriteFile subclass of RandomAccessFile is created which contains the palette_remap state and the GRF container version. The former is passed in the constructor, and the latter is determined in the constructor.
This class is used for sprites (gfxinit, newgrf, spriteloader and sprite cache). The sprite cache, since it already has references to the files it has to to load the sprites from, has been given a vector with the SpriteFiles used for the current game. Upon reinitializing the sprite cache these files are also closed. For the security/static scan of NewGRFs a temporary instance of SpriteFile is created which releases everything as soon as feasible.

Limitations

First and foremost, it is not finished yet. At the moment I am primarily looking for suggestions/comments regarding the structure of the classes and whether things should be in a different location, and whether we would rather use references, pointer or something like unique_ptr to pass the RandomAccess/SpriteFiles around.
As it is not finished, I have not checked for missing documentation myself, and some commits should be done in a different order and some should be squashed with each other. That will be done once I know the more generic preferences.

Likewise I have done some preliminary testing with some NewGRFs and that seems to work, however I am fairly certain there are many code paths that I have not tested yet.

Furthermore there is a ByteReader in newgrf.cpp that effectively reads a block of data from the SpriteFile and then provides similar functionality, and in the functions where it is used we often have to access the SpriteFile via _cur.file. I think a light weight variant of the ByteReader can be made on top of the SpriteFile which could reduce some of the unneeded copying of data, and might mean we can get rid of _cur.file. However, this is a big undertaking and will be considered future work and not part of this pull request.

Checklist for review

Some things are not automated, and forgotten often. This list is a reminder for the reviewers.

  • The bug fix is important enough to be backported? (label: 'backport requested')
  • This PR affects the save game format? (label 'savegame upgrade')
  • This PR affects the GS/AI API? (label 'needs review: Script API')
    • ai_changelog.hpp, gs_changelog.hpp need updating.
    • The compatibility wrappers (compat_*.nut) need updating.
  • This PR affects the NewGRF API? (label 'needs review: NewGRF')
@spnda
Copy link
Contributor

@spnda spnda commented Apr 14, 2021

Would it maybe be a good idea, as it is a rewrite anyway, that one could use std::filesystem more often?

@michicc
Copy link
Member

@michicc michicc commented Apr 14, 2021

Would it maybe be a good idea, as it is a rewrite anyway, that one could use std::filesystem more often?

Support for std::filesystem is still somewhat spotty unless you assume the absolute latest lib and compiler versions (e.g. macOS Xcode 11.something runs on 10.15+ only).

@rubidium42
Copy link
Contributor Author

@rubidium42 rubidium42 commented Apr 16, 2021

Would it maybe be a good idea, as it is a rewrite anyway, that one could use std::filesystem more often?

In theory yes; using C++ file IO would arguably be better, however it is not a full rewrite but only a rewrite of a small sub part of the whole file I/O, so using C++ file IO does not seem feasible without reimplementing the base this depends on. But then this patch's size would spiral out of control. Here I just wanted to accomplish that the "slotted" file I/O got removed with something more maintainable.

@rubidium42 rubidium42 marked this pull request as ready for review Apr 21, 2021
Copy link
Member

@LordAro LordAro left a comment

I think it'd be nice if it were consistent between references and pointers, though i think the vector of unique_ptrs is the way to go for the actual owning storage.
Right now though there's a weird mix of references and pointers, and it's not clear why...

src/random_access_file.cpp Outdated Show resolved Hide resolved
src/random_access_file_type.h Outdated Show resolved Hide resolved
src/spritecache.cpp Show resolved Hide resolved
* @param palette_remap Whether a palette remap needs to be performed for this file.
* @return The reference to the SpriteCache.
*/
SpriteFile &OpenCachedSpriteFile(const std::string &filename, Subdirectory subdir, bool palette_remap)
Copy link
Member

@LordAro LordAro Apr 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is (almost) the only function that doesn't return a pointer. Any particular reason why?

Copy link
Contributor Author

@rubidium42 rubidium42 Apr 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My reasoning has been: if it can be nullptr, then make it a pointer. If it cannot be nullptr make it reference. Though that is my C++03? brain talking; I have no real clue about the nuances of all the new pointer types.

src/random_access_file_type.h Show resolved Hide resolved
src/sound_type.h Show resolved Hide resolved
src/sound.cpp Outdated Show resolved Hide resolved
src/sound.cpp Outdated Show resolved Hide resolved
@rubidium42 rubidium42 force-pushed the fio_rewrite branch 2 times, most recently from 90eba82 to 473b8d0 Apr 23, 2021
src/sound.cpp Outdated Show resolved Hide resolved
@rubidium42 rubidium42 changed the title Fio rewrite Codechange: refactor File I/O slot functions to be object oriented and pass references instead of indices Apr 27, 2021
LordAro
LordAro approved these changes May 8, 2021
Copy link
Member

@LordAro LordAro left a comment

Good enough!

@rubidium42 rubidium42 merged commit fa6abe1 into OpenTTD:master May 8, 2021
12 checks passed
@rubidium42 rubidium42 deleted the fio_rewrite branch May 8, 2021
@PeterN
Copy link
Member

@PeterN PeterN commented May 8, 2021

Since fdc11a9 I get a message that I am missing 10 sprites in the baseset (using original_windows)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

6 participants