Permalink
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
192 lines (127 sloc) 16.5 KB

This is an updated file handling system for the ioquake3 project. It is a near complete rewrite of files.c and related components, designed to enhance the performance, stability, and security of the game while maintaining compatibility with almost all existing maps, mods, and configurations.

Here are the key features:

  • Improved resource precedence system which significantly reduces conflicts between pk3s
  • Greatly improved load times, especially with lots of pk3s installed
  • Safer downloading options and other security improvements
  • Improved download handling, especially when downloading from poorly configured servers
  • Ability to use resources in mods that aren't loaded, such as missionpack maps and textures when team arena mode isn't active
  • New debugging commands to trace file/shader origins
  • Ability to customize the download and pure lists when hosting servers
  • Fixed server-side pure list overflow issues
  • Semi-pure server support
  • Various other fixes and improvements

Feel free to contact me if you have any suggestions, feedback, or bug reports!

Author: Noah Metzger chomenor@gmail.com

Compiling

Should be very similar to regular ioquake3.

  • For make builds, run "make NEW_FILESYSTEM=1".
  • If you are using one of the build scripts instead of calling make directly, add the line "NEW_FILESYSTEM=1" to "Makefile.local"
  • For visual studio, set "NEW_FILESYSTEM" in the preprocessor definitions for both the main game and renderer dlls.

There are some changes to the renderer dlls in the new filesystem. If you mix a renderer dll and main application that have different filesystem versions, the game should still run, but this isn't recommended or well tested.

ID Pak Precedence

In this project, the ID paks, pak0.pk3 - pak8.pk3, automatically take precedence over other paks in baseq3. This is one of the key features that makes this project more stable than the original filesystem, because it prevents pk3s that are only intended to add maps or models from breaking the core game assets.

To allow users to intentionally modify the game, a new directory called "basemod" is supported. Files in this directory will always override files in baseq3, including the ID paks. In general regular maps and models will work fine in baseq3, but things that work by overriding existing game assets, such as crosshairs, enhanced texture mods, and VMs, will need to be moved to basemod to take effect.

Mod Settings Option

Currently ioquake3 stores a separate copy of the game settings for every mod. Whenever the mod changes, either by a user action or connection to a server, the settings are reset using the q3config.cfg from the new mod directory. This can lead to unexpected effects for users, such as graphics settings going haywire and changing resolutions when connecting to a server. This can be especially frustrating given that many mods are mostly server side and don't need any custom settings anyway.

This project introduces a command line cvar called "fs_mod_settings", with the following effects:

  • fs_mod_settings 0: All settings are loaded and saved in the q3config.cfg in baseq3, regardless of the current mod. Every mod uses the same settings. This is closer to the original q3 behavior.
  • fs_mod_settings 1: Existing ioquake3 behavior, with separately stored settings for each mod.

I favor fs_mod_settings 0 as the default value, because it's usually easier to manage a few settings that need to be changed between mods than to have every setting change between mods. Mods that require custom settings will be fine if they use deconflicted cvar names to store their settings. It appears most mods work under this setting without significant issues.

Download Directory Options

There are two new settings that can be used to control automatic downloads.

  • fs_saveto_dlfolder: If enabled, this will cause incoming downloads to be saved in the "downloads" folder within the target mod directory. For example, "baseq3/somefile.pk3" will be rewritten to "baseq3/downloads/somefile.pk3". Pk3s directly in the mod directory will take precedence over those in the downloads folder.

  • fs_restrict_dlfolder: This setting blocks potentially less secure content from pk3s in the downloads folder. It has three modes:

    • setting 0: No restrictions.
    • setting 1: Config files and qvm files that do not match a list of known trusted mod hashes will be blocked.
    • setting 2: Config files and all qvm files will be blocked.

These settings can increase security, but may break compatibility with some servers unless you manually move files out of the downloads folder.

Source Directory Options

A new cvar is introduced called "fs_dirs", which can be set from the command line to adjust which source directories the game uses to load/save files. The default is "*homepath basepath steampath gogpath". This means that homepath is the write directory, indicated by the asterisk, and the other locations are used for reading. The specific paths are still controlled by the "fs_homepath", "fs_basepath", "fs_steampath", and "fs_gogpath" cvars, respectively.

Notes:

  • You can set an asterisk on multiple directories. The additional directories will be used as backup write directories if the first one fails a write test.
  • The write directory selected will always be treated as the primary read directory.
  • If no directory passes a write test, or no write directory was set at all, the game will run in read-only mode.

Examples:

  • "fs_dirs" = "*basepath": Read and write to fs_basepath only.
  • "fs_dirs" = "homepath basepath": Read-only mode with homepath taking precedence over basepath.

Inactive Mod Support

This project supports loading files from all mod directories, not just the current active mod. This is intended to help smooth out discrepancies between server directory configurations and allows maps to work correctly even if they have dependencies that, for one reason or another, are located in the wrong mod directory.

The precedence system ensures that files located in the current mod and basegame directories always have first priority. Inactive mods are only accessed as a last resort if files cannot be found anywhere else.

There are two cvars used to control this feature:

  • fs_search_inactive_mods: Applies to primary file lookups used to load models, textures, and sounds. (default: 2)
  • fs_list_inactive_mods: Applies to file listing queries used by the ingame map and model menus. (default: 1)

Both cvars support the following settings:

  • 0: Disabled; all files from inactive mods blocked
  • 1: Enabled only for ID paks (including missionpack) and files on the connected server's pure list
  • 2: Enabled; all files from inactive mods allowed

Cache Mechanisms

This project adds two types of caches to improve load times, an index cache and a memory cache.

The index cache stores pk3 index data in a file called fscache.dat to reduce the initial game startup time. Cached data is matched to pk3s by filename, size, and timestamp, so it shouldn't cause problems when pk3s are modified. If you suspect the cache could be causing a problem, you can delete the cache file or disable it by setting "fs_index_cache" to 0 on the command line.

The memory cache is used to keep previously accessed files in memory for faster access and reduce load times between levels. The size of this buffer is controlled by the "fs_read_cache_megs" cvar. The default is currently 64 for the client and 4 for the dedicated server. This value can be set to 0 to disable the cache altogether.

Download / Pure List Configuration

Two new cvars, "fs_download_manifest" and "fs_pure_manifest", are added to allow customizing which pk3s are added to the download and pure lists when running a server. Both cvars use a space-seperated list of selector rules that can be either specific pk3 names or one of the following keywords:

  • *mod_paks - Selects all paks from the current active mod.
  • *base_paks - Selects all paks from com_basegame (baseq3).
  • *inactivemod_paks - Selects all paks that don't fit the previous two categories
  • *currentmap_pak - Selects the pak containing the bsp of the current running map.
  • *cgame_pak - Selects the pak containing the preferred cgame.qvm file.
  • *ui_pak - Selects the pak containing the preferred ui.qvm file.
  • *referenced_paks - Selects paks accessed during the loading process on the server.

The default download manifest selects all the paks from the current mod directory, as well as the current cgame and ui paks, and the current map pak. The *referenced_paks rule is currently added for consistency with original filesystem behavior, but in virtually all cases is redundant to *currentmap_pak and *mod_paks and can be dropped without issue.

set fs_download_manifest *mod_paks *cgame_pak *ui_pak *currentmap_pak *referenced_paks

Some server configurations have a lot of maps or optional mod files in the mod directory. This can lead to clients having too many files to download, since the default behavior is to place everything in the mod directory into the download list. To avoid this, you can specify only the core mod files that clients require instead of using the *mod_paks rule. For example, here is a reduced download manifest for the OSP mod.

set fs_download_manifest osp/zz-osp-pak3 osp/zz-osp-pak2 osp/zz-osp-pak1 osp/zz-osp-pak0 *currentmap_pak

The default pure manifest simply selects every pak available to the game.

set fs_pure_manifest *mod_paks *base_paks *inactivemod_paks

Pure servers with a large number of pk3s in baseq3 (300+) can run into problems with the pure list overflowing. To avoid such issues you can replace the *base_baks rule in the pure manifest with specific core paks. Note that any auxiliary paks in baseq3 required by maps or mods, as well as optional content like player models, will need be included manually as well.

set fs_pure_manifest *mod_paks baseq3/pak8 baseq3/pak7 baseq3/pak6 baseq3/pak5 baseq3/pak4 baseq3/pak3 baseq3/pak2 baseq3/pak1 baseq3/pak0 *currentmap_pak

Advanced Features

It is possible to specify pk3s by hash, to support conditions where the file may not physically exist on the server or may exist under a different name. If the file does not physically exist it can't be used for UDP downloads, but it can be used for pure lists and download lists on HTTP-only servers (with sv_dlURL active and sv_allowDownload set to 0). To use this feature, specify paks using the format [mod]/[name]:[hash]. The hash can be specified in either signed or unsigned integer format.

set fs_pure_manifest *mod_paks *base_paks *inactivemod_paks baseq3/md3-bender:-722067772 baseq3/md3-laracroft:1134218139 baseq3/md3-spongebob:-871946717

The order of the pure list determines the precedence of files on clients. Normally the pure list is sorted according to filesystem precedence conventions, rather than the order in the pure manifest, but there may be special conditions where it is useful to force a certain order in the pure list. This can be accomplished by separating sections in the pure manifest with a dash. In this example baseq3/somefile.pk3 will be the first entry on the pure list and have the highest precedence, regardless of where it stands in the normal filesystem ordering. Note that if the same pk3 is selected by multiple rules, its position will be determined by the first rule that selected it.

set fs_pure_manifest baseq3/somefile - *mod_paks *base_paks *currentmap_pak

In some cases it can be convenient to exclude a certain pk3 that would otherwise be selected by one or more rules within a manifest. This can be accomplished by using the "&exclude" command followed by a normal selector rule. All pk3s selected by the rule will be blocked, based on hash, from being selected by any subsequent rules. In this example, the "mod/somemap.pk3" file can be selected by the *currentmap_pak rule, but not subsequent rules such as *mod_paks because they come after the exclude command.

set fs_download_manifest *currentmap_pak &exclude mod/somemap *mod_paks *cgame_pak *ui_pak *referenced_paks

Semi-Pure Server

This option causes clients to prioritize data from paks on the server like a normal pure server, but also allows loading content from other paks when it is not available from the pure list paks. For example, it lets clients use custom models that are not on the server. To enable, set sv_pure to 2 on the server.

This a client-side feature, so it only affects clients using the new filesystem. Clients using the original filesystem will function the same as if sv_pure is 1.

Shader Precedence

In this project, shader precedence follows the same rules as normal filesystem precedence, e.g. a shader in pak2.pk3 will override a shader in pak1.pk3, and a shader in a mod will override a shader in baseq3. However, the original filesystem uses the opposite precedence when shaders are defined in .shader files with different names, which can make things confusing for mod authors. Here are some tips for mod authors working with shader definitions:

  • To update shaders in an existing pak from your mod to a new version, copy the .shader file from the earlier pak to the new pak, keeping the same filename. Then modify the shader(s) in the new file. The original filesystem will only load the newer file, so make sure it contains all the shaders from the original file as well.

  • If you wish to override a shader from paks in baseq3, such as the ID paks, you need to copy the most recent version of every .shader file containing that shader into your mod, then modify the shader within those files. If you simply define the shader without following these steps, it will work on the new filesystem, but not the original filesystem.

  • Conversely, make sure you don't accidentally define unwanted shaders that conflict with the ID paks or earlier paks from your mod in arbitrary .shader files. These will be overridden in the original filesystem but will take effect on the new filesystem, which could lead to problems.

Debugging Cvars

This project introduces some new cvars that can be set to 1 to enable debug prints.

  • fs_debug_state: logs changes to the primary filesystem state such as the current mod dir and server pak list
  • fs_debug_refresh: logs warnings that come up during file indexing phase; best set from the command line and combined with fs_index_cache 0
  • fs_debug_fileio: logs file reading/writing activity
  • fs_debug_lookup: logs file index lookup activity
  • fs_debug_references: logs pk3 referencing activity (used for downloads and pure lists)
  • fs_debug_filelist: logs directory listing calls

Debugging Commands

The following commands can be used to get an idea of where resources will come from in the filesystem. They generate a list of resources sorted from highest to lowest precedence. The first resource is typically what will be used by the game.

  • find_file : General purpose file lookup. Example: find_file maps/q3dm11.bsp
  • find_shader <shader/image name>: Locates both shaders and images with this name. Example: find_shader textures/base_floor/techfloor
  • find_sound : Locate sound resources with this name. Example: find_sound sound/weapons/noammo
  • find_vm <vm/dll name>: Locate vm or dll resources with this name. Example: find_vm cgame

Once you run one of the above commands, you can use the "compare" command to find why one element in the list was chosen over the other. For example, run "compare 1 2" to check why the top element was selected over the second element. The list from one previous find command is stored in memory for the purpose of supporting this command.

Known Issues

This project is fairly well developed at this point, so there are few significant known issues. If you find any new issues, feel free to email me or open an issue on GitHub.

  • This is currently an engine only modification, and does not remove limits in the UI on the number of maps and models that can appear in the menus. The UI code will need to be changed in mods to raise these limits. You can still load maps and models through the console or by connecting to a server even if they don't appear in the menu.

  • The code style (e.g. capitalization and brace placement) used in this project are different than typical ioquake3 conventions. This mostly applies to code under the filesystem directory; code integrated with existing source files follows the conventions in those files more closely.

  • Limited testing has been done on less common platforms and build environments or on big endian systems. Testing help and feedback for such platforms is appreciated.

  • Only the makefile and msvc12 project files are updated to use the new filesystem. Some of the other project files and build scripts aren't updated yet. It appears most of these files are in an unmaintained state in ioquake3 anyway.