Document filesystem behavior

[wtfbbqhax]

Document filesystem behavior

1. Load precedence of gpp folder vs base vs mods
2. Load precedence of pk3's, how they are sorted etc.
3. Load precedence on a platform specific basis
4. Load precedence of QVM's vs DLL's (especially with regards to Game Lua, Unit testing, CmdParser, Polymorphic VM interface #84)

[dGr8LookinSparky]

We touched a little bit on item 1. in the "new homepath layout" page ( https://github.com/GrangerHub/tremulous/wiki/New-homepath-layout ). Using that info, I'll start that section of the new file system precedence page. I would also include before that the currently anticipated behavior of the "Default files" and basemod.

[Chomenor]

The way I understand it so far, the basic precedence model will look something like this, in order of higher to lower precedence:

1. fs_game
2. basemod
3. core assets (e.g. hardcoded hashes, special cgame/ui dlls)
4. base directories
5. inactive mod directories

These sections probably won't matter much when connected to a pure server, because the order of the hash list will take precedence. If you want cgame/ui dlls to be usable on a pure server, there will need to be some special accommodation for that, since the current pure system is based on pk3 hashes and only supports pk3s.

In the case of connecting to an unpure server (or maybe running a mod that is identified as based on a pre-1.3 version?) the precedence of sections 3) and 4) can be adjusted in the engine to prioritize older version assets when necessary.

[dGr8LookinSparky]

I started to the new wiki page for the load precedence at this link: https://github.com/GrangerHub/tremulous/wiki/File-System-Load-Precedences . @wtfbbqhax & @Chomenor , feel free to expand and edit that page as you see fit.

[Chomenor]

One thing that is a bit confusing in that document is the use of terms like "loaded before/after" to describe precedence. It can be unclear if it refers to higher or lower precedence.

@dGr8LookinSparky Is the precedence you described in the wiki page different from my comment above, and if so, could you clarify the differences?

[dGr8LookinSparky]

@Chomenor , actually I might be a bit uncertain about which is which with higher/lower precedence, but I think for the more part my description matches the order you gave in your previous comment (there is some variation). But what I mean by "loaded before/after" is that files "loaded after" override files "loaded before".

[dGr8LookinSparky]

As a side note, the initial design I proposed in that wiki page is not written in stone, it is just my current thoughts, it can be changed for a better design.

[Chomenor]

Are there any specific issues with the order I described in my comment? You mentioned a potential "variation", could you clarify what this entails? This part of the precedence structure needs to be very well defined for implementation purposes.

[dGr8LookinSparky]

For clarification, does higher precedence files override lower precedence? I guess it makes sense to have inactive mods at the lowest precedence.

I think the one thing that should be different from your current recommendation @Chomenor , is that the "defaults" (i.e. what you call the "core assets", either term for it may be fine), its associated "game version", would have its relative precedence vary in relation to the "versioned base folders". (the "game versions" being 1.1, gpp, 1.3, and any future game version as required (there would be a game version associated with each versioned base folder doesn't have to be consecutive).

That way for a given "default file" set to a given "game version", it would overide any and all "default files" with a lower "game version" as well as any and all files in "base folders" with a lower game version. Any and all files in a base folder that has the same or higher game version as a given default file, would have a higher precedence than that default file (i.e. could override that file).

I'll find some time tomorrow to edit the wiki page more accordingly.

[Chomenor]

Yes, by my conventions higher precedence overrides lower precedence.

The reason I grouped the default/core paks for all versions ahead of the base directories in general is because it protects the integrity of the game from the arbitrary pk3s in the base directories. The way my filesystem is designed, assets that override the default paks are only supposed to go in basemod or an fs_game mod directory. The regular base directories can't override the default paks, which means you can freely add paks or let servers download to these locations without much risk of significant game breakage, because the default paks will always have priority.

If you allow some base directories to override the default paks, for example paks in trem_1.3 or gpp to override the original Tremulous 1.1 default paks, I believe there will be a higher risk of conflicts or breakage due to improperly located pk3s. Is there a technical reason why any content should requires this kind of precedence behavior, instead of being placed in a mod directory?

[Chomenor]

Just to add to this, my concern is that, for example, many maps would depend on the original Tremulous 1.1 pk3s. If any pk3 in trem_1.3 or gpp can override these, there is a lot of potential for breakage, so my thinking is that it would be better if paks that did this kind of override need to be part of a mod.

[dGr8LookinSparky]

Your design as described does sound like a better approach. With the exception of the default files, would the files listed for the fs_game that includes files that happen to be in the base directory (or any directory for that matter even in inactive mod directories) have precedence over all files not listed in the fs_game?

[Chomenor]

I'm not sure I understand the question, but files in the current fs_game would have precedence over any other files, even if the same file is in some other location or anything like that.

[dGr8LookinSparky]

So would the precedence of all files in that fs_game be based purely on the pk3 relative precedence (as if in the same location so to speak?) I think that default files that are listed in the fs_game should still have lower precedence than all other files in the fs_game list. That way it would be ensured that any other non-default file in the fs_game could always overide a default file.

[Chomenor]

In my implementation for q3, the files in the fs_game directory or sorted only by filename. The existence of a default pak mixed into a mod would be very unusual, but if it did happen I figured the safest route would be to follow the traditional filename policy. It could be a deliberate attempt by a mod author to prevent a lower precedence resource pk3 from corrupting something, for example.

[Chomenor]

Also if somebody is using fs_game to test a mod in development, and they put a default pak at a certain alphabetical position, using the alphabetical precedence is probably the desired behavior for that situation.

[dGr8LookinSparky]

@Chomenor, on the old file system at least, are the fs_game files entirely determined by sv_pakNames and sv_paks? It seems that for a few different servers I just tried on the server browser (Unitremia, GrangerLab, GrangerPub, GrangerClub, and a couple others), at least for sv_pure, they include the default pk3 files in sv_paks and sv_pakNames. I know that on our servers at least, we are keeping those in the base (and gpp folders respectively).

[Chomenor]

As far as I know, for a client connected to a pure server, everything is determined by sv_paks. All the client-side precedence stuff if for the purposes of a local game / unpure server.

[dGr8LookinSparky]

Ah, I see, in the case of an unpure server, how are the fs_game files communicated from that unpure server and designated as such?

[Chomenor]

An unpure server just send the the name of the fs_game directory, and the client loads whatever files are in it. If the server wants to communicate actual file hashes it needs to be pure.

[dGr8LookinSparky]

@jkent commented on slack "or semipure" :) .

So in the case of an unpure server, and there are multiple instance of files throughtout a client's homepath, with the same file name, would the client's file system search in the directories's for that file name in each directory based on that directory's precedence order from highest precedence to lowest precedence and load only the first instance of that file found?

[dGr8LookinSparky]

Oh wait, this is what happens when I comment on github at a time I should be asleep, lol. There wouldn't be any searching of any specific files in the case of unpure, right? But still, in the case of unpure, how does the file system handle multiple instances of files of the same file name (not necessarily the same file), occurring in different kinds of directories as per your described precedence order?

[dGr8LookinSparky]

Also with your file system, would non-default files located in the inactive mod directories only ever be loaded if specified by pure/semipure servers (never loaded for pure)?

[dGr8LookinSparky]

Another question I have is would downloads only be prompted for pure and semipure servers?

[Chomenor]

I'm not sure what you mean by multiple instances of the same file name - same pk3 file name, or same name of the file within the pk3?

In my filesystem, inactive mod directories are always loaded, as in indexed in memory. On a pure server, pk3s from inactive mod directories will be used if on the pure list, and not used if not on the pure list, same as any other pk3. On an unpure server they would be used as a last resort if a file can't be found anywhere else (this can be disabled by cvar).

The download and pure systems are generally completely separate, so I don't know any reason why the download prompt would be affected by whether a server is pure or not.

[dGr8LookinSparky]

In the case I was describing above I was referring to the same pk3 file name used by multiple pk3 files located throughout the homepath.

On an unpure server, when might the download prompt be used when the unpure server doesn't state any required files? Would we have the downloading attempted on an unpure server if the client is missing a pk3 file on the hardcoded default hash list? Would there be any other instances downloading might be attempted on an unpure server?

[Chomenor]

Then the "core assets" section can be moved as a subcategory of "precedence categories", along with the other existing subcategories.

[dGr8LookinSparky]

Or maybe it should be "Category Precedence"? How should "pk3 precedence" fit in the page? That is the precedence of pk3 files that are in the same category? Also is there going to be a precedence between the homepath and basepath? If there isn't for the most part, how do we handle the case of the two pk3 files with the same name, but different hashes, and in the same relative sub-directory location, but one in the homepath and one in the basepath?

[dGr8LookinSparky]

Maybe the first section should be "Category Precedence" that has the overall list, but each list item could include sub-list items for further precedence organization within that category (like "Engine Version" and "pk3 file")?

[dGr8LookinSparky]

Then there would be a section section called "Precedence Categories", that would include all of the categories as subsections. Maybe the first section should be called something like "Category Precedence Tree"?

[dGr8LookinSparky]

How should "QVMs vs Dynamic Libraries" category be treated on that page?

[Chomenor]

Yes, that is what I am thinking, to have the "Precedence Categories" that starts with the category list and then has subsections to elaborate on each category. As for whether the initial list should have its own title, I'm not sure about that.

Pk3 precedence in the same category is based on the pk3 file name, except for the core assets category.

In my filesystem currently, the "write directory", which is typically the homepath, has precedence when files (either pk3 files or files outside of a pk3) have the exact same name. That way when the game writes a file like a config it can expect to read back the same file without a file in the other directory permanently overriding it. For other purposes in general the homepath and basepath are treated as if they were the same directory.

[Chomenor]

Generally dynamic libraries would override qvms in the same mod directory, if dynamic library loading is enabled.

[dGr8LookinSparky]

Maybe we should have the "Precedence Categories" be a subsection to a "Precedence Types" section, and have other "Types" subsections be "Engine Version", "pk3", "QVM vs Dynamic Library", "Write Directory vs. basepath", and "Content Files"? Or maybe those should just each be sections outside of the "Categories" section?

[Chomenor]

I think the precedence categories should be its own section, and the primary one, since that represents the root criteria for how files are prioritized. The other parts seem to elaborate on different aspects of how files are prioritized within the precedence categories. Those could be in a separate section if they apply across multiple precedence categories, or in one of the precedence category subsections if they only apply to one category.

[dGr8LookinSparky]

I updated the wiki page again (https://github.com/GrangerHub/tremulous/wiki/File-System-Load-Precedences).

what is the precedence for files inside a pk3 vs files of the same name/path in the same mod folder outside of pk3 files (is there a good short term we could call those kind of files?)?

[dGr8LookinSparky]

How would pk3dir (uncompressed directories that are treated like pk3 files, advantageous for developing assets before packaging) be treated?

[Chomenor]

Currently files outside pk3s take precedence over files in pk3s. You could call them non-pk3 files. They are sometimes referred to as files "on disk" and in my filesystem "direct sourcetype" files.

Pk3dirs are treated the same as a pk3 with the same name (at least in theory they are supposed to be; there can be technicalities).

[dGr8LookinSparky]

I think in the case of a pk3dir having the same name as a pk3 in the same mod, the pk3dir should probably have the higher precedence (for easy overiding while developing), but other wise considered the same as a pk3 in the case of other pk3 files with different names in the same mod.

[Chomenor]

That makes sense; I will look into updating the implementation to make sure it works this way.

[Chomenor]

Should be updated now Chomenor@57d4706

[dGr8LookinSparky]

I updated the category precedence list further.

https://github.com/GrangerHub/tremulous/wiki/File-System-Load-Precedences

[Chomenor]

I think there are ways the precedence document could be changed to be more concise, but I don't know of a method to recommend specific changes on a wiki page. Would you mind if I made my own version of the precedence document on my forked repo, so we could compare and discuss any differences?

[dGr8LookinSparky]

Go for it @Chomenor :) , note: when you are creating the wiki page, be sure to have MediaWiki formatting selected instead of markdown.

[Chomenor]

Okay, here's my shot at a precedence document:

https://github.com/Chomenor/tremulous/wiki/Filesystem-Precedence

[wtfbbqhax]

@dGr8LookinSparky @Chomenor Great job with the documentation! I've read through the documents; please verify my understanding is correct. Please have a look at my following pseudo shell code code. Comments document my interpretation of the documents; I've also outlined example scenario's of possible basepath, homepath and basemod file hierarchies.

#
# BlowFish understanding of the core principles! v1.1
#
# 06/04/2018
#


if ($platform == "windows") 
    fs_homepath="%APPDATA%/Roaming/Tremulous/"
elif ($platform == "apple") 
    fs_apppath="/Applications/Tremulous.app/MacOS/gpp/" # Special for Apple. Points inside the .app bundle itself.
    fs_homepath="/Users/viroemer/Library/Application Support/Tremulous/"
elif ($platform == "linux") 
    fs_homepath="$HOME/.tremulous"
fi

fs_basepath="" # engine versioning?
basemod="$fs_homepath/basemod"

# CORE ASSETS
#
# UNDERSTOOD:
#
# * GUARENTEED to be loaded
# * MUST EXIST to run Tremulous.exe (What happens if a core asset is missing?)
# * IS a fixed set of pk3 checksums; the actual files are located in one of Basepath, Homepath or Basemod
#
# QUESTIONS:
# * CAN a pk3dir be a core asset?
# * CAN a DLL be a core asset?
#
core_asset_=(
    data-1.1.0.pk3
    map-atcs-1.1.0.pk3
    map-arachnid2-1.1.0.pk3
    map-atcs-1.1.0.pk3
    map-karith-1.1.0.pk3
    map-nexus6-1.1.0.pk3
    map-niveus-1.1.0.pk3
    map-transit-1.1.0.pk3
    map-tremor-1.1.0.pk3
    map-uncreation-1.1.0.pk3
  )

# BASEPATH 
# 
# PERMISSIONS:
# * CAN NOT be modified by a server (tight coupling to the installation)
# * CAN be modified by a Super User (root, admin etc..)
# * Content are CREATED by the Makefile
# 
# PRECEDENCE:
# * LOWER precedence than "core assets" 
# * LOWER precedence than basemod (WHEN APPLICABLE)
# 
# WHEN:
# * Loaded when connected to a pure server
# * Loaded when connected to a unpure server
#
# EXAMPLE:
# Same as Quake3 
# 
# SPECIAL NOTES:
# * Basepath WILL CONTAIN the CORE ASSETS (Normal Operation)
# * Some installations may contain "legacy" installations from one or all of 1.1.0, GPP/1.2 or any previous 1.3.0 release.
#
basepath_set=(
    $fs_basepath/base/data-1.1.0.pk3
    $fs_basepath/base/vms-1.1.0.pk3
    $fs_basepath/base/map-atcs-1.1.0.pk3
    $fs_basepath/base/map-arachnid2-1.1.0.pk3
    $fs_basepath/base/map-atcs-1.1.0.pk3
    $fs_basepath/base/map-karith-1.1.0.pk3
    $fs_basepath/base/map-nexus6-1.1.0.pk3
    $fs_basepath/base/map-niveus-1.1.0.pk3
    $fs_basepath/base/map-transit-1.1.0.pk3
    $fs_basepath/base/map-tremor-1.1.0.pk3
    $fs_basepath/base/map-uncreation-1.1.0.pk3
    $fs_basepath/gpp/data-gpp.pk3
    $fs_basepath/gpp/vms-gpp.pk3
    $fs_basepath/gpp/map-atcs-gpp.pk3
    $fs_basepath/gpp/map-arachnid2-gpp.pk3
    $fs_basepath/gpp/map-atcs-gpp.pk3
    $fs_basepath/gpp/map-karith-gpp.pk3
    $fs_basepath/gpp/map-nexus6-gpp.pk3
    $fs_basepath/gpp/map-niveus-gpp.pk3
    $fs_basepath/gpp/map-transit-gpp.pk3
    $fs_basepath/gpp/map-tremor-gpp.pk3
    $fs_basepath/gpp/map-uncreation-gpp.pk3
    $fs_basepath/base_1.3/data-v1.3.0-alpha.0.13+1-abcdeg.pk3
    $fs_basepath/base_1.3/data-v1.3.0-alpha.0.13+11-hijklm.pk3
    $fs_basepath/base_1.3/data-v1.3.0-alpha.0.13+12-nopqrs.pk3
    $fs_basepath/base_1.3/data-v1.3.0-alpha.0.13+12-nopqrs.pk3dir
    $fs_basepath/base_1.3/data-v1.3.0-alpha.0.13+3-tuvwxy.pk3dir
    $fs_basepath/base_1.3/vms-v1.3.0-alpha.0.13.pk3
    $fs_basepath/base_1.3/vms-v1.3.0-alpha.0.13.pk3
    # DLLs
    $fs_basepath/base_1.3/game.dll
    $fs_basepath/base_1.3/cgame.dll
    $fs_basepath/base_1.3/ui.dll
  )

# HOMEPATH
# 
# PERMISSIONS:
# * CAN be modified by users 
# * CAN be modified by servers
# 
# PRECEDENCE:
# * HIGHER precedence than basepath
# * LOWER precedence than core assets
# * LOWER precedence than basemod (WHEN APPLICABLE)
# 
# WHEN:
# * Loaded when connected to a pure server
# * Loaded when connected to a unpure server? 
#
# EXAMPLE:
# Same as Quake3 
#
# UNDERSTOOD: 
# * If an asset found in homepath matches a Core Asset, it will be loaded from
#   the homepath; override any prior matching Core Asset in Basepath and Basemod 
# 
# * WILL likely contain Basemod
# 
homepath_set=(
    # Mixed pk3/pk3dir
    $fs_homepath/base/data-1.1.0.pk3
    $fs_homepath/base/vms-1.1.0.pk3
    $fs_homepath/base/map-atcs-1.1.0.pk3
    $fs_homepath/base/map-arachnid2-1.1.0.pk3
    $fs_homepath/base/map-atcs-1.1.0.pk3
    $fs_homepath/base/map-karith-1.1.0.pk3
    $fs_homepath/base/map-nexus6-1.1.0.pk3
    $fs_homepath/base/map-niveus-1.1.0.pk3
    $fs_homepath/base/map-transit-1.1.0.pk3
    $fs_homepath/base/map-tremor-1.1.0.pk3
    $fs_homepath/base/map-uncreation-1.1.0.pk3
    $fs_homepath/gpp/data-gpp.pk3
    $fs_homepath/gpp/vms-gpp.pk3
    $fs_homepath/gpp/map-atcs-gpp.pk3
    $fs_homepath/gpp/map-arachnid2-gpp.pk3
    $fs_homepath/gpp/map-atcs-gpp.pk3
    $fs_homepath/gpp/map-karith-gpp.pk3
    $fs_homepath/gpp/map-nexus6-gpp.pk3
    $fs_homepath/gpp/map-niveus-gpp.pk3
    $fs_homepath/gpp/map-transit-gpp.pk3
    $fs_homepath/gpp/map-tremor-gpp.pk3
    $fs_homepath/gpp/map-uncreation-gpp.pk3
    $fs_homepath/base_1.3/data-v1.3.0-alpha.0.13+1-abcdeg.pk3
    $fs_homepath/base_1.3/data-v1.3.0-alpha.0.13+11-hijklm.pk3
    $fs_homepath/base_1.3/data-v1.3.0-alpha.0.13+12-nopqrs.pk3
    $fs_homepath/base_1.3/data-v1.3.0-alpha.0.13+12-nopqrs.pk3dir
    $fs_homepath/base_1.3/data-v1.3.0-alpha.0.13+3-tuvwxy.pk3dir
    $fs_homepath/base_1.3/vms-v1.3.0-alpha.0.13+11-hijklm.pk3
    $fs_homepath/base_1.3/vms-v1.3.0-alpha.0.13.pk3
    # DLLs
    $fs_homepath/base_1.3/game.dll
    $fs_homepath/base_1.3/cgame.dll
    $fs_homepath/base_1.3/ui.dll

    $fs_homepath/base_1.3.0-beta.1/data-v1.3.0-beta.0.13.pk3
    $fs_homepath/base_1.3.0-beta.1/vms-v1.3.0-beta.0.13.pk3
    # DLLs
    $fs_homepath/base_1.3.0-beta.1/game.dll
    $fs_homepath/base_1.3.0-beta.1/cgame.dll
    $fs_homepath/base_1.3.0-beta.1/ui.dll
  )

# BASEMOD 
# 
# PERMISSIONS:
# * Basemod CAN be modified by users.
# * Basemod CAN NOT be modified by server.
#
# WHEN:
# * Loaded when in Main Menu.
# * Loaded when in Download Menu.
# * Loaded when connected to UNPURE server.
#
# PRECEDENCE:
# * HIGHER precedence than homepath, basepath. 
# * LOWER precedence than "Core Asset" list.
# 
# ---
# EXAMPLE USAGE:
# * Custom Hud
# * Custom model skins
#
# QUESTION:
# * Can Core Assets be contained in this directory?
# * Can DLLs be loaded from Basemod?
#
# UNDERSTOOD:
# * MAY EXIST under homepath or basepath
#
basemod_set=(
    # Extra Quake3 model paks
    model-Bender.pk3
    model-Zorak.pk3

    # Custom Main Menu assets perhaps
    hud-Blowfish2018.pk3dir

    # QVMs pak?
    vms-1.3.0-alpha.0.13-1-specialhotfix.pk3

    # DLLs
    ui.dll # Allowed?
    cgame.dll # Allowed?
    game.dll # Allowed?
  )

#  Assume all duplicates are identical
# vim:ft=sh

** Updated based on feedback from @dGr8LookinSparky (06/04/2018) - v1.1 **

[Chomenor]

Currently fs_homepath, fs_apppath, and fs_basepath are determined by calls to Sys_DefaultHomePath, Sys_DefaultInstallPath, and Sys_DefaultAppPath functions.

Currently basemod, like all other mod-type directories, can be located in any source directory (homepath, basepath, etc.)

The default content is anything specially identified by some engine mechanism, like hardcoded hash or some special dll path. Pk3s identified by hash can be picked out of any mod-like directory (base, trem_1.3, basemod, etc.) in any source directory (basepath, homepath, etc.) This category could include optional content that should be prioritized if it exists but isn't required. If you want to make certain pk3s required that can be done by verifying they exist at startup and displaying an error/warning or initiating a download if they don't.

Anything can be considered a core asset/default content if you want to implement it that way. It could include dlls and maybe even pk3dirs, theoretically, but it probably doesn't make sense to set up pk3dirs in the default content category as they are typically meant for testing and should just go in basemod or a specific mod directory instead.

Currently the source directory (basemod and homepath) is not really prioritized on the same level as basemod or default content. A current mod in any source directory will override basemod in any source directory, basemod in any source directory will override default content in any source directory, and so on. In practical usage, basepath generally represents the read-only install location and homepath is the write location, but when it comes to how they are searched and prioritized by the filesystem they are treated basically the same. Homepath does take priority in the special case of the exact same named file in the exact same location.

Placing game dlls in basemod is allowed and would be a normal use case. Generally custom dlls will need to be in either basemod or an active mod to be loaded, as otherwise whatever dll/qvm is selected by the default content category will be used.

Hope that helps. I also updated the wiki page in response to your questions there.

[wtfbbqhax]

I've responded -> 6/4/2018 blowfish

[wtfbbqhax]

@Chomenor Thank you, I think this is a good approach. How long do you estimate? Do we have any impeding tasks to address first? Do you have any other concerns?

[dGr8LookinSparky]

For the versioned base folders, we may want to have semantic version for future (and maybe current) considerations (like base_X.X.X-alpha.X.X having lower precedence than base_X.X.X-beta.X.X having lower precedence than base_X.X.X). We wouldn't have a new versioned base folder for each engine version, but semantic versioning could come in handy allowing us to make intermediate use of alpha and beta for testing purposes where a new base version folder might be needed. For consistency in naming convention, we may also want to consider aliasing the gpp folder name to base_1.2.0-beta.0.0.

For missing core assets that are required on startup, I think we should have them auto downloaded on startup from dl.grangerhub.com.

While the core assets could be located anywhere the file system can read, I think it would be good practice to have them located in the appropriate version base folders by the installation and updater, having their location reflecting the hardcoded precedence behavior as best as it can.

I don't think it would make sense to have pk3dirs used as an option for the core assets, as the core assets are meant to be stable, and the main purpose of pk3dirs is for development. instead it would make more sense to use them in basemod (and anywhere else ofc).

[dGr8LookinSparky]

I was just thinking that having the cgame/ui dynamic libraries be only based on location could be problematic, as they could easily be replaced by cheat versions when playing on servers whose fs_game might be set to base_1.3.

I think the way we should handle this is to have the hashes of "stable" versions of the game/cgame/ui dynamic libraries that were built before building the engine hardcoded on the core assets hash list, and have the game/cgame/ui libraries that were built with the engine be automatically placed in the basemod directory (if a version of the game.cgame/ui dynamic libraries that were built and placed in the basemod is deemed stable, then those binaries could be distributed in a subsequent version of the engine that would have their hashes hardcoded).

If we are going to have dynamic libraries included in the core assets hash list, I would assume that the platform is going to have to be factored in, and have a hash for each supported platform included for each dynamic library.

[Chomenor]

@wtfbbqhax The precedence system as described in the document is already mostly implemented. The main area that needs work is in the details of the "default content" category. Currently it is set up using hardcoded hashes from your "v1.3.0-alpha.0.13" release, but there are issues to resolve when it comes to getting updated hashes from pk3s generated in your automated build/release system. Also if you are switching to using native DLLs instead of qvms for the default modules that will need additional design work, as those can't be in pk3s and don't receive the kind of pk3 hash used for the regular hardcoded hash lists.

@dGr8LookinSparky I think using semantic versioning for base directories might be overkill. In general new versions of content should just be added to one of the existing base directories, with a higher-versioned pk3 name to override old versions. It should be rare that you need to add an entirely new base directory. As I understand it the base directories would generally correspond to different protocol versions, and are meant to help prevent new version content from causing problems when running in compatibility mode for an older version, or running an actual older client that uses the shared homepath location. For testing purposes, you should be able to just use an fs_game mod or basemod, and set up the file hierarchy within that directory.

Perhaps for pk3s within the same mod directory semantic versioning could be useful. Personally I'm not sure it's worth the complexity, given it's irrelevant on pure servers and simpler naming schemes (like adding a padding zero to version numbers) can achieve the same results, but if it's a high priority for you it should be technically possible to implement.

I don't like the idea of using basemod for default dlls/other content in general, because the standard meaning is that it is for "modifications" and the user should be able to remove this folder to revert the game to a default state. However as a temporary kludge this could be a viable approach.

Right now there isn't even a system in place for hashing dlls, so that would need to be devised, and it's not necessarily trivial because you don't really want to hash a whole bunch of versioned dlls on every game startup because of the load time implications. I prefer the idea of having a single set of dlls as the "defaults" in a fixed path location that gets updated along with the main binary. I wouldn't worry very much about cheat versions as long as somebody can just replace the main binary with a cheat version as well. Simply getting a cheat dll to load probably isn't going to be a challenge for a potential cheater no matter what you do in the engine.

[dGr8LookinSparky]

Regarding potential cheat dynamic libraries, yes with the engine source available, it would not be a difficult task for someone to circumvent whatever measures we have in place in the code to build an engine that would be able to load cheat modules. However, the distribution of cheat modules could be mitigated to a significant degree if we don't allow them to function in our engine distributions out of the box. I'm assuming that a significantly large number of players won't be building from the source code and/or modifying it. Also it would be a risk to download and use binaries for an engine distribution that is already known to have a nefarious purpose (cheating). Sure it would be a risk just to download cheat dynamic libraries even if used in our distributed engine, but it might be perceived as a less of a risk as downloading an entire cheat client.

I was thinking that in addition modifications added manually by the user to their basemod, we could also use the basemod for assets and dynamic libraries for "unstable" updates, and have the built game/cgame/ui dynamic libraries placed there. The basemod is already location based, and for the most part acts like the core assets without the hardcoded hash requirements.

A possible variation to consider is if we want to keep the basemod directory strictly for use by the user, we could have another folder that acts like the basemod, but would have lower precedence than the basemod and higher precedence than the core assets, and this new folder would have the chief purpose of being used by the installation and updater for "unstable" versions.

[Chomenor]

Nobody with the skills to design a cheat in the first place would find building the engine from source more than a 5 minute obstacle, and an engine binary is no less safe than a dynamic library. It seems hard to justify going to any trouble over the possibility of loading cheat dlls because the default dlls are basically just an extension of the engine anyway and you can compile the whole engine plus dlls from source just as easily as the dlls only.

An additional directory alongside basemod is an option, but I think proper hash-based precedence would be better in the long run for pk3-based content. I think location-based precedence is okay for the special case of the main default game modules, because that is a fixed set of only 2 or 3 files that doesn't have the potential to pile up or have complex hierarchical relationships the way pk3s can.

I also don't like the idea of pushing potential bugs and extra directories onto the user just because of something that is really an issue with the build system.

[dGr8LookinSparky]

Hi @Chomenor :) , I just wanted to check in, I have been tied up lately, but I will have some time on Tuesday to review the precedence discussions and pages, and update the wiki page accordingly with a design we can further review.

[Chomenor]

Okay, sounds good.