Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

introduce dpk format #14

Closed
wants to merge 2 commits into from
Closed

introduce dpk format #14

wants to merge 2 commits into from

Conversation

illwieckz
Copy link
Member

@illwieckz illwieckz commented Mar 27, 2017

See this forum post for the reasons and explanations behind this change.
See this merge request for related changes in NetRadiant/q3map2.

What this PR does:

  • dpk/dpkdir format for versioned packages

  • pk3/pk3dir are for legacy unversioned packages

  • legacy pk3/pk3dir now uses internal version "" so

    • it's always smaller than a dpk/dpkdir version since dpk/dpkdir forbid this version
    • code can rely on that version to know if it's a legacy pak, and it now does
  • the engine now loads dpk as versioned ones and pk3 as legacy unversioned one
    when fs_legacypaks is set to on, previously the code was loading versioned pk3
    as unversioned one when fs_legacypaks was set to on

  • the engine now prefers the versioned pak over the unversioned one if both exist

  • this code fixes the autodownload mechanism for legacy pk3s
    for sure, it was not able to work before since the code was adding version string
    to legacy pk3s that do not have version string, now the code relies on that
    special internal "" version to compute legacy pk3 names

  • modify FS::MakePakeName() to include + extension

    • engine/qcommon/files.cpp uses it now instead of calling FS::MakePakName()
      plus adding extension on every FS::MakePakName() call
    • src/engine/server/sv_client.cpp uses it instead of duplicating Fs:MakePakName()
      plus adding extension
  • factorise pak name parsing
    FS:PakParseName is used to parse both legacy and non-legacy pak names

  • always look for pak with version, this way, if server host a legacy pak
    but the client owns both legacy and non-legacy paks sharing the same name,
    the client is ensured to load the legacy one.

  • do not compare pk3 checksum
    legacy paks can't have checksum, so do not check it

  • ensure dpk paks with bad version are not loaded when fs_legacypaks is enabled

  • do not read DEPS from legacy paks

  • enable pk3 downloading

Previously, the server was sending pak list to client that way:

[
name1_version_checksum,
name2,
n_a_m_e_3,
name4_version_checksum,
]

So, the client was not able to parse this names with dpk/pk3
specificities, and to know the file format.

Now, the server sends pak list to client that way:

[
name1_version_checksum.dpk,
name2.pk3,
n_a_m_e_3.pk3,
name4_version_checksum.dpk,
]

Extension is used by client to know how to parse these names (pk3 have
empty version and name can't contain checksum) and reconstruct final
file name in local file system. On local file system the package can
be a dpkdir or pk3dir, that information is not told in that list since
there is no need to tell it.

So, this PR not only introduces dpk format, it is also meant to fix autodownloading legacy pk3 that was broken before, and factorize some code.

@illwieckz
Copy link
Member Author

illwieckz commented Mar 27, 2017

The only thing I haven't tested yet is autodownloading.

Here I describe my test environment, on engine path:

pkg/map-antares_a04+1.dpk
pkg/map-antares_a04.dpk
pkg/map-chasm_b2.dpk
pkg/map-forlorn_a15.dpk
pkg/map-parpax_d05.dpk
pkg/map-perseus_b5a+1.dpk
pkg/map-plat23_b13+1.dpk
pkg/map-spacetracks_1.0+1.dpk
pkg/map-station15_1.0+1.dpk
pkg/map-station15_1.0+2.dpk
pkg/map-station15_1.0.dpk
pkg/map-thunder_b3+2.dpk
pkg/map-vega_b4.dpk
pkg/map-yocto_b2c+1.dpk
pkg/tex-all_2.0.dpk
pkg/tex-common_2.0.dpk
pkg/tex-ej01-clean_1.0.dpk
pkg/tex-ej01-common_1.0.dpk
pkg/tex-ej01-ice_1.0.dpk
pkg/tex-ex_1.0.dpk
pkg/tex-exm_1.0.dpk
pkg/tex-pk01_1.0.dpk
pkg/tex-pk02_1.0.dpk
pkg/tex-space_1.0.dpk
pkg/tex-tech_1.0.dpk
pkg/tex-trak5_1.0.dpk
pkg/tex-vega_b4.dpk
pkg/unvanquished_0.25.0.dpk
pkg/unvanquished_0.26.0.dpk
pkg/unvanquished_0.27.0.dpk
pkg/unvanquished_0.28.0.dpk
pkg/unvanquished_0.29.0.dpk
pkg/unvanquished_0.30.0.dpk
pkg/unvanquished_0.31.0.dpk
pkg/unvanquished_0.32.0.dpk
pkg/unvanquished_0.33.0.dpk
pkg/unvanquished_0.34.0.dpk
pkg/unvanquished_0.35.0.dpk
pkg/unvanquished_0.36.0.dpk
pkg/unvanquished_0.37.0.dpk
pkg/unvanquished_0.38.0.dpk
pkg/unvanquished_0.39.0.dpk
pkg/unvanquished_0.40.0.dpk
pkg/unvanquished_0.41.0.dpk
pkg/unvanquished_0.42.0.dpk
pkg/unvanquished_0.43.0.dpk
pkg/unvanquished_0.44.0.dpk
pkg/unvanquished_0.45.0.dpk
pkg/unvanquished_0.46.0.dpk
pkg/unvanquished_0.47.0.dpk
pkg/unvanquished_0.48.0.dpk
pkg/unvanquished_0.49.0.dpk
pkg/unvanquished_0.50.0.dpk

on homepath:

pkg/map-1984b4.pk3dir
pkg/map-flap.pk3
pkg/map-parpax.pk3
pkg/map-rsmse_test.dpkdir
pkg/map-thermal.pk3dir

While having fs_legacypaks set to on, I was able to load chasm map from map-chasm_b2.dpk, rsmse map from map-rsmse_test.dpkdir, flap from map-flap.pk3, and both 1984b4 map from map-1984b4.pk3dir and map-thermal.pk3dir.

Of course loading badly named pk3/bsp does not work, it's out of scope there, and it's not my own problem.

The map-parpax.pk3 in homepath was the map-vega_b4.dpk file renamed, and the engine successfully loads map-parpax_d05.dpk instead, loading the parpax map instead of the vega one.

@illwieckz
Copy link
Member Author

illwieckz commented Mar 27, 2017

Loading automatically legacy pak0.pk3 then loading a dpkdir map package that implicitly relies on it:

$ mkdir etmain
$ cp ~/.etlegacy/etmain/pak0.pk3 etmain/pak0.pk3
$ mkdir etmain/map-railgun_2.60b.dpkdir/
$ ./daemon -set fs_legacypaks on -pakpath etmain +devmap railgun

Works.

@illwieckz
Copy link
Member Author

illwieckz commented Mar 27, 2017

- obsolete -

@illwieckz
Copy link
Member Author

illwieckz commented Mar 27, 2017

This change also means a game like Xonotic on Daemon can switch to dpk versioned format for their official packages while allowing third-party legacy unversioned pk3 being loaded. It adds safety since now the engine sets priority for dpkover pk3.

@illwieckz
Copy link
Member Author

illwieckz commented Mar 27, 2017

Beware, do not merge this PR too fast, we must first ensure every asset contributor is able to switch to dpk vfs, so it's better to wait for myself uploading the assets repositories I'm working on (it's now a matter of days).

{
if ( version.compare("") != 0 ) {
// versioned dpk
return Str::Format("%s.%s", MakePakName(name, version, *checksum), "pk3");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should "versioned dpk" paks not have the suffix "dpk" and "legacy unversioned pk3" have the suffix "pk3"?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh yes, that's the only part I haven't fully tested yet since it's used in autodownload stuff (have to setup a server etc.), nice catch!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

@@ -2386,12 +2395,36 @@ bool ParsePakName(const char* begin, const char* end, std::string& name, std::st
return true;
}

std::string MakePakFileName(Str::StringRef name, Str::StringRef version, Util::optional<uint32_t> checksum)
{
if ( version.compare("") != 0 ) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For readability I would prefer:
if ( !version.empty() )

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

if (!UseLegacyPaks()) {
suffixLen = type == pakType_t::PAK_DIR ? strlen(PAK_DIR_EXT) : strlen(PAK_ZIP_EXT);
} else {
suffixLen = type == pakType_t::PAK_DIR ? strlen(LEGACY_PAK_DIR_EXT) : strlen(LEGACY_PAK_ZIP_EXT);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that part makes no sense at all and only works because strlen(pk3) == strlen(dpk), I'm fixing this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

@illwieckz
Copy link
Member Author

illwieckz commented Mar 28, 2017

This new code adds a LoadLegacyPak() function, so the code checks against the right extension length. It was not a real issue since the two known extensions had same length and these extensions were hardcoded, so the code would always have been running well, but the new code is cleaner.

My first code was likely to loads dpk without version when fs_legacypaks was set to on, the new code ensures this to not occurs.

This new code also forbid dpk zip version to start with a letter, this way people can't make the mistakes to name their map-castle_b5.dpk with a version number greater than map-castle_1.0.dpk, the right way to specify a alpha/beta version would be map-castle_0.5b.dpk for example, if you really want to write this information in version string. The dpkdir loading code allows for version to start with a letter, this way, you can ensure your source or testing dpkdir is always greater than your release dpk, using src or test version number for example.

@illwieckz
Copy link
Member Author

After @Viech remarks I modified the version check to only check on dpk. I agree it's not meant to change the debian sorting, it is meant to rely on it.

Copy link
Contributor

@DolceTriade DolceTriade left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did a first pass of the code


bool UseLegacyPaks()
{
return *fs_legacypaks;
}

// Pak zip and directory extensions
#define PAK_ZIP_EXT ".pk3"
#define PAK_DIR_EXT ".pk3dir/"
#define LEGACY_PAK_ZIP_EXT ".pk3"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should change these to variables. const char LEGACY_PACK_ZIP_EXT[] = ".pk3";

@@ -122,16 +122,18 @@ template<> struct SerializeTraits<FS::LoadedPakInfo> {

namespace FS {

static Cvar::Cvar<bool> fs_legacypaks("fs_legacypaks", "Don't use versioned pk3 files", Cvar::NONE, false);
static Cvar::Cvar<bool> fs_legacypaks("fs_legacypaks", "Loads unversioned legacy pk3 files too", Cvar::NONE, false);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also load pk3s, ignoring version.

@@ -1114,7 +1129,8 @@ static void InternalLoadPak(const PakInfo& pak, Util::optional<uint32_t> expecte
return;
}
realChecksum = crc32(*realChecksum, reinterpret_cast<const Bytef*>(&crc), sizeof(crc));
if (filename == PAK_DEPS_FILE) {
// can't reuse isLegacy in lambda
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can if you capture it in the capture stanza like so:

	zipFile.ForEachFile([&pak, &realChecksum, &pathPrefix, &hasDeps, &depsOffset, &isLegacy](Str::StringRef filename, offset_t offset, uint32_t crc) {

}

if (type == pakType_t::PAK_ZIP) {
if (!isdigit(version.at(0))) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shouldn't be required. Our version sorting algorithm accounts for letters in the version.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I modified that part like this:

        if (type == pakType_t::PAK_ZIP) {
+               if (version.at(0) > '9') {
+                       fsLogs.Warn("Pak version is very high: '%s'", fullPath);
+               }
                fsLogs.Verbose("Found pak: '%s'...", fullPath);

It enforces nothing, user is free to make mistakes, but he gets a warning saying a01 (for example) is a very high version number, something that he probably not expect from an alpha version number. Can be helpful.

std::string name, version;
Util::optional<uint32_t> checksum;

ParsePakName(filename.begin(), filename.end() - suffixLen, name, version, checksum) || (isPakDir && checksum);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like you don' use this, so why not delete this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice catch, removed

}
fsLogs.Verbose("Adding pak: '%s'...", fullPath);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This log seems redundant compared to the one in InternalLoadPak

Copy link
Member Author

@illwieckz illwieckz Apr 10, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They do not have the same purpose and are not printed for the same events. This one is printed when a pak is added to the known list of paks. It will be printed on all pak successfully found. The other one ("loading pak: ") is printed only when the pak is effectively loaded. The idea of this other log line is to allow user to check if his pak is found even if it's not loaded, using +set logs.logLevel.fs verbose or debug.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I see. Sounds good.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I modified these lines like this:

-		fsLogs.Verbose("Adding pak: '%s'...", fullPath);
+		fsLogs.Verbose("Found pak: '%s'...", fullPath);

This way the meaning of this log line looks more obvious.

{
if ( !version.empty() ) {
// versioned dpk
return Str::Format("%s.%s", MakePakName(name, version, *checksum), "dpk");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PAK_ZIP_EXT, not dpk

return Str::Format("%s.%s", MakePakName(name, version, *checksum), "dpk");
} else {
// legacy unversioned pk3
return Str::Format("%s.%s", MakePakName(name, version, *checksum), "pk3");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LEGACY_PAK_ZIP_EXT, not pk3

@illwieckz
Copy link
Member Author

Just a side note that comes to my mind, currently it's not possible to support some other legacy formats (like pk4) because the code does not store the extension internally. For example the http download code guesses the final path like that (pseudo-code):

if version.empty():
  get http://<path>/<name>_<version>.dpk
else:
  get http://<path>/<name>.pk3

Making Dæmon being able to handle more than one legacy file extension would require to change many more files and it's outside the scope of this PR.

@illwieckz
Copy link
Member Author

@DolceTriade I addressed all the issues you pointed out.

@Viech
Copy link
Member

Viech commented Apr 10, 2017

enforce dpk version to start with a digit, dpkdir is allowed to start with an alphanumeric character, this way a mapper can be sure his source or test dpkdir always have greater version string than any released dpk using special version string like "src" or "test"

As I already pointed out on the Unvanquished forums:

  • Please don't distinguish .pk3 and .pk3dir at all, despite for the obvious difference of one being a folder and the other being a packed folder. Apply the KISS principle.
  • Please don't enfore a subset of version strings used by debian, instead allow the full range so we can just point people at the debian docs and don't need to explain our little differences.

Mappers should have the full freedom to make errors when they have no idea what a version string is, while people who do know should not have the extra complications.

(On a sidenote, I feel a01, b05, r23 are very good official release version strings for our .dkp.)

@Viech
Copy link
Member

Viech commented Apr 10, 2017

Also, is this PR implying that you simply discard the version of pk3s? That doesn't sound like sufficiently good legacy support, then, since people may want to have multiple pk3s and load only the newest. Ideally pk3s would even use the old ioquake3 version order.

@illwieckz
Copy link
Member Author

is this PR implying that you simply discard the version of pk3s?

This PR fixes version sorting and dependency loading for non-legacy paks when legacy pak support is enabled.

Ideally pk3s would even use the old ioquake3 version order.

This PR tries to fix non-legacy paks, legacy pak support improvement is welcome but out of scope. By the way the changes introduced in this PR can be helpful for someone wanting to improve legacy pak support.

@illwieckz
Copy link
Member Author

illwieckz commented Apr 23, 2017

So, last commit fixed legacy pak (pk3) download, since pk3 can't have checksum, the fix was simply to only validate checksum for dpk packages.

So, current code was tested successfully with all these scenarii:

  • pk3 udp download
  • pk3 http download
  • dpk udp download
  • dpk http download
  • dpk download with DEPS
  • pk3 download with pk3 extrapaks
  • dpk download with DEPS with dpk extrapaks with DEPS
  • pk3 download with pk3 extrapaks with dpk extrapaks with DEPS
  • pk3dir loading
  • dpkdir loading

If local dpk has wrong checksum, client re-downloads it again and uses the one with right checksum.
If local pk3 is different, engine loads it notwithstanding (that's expected legacy behavior).

Copy link
Contributor

@DolceTriade DolceTriade left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Kangz Can you take a look?

if (isDemo || allowRemotePakDir.Get())
continue;
Com_Error(errorParm_t::ERR_DROP, "The server is configured to load game data from a directory which makes it incompatible with remote clients.");
if (Str::IsSuffix(FS::LEGACY_PAK_ZIP_EXT, x.data())) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move this code inside FS::ParsePakName since you use it twice.

Copy link
Member Author

@illwieckz illwieckz May 20, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well, in fact not, but I can be used twice. I have some ideas to improve things on that part (and perhaps reduce even more code).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For explanations, currently FS::ParsePakName is a function only used with non-legacy pak, that's why it does not make sense to move this legacy-pak related code inside FS::ParsePakName, but I'm looking for a way to use FS::ParsePakName for every pak.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I refactored that part, I was also able to remove the AddLegacyPak function I added before… 😃

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Smaller the footprint the better :D

Copy link
Contributor

@Kangz Kangz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall, added a couple comments in addition to @DolceTriade's

@@ -122,16 +122,18 @@ template<> struct SerializeTraits<FS::LoadedPakInfo> {

namespace FS {

static Cvar::Cvar<bool> fs_legacypaks("fs_legacypaks", "Don't use versioned pk3 files", Cvar::NONE, false);
static Cvar::Cvar<bool> fs_legacypaks("fs_legacypaks", "Loads unversioned legacy pk3 files too", Cvar::NONE, false);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Load

@@ -1114,7 +1131,8 @@ static void InternalLoadPak(const PakInfo& pak, Util::optional<uint32_t> expecte
return;
}
realChecksum = crc32(*realChecksum, reinterpret_cast<const Bytef*>(&crc), sizeof(crc));
if (filename == PAK_DEPS_FILE) {
// can't reuse isLegacy in lambda
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can by adding isLegacy to the lambda capture list like so: [&pak, ...., &depsOffset, isLegacy]

AddLegacyPak(pakType_t::PAK_ZIP, Path::Build(subPath, filename), basePath);
} else if (useLegacyPaks && Str::IsSuffix(LEGACY_PAK_DIR_EXT, filename)) {
AddLegacyPak(pakType_t::PAK_DIR, Path::Build(subPath, filename), basePath);
} else if (Str::IsSuffix("/", filename)) {
FindPaksInPath(basePath, Path::Build(subPath, filename));
Copy link
Contributor

@Kangz Kangz May 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here: https://github.com/DaemonEngine/Daemon/blob/master/src/common/Assert.h#L156 can you add a

#define UNREACHABLE() DAEMON_ASSERT_CALLSITE(false, , false, "Unreachable code hit.")

and add an else clause containing UNREACHABLE();

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, I thought about something like that but I was wondering how to implement it, thanks.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

end -= strlen(PAK_ZIP_EXT);
} else if (Str::IsSuffix(PAK_DIR_EXT, begin)) {
end -= strlen(PAK_DIR_EXT);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, add an UNREACHABLE in an else clause.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

} else {
fsLogs.Notice("Loading legacy pakdir '%s'...", pak.path.c_str());
}
}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Kangz do you think I have to add an else UNREACHABLE() there?
In fact everywhere else the code is just doing if PAK_DIR then else (and does not test for PAK_ZIP).
I can also modify the other occurrence with explicit test for PAK_ZIP when not PAK_DIR, plus an else UNREACHABLE. What do you think about it?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are two choices, either test all cases in else-ifs and add an else clause with UNREACHABLE, or instead of finishing with else if (some condition) {code ...} do else {ASSERT(some condition), code...}. This way if a new case arises, we'll get a warning and know that things should be fixed.

- dpk/dpkdir format for versioned packages

- pk3/pk3dir are for legacy unversioned packages

- legacy pk3/pk3dir now uses internal version "" so
  * it's always smaller than a dpk/dpkdir version since dpk/dpkdir forbid this version
  * code can rely on that version to know if it's a legacy pak, and it now does

- the engine now loads dpk as versioned ones and pk3 as legacy unversioned one
  when fs_legacypaks is set to on,  previously the code was loading versioned pk3
  as unversioned one when fs_legacypaks was set to on

- the engine now prefers the versioned pak over the unversioned one if both exist

- this code fixes the autodownload mechanism for legacy pk3s
  for sure, it was not able to work before since the code was adding version string
  to legacy pk3s that do not have version string, now the code relies on that
  special internal "" version to compute legacy pk3 names

- modify FS::MakePakeName() to include + extension
  * engine/qcommon/files.cpp uses it now instead of calling FS::MakePakName()
  plus adding extension on every FS::MakePakName() call
  * src/engine/server/sv_client.cpp uses it instead of duplicating Fs:MakePakName()
  plus adding extension

- factorise pak name parsing
  FS:PakParseName is used to parse both legacy and non-legacy pak names

- always look for pak with version, this way, if server host a legacy pak
  but the client owns both legacy and non-legacy paks sharing the same name,
  the client is ensured to load the legacy one.

- do not compare pk3 checksum
  legacy paks can't have checksum, so do not check it

- ensure dpk paks with bad version are not loaded when fs_legacypaks is enabled

- do not read DEPS from legacy paks

- enable pk3 downloading

Previously, the server was sending pak list to client that way:

[
	name1_version_checksum,
	name2,
	n_a_m_e_3,
	name4_version_checksum,
]

So, the client was not able to parse this names with dpk/pk3
specificities, and to know the file format.

Now, the server sends pak list to client that way:

[
	name1_version_checksum.dpk,
	name2.pk3,
	n_a_m_e_3.pk3,
	name4_version_checksum.dpk,
]

Extension is used by client to know how to parse these names (pk3 have
empty version and name can't contain checksum) and reconstruct final
file name in local file system. On local file system the package can
be a dpkdir or pk3dir, that information is not told in that list since
there is no need to tell it.
was a nice kangz's idea

extra fix: there was some places where zip format was guess
testing if it was not a dir, since we don't trust pak.type
the code now guess if it's a zip by testing if it's a zip
@illwieckz
Copy link
Member Author

@Kangz I also modified some existing lines that were testing if a pak was a zip one by checking if it wasn't a dir one. Since you recommended to assert when a pak is not a zip and not a dir, I assume it's not good to trust a pak to be a zip just because it's not a dir. So, these lines now check if a pak is a zip by checking if it's a zip (bonjour Lapalisse).

@illwieckz
Copy link
Member Author

I squashed all the commits related to dpk support (because some were adding code the later removed). I left the one with UNREACHABLE asserts unsquashed since it acts as a very good standalone implementation example.

I verified this code works with both udp and http download with both dpk and pk3 archives.
I was able to load maps from both dpkdir and pk3dir.

@Kangz
Copy link
Contributor

Kangz commented May 22, 2017

Thanks @illwieckz, LGTM

@illwieckz
Copy link
Member Author

So, since this PR is now approved, is there an agenda to merge this PR?

Note: once this PR is merged, git Dæmon client will only be able to connect properly do git Dæmon server.
Probably not a problem since there is not so many people hacking Unvanquished code, and all of them can handle that. Perhaps it's time to raise protocol number, so we can host testing servers without having to care about stable clients mistakenly connecting to.

@DolceTriade
Copy link
Contributor

Yes, I would strongly prefer this to be merged into another branch. As soon as we prepare our next release, we will merge this in including the new paks based on your asset repo.

@illwieckz
Copy link
Member Author

So I just merged it in for-0.51.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-VFS T-Improvement Improvement for an existing feature
Projects
Unvanquished
  
Done
Xonotic
  
Done
Development

Successfully merging this pull request may close these issues.

None yet

5 participants