Compressed Fast Download in GoldSrc
I am not responsible for any damages that may occur while attempting to follow the suggestions made in this tutorial. USE IT AT YOUR OWN RISK!
Setting up a fastdl for custom content in GoldSrc Engine or Source Engine servers is common these days, uploading files to a webhost and setting url path in server.cfg sv_downloadurl makes players download files faster than fetching from gameserver itself!
For Source Engine based servers, we have bzip2 compression support which decrease up to 50% the size of a file and speed up even more the download process! For GoldSrc, being a 1998 game engine, unfortunately don't have such feature, but there is a way to do it!
In 2013, Valve has started to port GoldSrc Engine to Linux, updating various elements including the http engine used by GoldSrc and they were accepting feature requests as well.
I have opened a feature request (https://github.com/ValveSoftware/halflife/issues/922) requesting bzip2 support, but instead, Alfred replied:
The download client supports gzip content type, so just setup your download host to gzip files in your served folder.
With further researching, I found that GoldSrc is using Steam process to download files from fastdl enabled servers, and Steam uses Chromium Embedded Framework, the same web engine used by Google Chrome browser, supporting almost every browser features, including gzip content encoding as well.
We are going to use the same technique used to compress website pages, since CEF supports it and everything is being transparent to the final user. If you want to know how this works, Google is your friend.
This flowchart explains how GoldSrc download system currently works:
This setup is different for each webserver software, so we will focus on three major web servers for now:
Apache Web Server:
- Custom config support via .htaccess or direct editing webserver config files
- Rewrite module support (mod_rewrite)
- GZIP module support (mod_deflate)
- Headers module support (mod_headers)
NGINX (non reverse proxy mode):
- Direct editing webserver config files
- GZIP static module support (ngx_http_gzip_static_module)
- GZIP dynamic module support (ngx_http_gzip_module)
Microsoft IIS (Windows Server and/or Azure Web Services):
- Dynamic and/or Static compression features (httpCompression)
- URL Rewrite extension (URL Rewrite)
General:
- 7-zip or any file compressor (for pre-compressing files with gzip if you don't want to use on the fly compression)
All of the testing was done in the following systems:
Windows Server 2016 64-bit
Ubuntu 14.04 LTS 32-bit
Windows Server 2012 R2 64-bit
Windows Server 2016 64-bit
It's highly recommended to compress all the files in your fastdl. For this I made a tiny drag 'n drop batch file. Save this content as compress.bat with notepad or similar text editor.
@echo off
setlocal disabledelayedexpansion
rem R4to0's drag and drop compression script v2.1
rem Drag n drop your folders and files over this script
rem Requires 7-zip installed
rem If you want to use custom extensions, change this
set extension=gz
rem Make sure this is pointed to a 7zip binary
set compressor="C:\Program Files\7-Zip\7z.exe"
rem WARNING WARNING WARNING: THIS WILL DELETE UNCOMPRESSED FILES AFTER COMPRESSION.
rem "true" to delete, "false" to keep
set deleteorigin=false
rem Check if 7zip binary exists
if not exist %compressor% (
echo Error: compressor not found. Open this script with notepad and set 7zip path.
pause
exit
)
:start
rem Use single input dir/file at time
pushd %~pd1
rem If input is empty, go to end of file (ends script)
if [%1==[ goto :EOF
rem If input is a directory
if exist %1\* (
rem Enumerate files in every folder and compress
for /r %1 %%g in (*.*) do (
set file="%%g"
call :compression
)
) else (
rem if input is a file
set file=%1
call :compression
)
rem Compression process
:compression
if exist %file% (
%compressor% u -tgzip -mx=9 %file%.%extension% %file% -x!*%extension%
if exist %file%.%extension% (
if %deleteorigin% == true (
echo WARNING! DELETING ORIGIN FILE %file%
rem remove read only flag
attrib -R %file%
rem Delete file!!!
del %file%
)
) else (
echo Compression failed! File %file%
)
)
shift
popd
goto startSelect your files and drag them over your compress.bat file.
For Linux users, you can use the following bash script. You need to specify a folder in the -dir parameter where fastdl files are or create one called 'compress' within this script and copy files to that folder. Requires p7zip-full package!
#!/usr/bin/env bash
# R4to0's mass gzipper script
# GZip files recursively for a specified folder
init() {
# Parse arguments
while [ $# -gt 0 ]; do
case "$1" in
"-dir")
if [ -z "$2" ]; then
echo "Please specify a directory."
exit 1
fi
WORKDIR="$2"
shift ;;
"-sdel")
export SDELETE="-sdel"
shift ;;
"-threads")
if ! [[ $2 =~ ^[0-9]+$ ]] || [ $2 == 0 ]; then
echo "Error: -threads value is not specified or invalid." >&2; exit 1
fi
THREADS="$2"
shift ;;
"-help")
syntax
exit 0
;;
esac
shift
done
# Default folder if -dir is not specified
if [ -z "$WORKDIR" ]; then
echo "Warning: -dir not specified, defaulting to \"compress\""
WORKDIR="compress"
fi
if [ -z "$THREADS" ]; then
THREADS=$(getconf _NPROCESSORS_ONLN)
fi
if [ ! -d "$WORKDIR" ]; then
echo "The specified folder ($WORKDIR) does not exist!"
exit 1
fi
if [ -z $(command -v 7z) ]; then
echo "ERROR: 7-zip not found. Please install p7zip-full package."
exit 1
fi
if [ -n "$SDELETE" ]; then
echo "WARNING: SOURCE FILE WILL BE DELETED AFTER PROCESSING!"
fi
processfiles $WORKDIR
}
processfiles() {
echo "Working with $THREADS threads."
find $1 -type f ! -name '*.gz' -print0 | xargs -I % -0 -P $THREADS --process-slot-var=index bash -c 'echo "Processing % in thread $index"; 7z a -tgzip -mfb=256 -mx=9 "%".gz "%" "$SDELETE" >/dev/null 2>&1'
}
syntax() {
helpmsg=(
"Syntax:"
"$0 [-dir <dir>] [-threads <number>] [-sdel]"
""
"Params:"
"-dir <dir> Specifies the directory to be compressed. [Default: compress]"
"-sdel Deletes source files after processing (v9.30 and up). [Default: disabled]"
"-threads How many instances to run. [Default: $(getconf _NPROCESSORS_ONLN)]"
)
for dsphelp in "${helpmsg[@]}"
do
echo "$dsphelp"
done
}
init "$@"
Usage:
Syntax:
./script.sh [-dir <dir>] [-threads <number>] [-sdel]
Params:
-dir <dir> Specifies the directory to be compressed. [Default: compress]
-sdel Deletes source files after processing (v9.30 and up). [Default: disabled]
-threads How many instances to run. [Default: 2]
If compressed files doesn't exist, the webserver will send the non-compressed file instead, unless if set it up to compress on the fly (not recommended).
Add the following to your .htaccess file, inside of your fastdl folder:
### START GOLDSRC COMPRESSION SETUP ###
# Add this to your .htaccess or your .conf webserver files
# If pre-compressed files does not exist,
# apache will fallback to standard file extensions!
<IfModule mod_rewrite.c>
# Enable rewrite engine
RewriteEngine on
# Make sure the browser supports gzip encoding before we send it
# without it, Content-Type will be "application/x-gzip"
# We use original gz extension, feel free to change if you want
# just don't forger to change in FilesMatch and ForceType directives
# as well renaming all files.
RewriteCond %{HTTP:Accept-Encoding} \b(x-)?gzip\b
RewriteCond %{REQUEST_FILENAME}.gz -s
RewriteRule ^(.+) $1.gz [L]
</IfModule>
# Do stuff if requested file extension is .gz
# Change it if you modified in RewriteCond
<FilesMatch .gz$>
# Set gz mimetype. This is needed to avoid problems
# with certain files such as txt files
ForceType application/octet-stream
<IfModule mod_headers.c>
# REQUIRED! Send gzip response.
# In other words: tell to the client that file is gzipped
Header set Content-Encoding gzip
</IfModule>
</FilesMatch>
### END GOLDSRC COMPRESSION SETUP ###
If you use NGINX as your webserver software add the following to your server {} block for your fastdl location directive:
location /path/to/your/fastdl/ {
gzip_static on; # Enable static GZIP support
}
If you use Microsoft's Internet Information Services Webserver, add the following to your web.config inside of your fastdl folder
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<system.webServer>
<rewrite>
<rules>
<!-- Main rule to 'redirect' ALL requests to gzip version -->
<rule name="Set GZIP rule" enabled="true" stopProcessing="true">
<match url="\.*" ignoreCase="true" />
<conditions logicalGrouping="MatchAll">
<add input="{HTTP_ACCEPT_ENCODING}" pattern="gzip" ignoreCase="true" /> <!-- If client accepts gzip encoding -->
<add input="{REQUEST_FILENAME}.gz" matchType="IsFile" negate="false"/> <!-- If .gz file exists -->
</conditions>
<action type="Rewrite" url="{URL}.gz" /> <!-- Redirect to .gz file -->
</rule>
</rules>
<outboundRules>
<!-- IMPORTANT! Set the switches for the response encoding -->
<rule name="Rewrite GZIP header" preCondition="IsGZ" stopProcessing="false">
<match serverVariable="RESPONSE_Content_Encoding" pattern=".*" />
<action type="Rewrite" value="gzip" />
</rule>
<!-- Set the matches for the response encoding switches -->
<preConditions>
<preCondition name="IsGZ">
<add input="{PATH_INFO}" pattern="\.gz$" /> <!-- If request match a .gz file, send gzip response header -->
</preCondition>
</preConditions>
</outboundRules>
</rewrite>
<staticContent>
<!-- Set .gz mime type -->
<remove fileExtension=".gz" />
<mimeMap fileExtension=".gz" mimeType="application/octet-stream" />
</staticContent>
<!-- Enable static compression, disable dynamic compressison //-->
<urlCompression doStaticCompression="true" doDynamicCompression="false" />
</system.webServer>
</configuration>
As an alternative, you can configure your webserver to dynamically compress files on every request. I really recommend you to use the above solution with pre-compressed files, as the on-the-fly compression is less efficient and costs a lot of CPU since it does on every request for every user (except on IIS, explained below) unless if you host both gameserver and FastDL on same place with symlinked dirs (like I do with IIS). I left this part just for educational propurses only.
Remember: Google is your friend (or not).
Warning: Doing that on a restricted environment (such as free webhosts or shared plans) will lead you to get banned from your provider since it will cost more CPU time. You have been warned!!!
Add the following to your .htaccess file, inside of your fastdl folder:
<IfModule mod_deflate.c>
AddOutputFilterByType DEFLATE */*
SetOutputFilter DEFLATE
</IfModule>
(Looks a pretty simple config now, right? But you trade off your precious CPU time...)
For NGINX is a similiar way like the pre-compressed one, except that we use a different directive for setting it up.
location /path/to/your/fastdl/ {
gzip on; # Enable dynamic GZIP support
}
I never had any single major issues with other webservers like I had (and raged a lot) with IIS, got really mad with the bad experiences with Microsoft's webserver leading to a full reinstallation of the entire IIS system. When it works, it works. If won't work, then you're doomed! The only good thing is that the IIS Static Module does caching for static files that are dynamically compressed on requests, so the compression process is only done once until the origin file is modified. You can do in two ways: using IIS manager or with web.config file.
Go to your IIS manager and make sure the Enable static content compression is enabled. Optionally you can tweak or disable the other options here.
Now, go to Configuration Editor, then in Section dropdown box select system.webServer/httpCompression
Then, click on the three dots button in statictype
In this window, you have to add the mime type associated with your files. Since there is a lot of them in GoldSrc, you can just set the */* to true if you want. This option will enable compression globally for the whole IIS webserver.
If you want to go the classic way, or if you use Azure Services, then save the following to your web.config inside of your FastDL folder.
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<system.webServer>
<httpCompression directory="%SystemDrive%\inetpub\temp\IIS Temporary Compressed Files" doDiskSpaceLimiting="false" minFileSizeForComp="0" staticCompressionIgnoreHitFrequency="true">
<scheme name="gzip" dll="%Windir%\system32\inetsrv\gzip.dll" />
<staticTypes>
<add mimeType="*/*" enabled="true" />
</staticTypes>
</httpCompression>
<urlCompression doStaticCompression="true" />
</system.webServer>
</configuration>
- Work in Progress