Virtual Lua filesystem is very slow on client

[DBotThePony]

Continuation from Garry's Mod discord channel #next-update

Since i brought this ancient issue with Virtual Lua filesystem performance on client, i will open an issue ticket here.

Basically, Virtual Lua filesystem is very slow on client when joining non-local servers (probably also singleplayer?).
Since include/CompileFIle/file.Find(..., 'LUA') are handled by Virtual LuaFS, it is safe to assume on client performance of these functions is mostly affected by the FS performance. So here is the most simple test:

local systime = SysTime()
local filelist = file.Find('autorun/*.lua', 'LUA')
print((SysTime() - systime) * 1000)
print(#filelist)

local systime = SysTime()
CompileFile('autorun/!sh_dlib.lua')
print((SysTime() - systime) * 1000)

Outputs:


Serverside and

Clientside

These results get drastically worse the more folders are present in addons/ folder serverside. At point of 30 folders inside addons/ serverside Starting Lua stage takes forever for client. (those were measured with only 2 folders inside addons/ serverside. One with quickly-being-developed addon on git, second is basically a folder containing only symbolic links to all files inside folders that could be in addons/, but they are not due to this slow performance)

I suspect that either file.Find inside LUA also ask physical filesystem (even so, it shouldn't be this slow because of system cache!) or Virtual LuaFS is slow on itself, or both.

As @Kefta pointed out, it might be related that each addon in addons/ get it's own virtual filesystem on client. If so, then it is safe to assume that if Virtual LuaFS is marginally slower than physical filesystem then separate instances of them make situation only worse.

From my Discord post:

I'm saying server take ~15 seconds to start lua, client take >80 in such situation

And this have nothing to do with asset precache or Lua compilation time, because client
still load slow as fk when server changed level (and server in this case load as little as
6 seconds, which is insanely fast for me, considering there is more than 20 mb of Lua code to load)

Parsing and compiling 16 mb of Lua files take ~200 ms on my gmod client

As for me, approach of LuaPack is somewhat right, but instead of sending client a single file,
send only needed (as it is done now). And on client, each server get it's own single Lua file bank, instead of per
file basis (because that's slow on Windows, probably not so on Linux. Not to mention that
antimalware scanners (including Windows defender on Windows 10) get mad about gmod reading
such amount of small files in quick succession, so we lose a lot of time on that aside from slow LuaFS).
And also create one virtual file system with all Lua files in it, since i'm pretty sure it is almost useless 
for client to be aware which folders inside addons/ are present on serverside, since we will get only
first-match Lua files, and not every Lua with same name inside addons/

And don't read/check physical file system on client when doing include/CompileFile/file.Find(..., 'LUA')

And files in send.txt just need to be read from disk onto virtual file system

So we will minimize any possible overhead

And the said overhead is insane: some addons which scan virtual Lua filesystem (using file.Find) with
30 folders in addons/ serverside took up to 17 seconds doing so. But when everything was combined into
one folder serverside, scan time reduced to little 1.6 seconds (this is a real example, measured on client
connected to dedicated server)

[thegrb93]

If you're running x64 client and x86 server, that might be why. x64 has notably lower performance.

[DBotThePony]

Both server and client are on amd64 branch

[WinterPhoenix]

If you're running x64 client and x86 server, that might be why. x64 has notably lower performance.

Do you have a source or some data on this?

[thegrb93]

@WinterPhoenix one example #4100 . You'll also see starfall and E2 use more cpu on x64 servers, there's probably more examples but haven't found them.

[DBotThePony]

After more measurements in best case scenario around 93% of loading time is spent on file.Find(..., 'Lua') calls + (CompileFile/include - CompileString) time (e.g. the file read time (from virtual lua fs), and not time spent on Lua parsing and compilation), other 7% is actual code parsing + execution.

And no, this is not real file system bound since gmod is on a ssd and system file cache is a thing.

[Grocel]

I am not sure, but is this related to: #5674?

So far I can tell it should be fixed by now.