Server subdirectories instead of mass dumping resources

[yannickboy15]

Is your feature request related to a problem? Please describe.
I'm frustrated to see that multiple communities use each others scripts, but with different settings. Because of this you'll have to redownload all the time. Also when you need to clear your resource cache, you're clearing all the cache instead of a specific server cache.

Describe the solution you'd like
Instead of putting everything in MTA/mods/deathmatch/resoures/ [ALL RESOURCES], make it: MTA/mods/deathmatch/resources/serverip:port/ [ALL RESOURCES]

With this we can just clear server specific resources instead. It should be a matter of creating a sub directory with the serverip:port and then change the caching function to put it inside that folder.

Describe alternatives you've considered
None really, checking it manually is a pain in the ass.

Additional context
Nothing, couldn't write it myself as I don't understand the MTA logic. Codewise it shouldn't be hard atleast.

[PlatinMTA]

We can make the complete reverse argument and see this as a good thing, some servers will use the same resources with the same files, maybe one or two changed. Not having to download the same resource again is certanly a bonus, specially if you are the kind of guy to be jumping from low player/empty server to empty server.

The other problem I can think of is when a server changes it's IP or even it's port. Big communities also tend to use the same resources through various servers, making the client download two times the same thing is not worth it.

This problem increases exponentially when taking into account skins/mods resources, where the files itself could range from a couple megabytes into absurd levels like 1GB. Downloading a GB is painful enough, most people won't like to have it duplicated in their system.

I argue that, if this is somewhat added, it should be the clients decision to download the resources by server IP or just in a general folder.

Also I think the servers should already be aware to this. In my case I just use a simple "tag" in front of my resource folder and call it a day.

PS: If someone really wants to work on this, maybe it would be great to wait until "DNS support" is added? #1314

[AlexTMjugador]

An idea: maybe we can categorize the downloaded files by hashes, so the client won't ideally download the same files twice, and neither replace other versions of the file. Hash collisions are rare, but if they happen we could fall back to the current behavior of overwritting whatever there is.

OTOH, scripts can already create private files that are stored separately, per server and resource.

[yannickboy15]

An idea: maybe we can categorize the downloaded files by hashes, so the client won't ideally download the same files twice, and neither replace other versions of the file. Hash collisions are rare, but if they happen we could fall back to the current behavior of overwritting whatever there is.

OTOH, scripts can already create private files that are stored separately, per server and resource.

Would be possible, but how would you clear the cache of 1 specific server only without having them bulked out in the resources folder?

We can make the complete reverse argument and see this as a good thing, some servers will use the same resources with the same files, maybe one or two changed. Not having to download the same resource again is certanly a bonus, specially if you are the kind of guy to be jumping from low player/empty server to empty server.

The other problem I can think of is when a server changes it's IP or even it's port. Big communities also tend to use the same resources through various servers, making the client download two times the same thing is not worth it.

This problem increases exponentially when taking into account skins/mods resources, where the files itself could range from a couple megabytes into absurd levels like 1GB. Downloading a GB is painful enough, most people won't like to have it duplicated in their system.

I argue that, if this is somewhat added, it should be the clients decision to download the resources by server IP or just in a general folder.

Also I think the servers should already be aware to this. In my case I just use a simple "tag" in front of my resource folder and call it a day.

PS: If someone really wants to work on this, maybe it would be great to wait until "DNS support" is added? #1314

I see where you're coming from and I think it should indeed be an option for the user to select. Considering the extra data it'll download.

Thing could be too that if servers use the same resources and they store XML files locally with passwords (a remember me function for example) unencrypted, people their stuff could be stolen too. Yea, extreme situation but sadly this has happened before.

[PlatinMTA]

An idea: maybe we can categorize the downloaded files by hashes, so the client won't ideally download the same files twice, and neither replace other versions of the file. Hash collisions are rare, but if they happen we could fall back to the current behavior of overwritting whatever there is.

This is a great idea, but I'm kinda worried that #1340 could be made worse because of this. And my other concern, could checking the hash make the loading of a server go noticeable slowly? Specially on low end PCs.

Thing could be too that if servers use the same resources and they store XML files locally with passwords (a remember me function for example) unencrypted, people their stuff could be stolen too. Yea, extreme situation but sadly this has happened before.

That's just bad coding on part of the server, and bad password management on the clients end. You need to always use different passwords for different servers, specially if you don't trust them. If you are an scripter and want to fix this, use @ in the path of files that could have sensitivy data, more info on: https://wiki.multitheftauto.com/wiki/Filepath

And if you question your ability to make a secure loginpanel, see this: https://forum.mtasa.com/topic/119243-user-authentication-password-handling-and-doing-it-safely/ (don't store the password in an xml, store a token instead!!!)

[yannickboy15]

An idea: maybe we can categorize the downloaded files by hashes, so the client won't ideally download the same files twice, and neither replace other versions of the file. Hash collisions are rare, but if they happen we could fall back to the current behavior of overwritting whatever there is.

This is a great idea, but I'm kinda worried that #1340 could be made worse because of this. And my other concern, could checking the hash make the loading of a server go noticeable slowly? Specially on low end PCs.

I've noticed lately that above issue mentioned took an increase for me too, could instead use maybe another way of hashing to handle those mismatches and all. Would be a huge overhaul.

Thing could be too that if servers use the same resources and they store XML files locally with passwords (a remember me function for example) unencrypted, people their stuff could be stolen too. Yea, extreme situation but sadly this has happened before.

That's just bad coding on part of the server, and bad password management on the clients end. You need to always use different passwords for different servers, specially if you don't trust them. If you are an scripter and want to fix this, use @ in the path of files that could have sensitivy data, more info on: https://wiki.multitheftauto.com/wiki/Filepath

And if you question your ability to make a secure loginpanel, see this: https://forum.mtasa.com/topic/119243-user-authentication-password-handling-and-doing-it-safely/ (don't store the password in an xml, store a token instead!!!)

I understand that, but leaked gamemodes like the old OwlGaming gamemode we built (leak from 2016 or so) were by serveral new community owners changed into exposing passwords when logging in to gain access to the main server. Now we closed down, which is not a big thing anymore but just to point out what people have done in the past and could still do.

I do believe we should look first into issue 1340 before this is to be done, to see what the behavior will be.

[AlexTMjugador]

Would be possible, but how would you clear the cache of 1 specific server only without having them bulked out in the resources folder?

MTA could track which hashes belong to which resource on which server, and could offer a nice way to clean up the resources folder for you using that information. AFAIK, currently resources are just stashed in that folder without any regard to what server they belong to, so it's not like introducing these hashes would be a regression either.

This is a great idea, but I'm kinda worried that #1340 could be made worse because of this. And my other concern, could checking the hash make the loading of a server go noticeable slowly? Specially on low end PCs.

I also think that these concerns need some extra attention if the hashes idea is to be used. Of course we can use even cheaper checksums like CRC32, but that would come with the cost of increased collisions.

[yannickboy15]

Would be possible, but how would you clear the cache of 1 specific server only without having them bulked out in the resources folder?

MTA could track which hashes belong to which resource on which server, and could offer a nice way to clean up the resources folder for you using that information. AFAIK, currently resources are just stashed in that folder without any regard to what server they belong to, so it's not like introducing these hashes would be a regression either.

This would be indeed a good solution in that case. Possibly somewhere in the settings panel as an extra UI tab.

This is a great idea, but I'm kinda worried that #1340 could be made worse because of this. And my other concern, could checking the hash make the loading of a server go noticeable slowly? Specially on low end PCs.

I think that with a lightweight enough hashing algorithm this could succeed already without causing too much pain. Thinking indeed of CRC or Adler.

[Lpsd]

Generating a hash for each resource and storing which server(s) it's used on sounds reasonable - then we can offer management from within the MTA client.

As for the initial suggestion of splitting up the resources into folders per-server - while this fixes the issue of identifying which resources a server uses, it's not really compatible with better solutions (such as the hashing one above).

This most likely requires more work than it appears on the surface, I assume there's a ton of stuff in the net module to be considered?
cc @ccw808

[yannickboy15]

Generating a hash for each resource and storing which server(s) it's used on sounds reasonable - then we can offer management from within the MTA client.

As for the initial suggestion of splitting up the resources into folders per-server - while this fixes the issue of identifying which resources a server uses, it's not really compatible with better solutions (such as the hashing one above).

This most likely requires more work than it appears on the surface, I assume there's a ton of stuff in the net module to be considered?
cc @ccw808

Are we talking here about a .net module? If so, is this open source and can we contribute to this? I might be able to do the C# part of it, if that's the case.

[Lpsd]

I'm referring to MTA's net module, net(c).dll - part of which is closed source. Not to be confused with .NET/C#.

[ccw808]

I assume there's a ton of stuff in the net module to be considered?

I don't think so

Generating a hash for each resource and storing which server(s) it's used on sounds reasonable

Where would the client store this information?

[AlexTMjugador]

Where would the client store this information?

I think that a SQLite database would be a nice way to store this data, but XML files would also work, like it is already done to store keybinds and other settings.

[sbx320]

The relevant stuff for file downloads is inside the main source code, so I wouldn't worry about net. We also don't even need to calculate the file hashes, as those are already provided by the server whenever files are downloaded. We can just store them at that point. As a result the performance impact should be fairly small (one simple lookup per file).

I'd suggest using a simple SQLite DB (Table for files <hash, cache path, filename>, Table for servers <serverip, hash>) as parsing massive XML files will end up being fairly slow. Then whenever a file is required by aresource, the client would start off by checking the DB for a matching hash and if present, copy [0] the cached file into the local resources directory and validate the hash once more.

[0]: Ideally we'd use some virtual filesystem for this, which would allow us to map paths directly to the cached versions, rather than duplicating files all the time. But that might go too far for this simple request.

[ArranTuna]

This sounds like a load of hassle for probably nothing. So what servers is it exactly that you keep switching between and how much file download is there that you must re-download every join?

Really this is the problem of the server owners for not renaming files which they've edited, whenever I edit a community resource I'll either add a tag to the resource or the file for this very reason. So maybe you should just suggest this to the servers you play on if the re-downloading bugs you.

[yannickboy15]

This sounds like a load of hassle for probably nothing. So what servers is it exactly that you keep switching between and how much file download is there that you must re-download every join?

Really this is the problem of the server owners for not renaming files which they've edited, whenever I edit a community resource I'll either add a tag to the resource or the file for this very reason. So maybe you should just suggest this to the servers you play on if the re-downloading bugs you.

If you look at the amount of copycat servers of the OwlGaming gamemode as an example that are laying around, you'll be looking into 500mb of server download due to mods that are applied and required for those servers.

I'm now speaking out of a server owner perspective, as I know that if people play on multiple servers with different script adjustments, then they'll have to redownload often a shitload of things. It's been a thing for ages and yes, we indeed have custom resource names to avoid this. Yet it is still an issue others might experience aswell.