Riot CDN is now served over HTTPS

[ti-mo]

As reported by @fsalazar1977, Riot is now serving its content over SSL. (at least the client requests https:// resources) This cripples our ability to cache game content and breaks the client's update mechanism until fixed.

Looking for possible downgrade mechanisms to HTTP. Since Akamai's certs are used for l3cdn.riotgames.com, I assume cert validation is disabled, issuing self-signed to the client is rumored to work. (to be confirmed)

@RiotGames, please don't do this, run artifact integrity checks on the client. :)

[ti-mo]

TLS seems to be properly implemented, cert validity is enforced and there are no endpoint URLs going over plain HTTP that would allow them to be re-written. Schema cannot be reconfigured on the client as far as I can tell, and SSL cannot be disabled anywhere.

One thing, though: RADS/system/system contains the DownloadURL=l3cdn.riotgames.com directive, opening the possibility of specifying a domain we do own a valid certificate for, and translating that to l3cdn requests. If anyone can come up with any other suggestions, feel free to post.

Just contacted @RiotGames support with an explanation. Maybe, just maybe, they'll escalate so we can work on a solution together.

[v3n]

Hi guys,

Try adding the following line to {League install dir}/RADS/system/system.cfg:

UseTLS=false

Cheers!
Jonathan Howard // Riot Venia

[ti-mo]

Hey @v3n, thanks for sharing. This is fine as a short-term workaround for smaller events, but I'm afraid this won't do at scale. It also leaves clients 'unprotected' if they forget to remove this after the event, which kind of defeats the point. We have no way of pushing this variable to the client automatically.

I've gotten a response to my Riot ticket (#20689596) stating this has been escalated to operations internally, so I'm hoping for the best! Feel free to give this a little bump. :)

Thanks for reaching out, let me know when you know more!

[v3n]

@ti-mo,

Thanks for the feedback. We're currently investigating potential ways to allow LAN parties/PC cafes to patch as a long-term solution. Please do note that this config flag will not leave clients unprotected, as it will soon revert to using TLS after every patch.

Out of curiosity, would a command line parameter work better than a config option or do you expect similar problems?

Unfortunately, we cannot distribute signed certificates, as that undermines TLS security for Riot as a whole (ex: Microsoft's MD5 collision).

We are open to potential changes as long as they require player override/input, however, the proposed WAN/port 443/TLS failover undermine the security of our TLS implementation as it's only secure as the weakest link.

Are the computers in your LAN event participant-supplied or supplied by you?

Jonathan Howard // Riot Venia // Patcher Team

[ti-mo]

Hey @v3n, gotcha, makes sense to overwrite the config with every patch. I agree with all the points you make, any TLS downgrade/disable vector in the client compromises the whole implementation. I've written my thoughts below in a lengthy (sorry) post, hope it helps!

We run BYOC events ranging from 40 to 1000 participants, so any client-side intervention from our side is either highly inconvenient or not possible at all. The average tech-savvyness of participants is low; we could get away with requiring a config file edit, but a command line switch would be too much to ask, no one would bother. The goal is to do this completely transparently, 100% plug & play. All we really have control over is DNS, so this is what we use to redirect requests to our local servers. Before the participants arrive, we reinstall many popular game clients on our own machines to heat up the cache.

We're always trying to inconvenience the user at little as possible. With our current cache configuration, we don't listen on port 443 at all, so this is what the user sees:

This warning is generic and seems to be triggered when libcurl fails a GET operation from l3cdn. I've tried serving from 443 with a self-signed cert, the error's the same in both cases (serving an invalid cert as well as refusing connections on 443).

Personally, I'd go for a configuration setting to 'disable patching over secure connection', but also adding a shortcut button on the retry screen shown in the screenshot, to make it as painless for the user as possible. We don't want to make this permanent, so something like 'Disable TLS for 3 days', maybe? (in our case) Extra options in the settings menu would be nice, so PC Bangs wouldn't have to click the annoying pop-up every few days, but this currently doesn't apply to us. 👍

As a long-term solution, I would suggest having your build pipeline pack, sign, checksum and/or encrypt all artifacts you push out to the client. Your artifacts would still be cacheable, the client can detect and minimize errors in transit (I assume this is part of your reason for adopting TLS), you could also implement chunked transfer allowing the client to pause & resume and efficiently repair chunks if needed. (not to mention, this can be cached very efficiently! :) e0951c4 )

To make life easy for event organizers, you could provide an rsync endpoint so they can keep their copy in sync with a 100% integrity, without having to rely on a user to fetch it through the cache first.

[v3n]

@ti-mo, shooting off a quick reply before lunch:

Brainstorming here, you could provide a League shortcut that runs with the command line flag? This runs a risk of players using it during normal operations however, which is not ideal. Another idea would be a Powershell script that patches the config file.

I believe we are wary about placing a disable option in such a player facing way, just due to general lack of tech-savyness.

I'll see about updating that messaging for libcurl failures, definitely agree about the overly generic message.

One big reason for us adopting TLS besides general security is the prevalence of HTTP errors being injected by ISPs and other third parties. For example, ISPs attempt to (poorly) cache our updates and cause broken builds for players (that they could never recover from since there was no way for them to bypass the ISP cache).

Great post, passing your feedback onto the rest of the team, will keep you updated as towards their response.

[v3n]

We are exploring potential options internally for securely downgrading to HTTP.

[ti-mo]

@v3n, thanks for the quick response, really appreciate the openness!

I see the main concern here being broken caching layers, TLS really is the best way to poke through those caches indeed. If those ISPs manipulate DNS like we do, then there's no other way to distinguish between 'us' and 'them' besides checking the address class the l3cdn record resolves to.

How do you feel about only allowing (temporary or permanent) plain HTTP fallback when the endpoint is an RFC1918 address? It would require non-negligible effort for an ISP to implement a 'magic' bogon address in a private range that performs NAT to their caching layer. Either way, the user would still be warned about TLS errors, and if they start troubleshooting themselves, the launcher presents them with the information they need (including mandatory big red button to disable TLS). I don't see this part being really necessary.

The only concern here is the user getting MITM'd, since I've noticed plain .exe files going over the wire in previous iterations of your client. Also cases of cache corruption during some of our events, but we always carry the responsibility for that. You'd want to sign your artifacts precisely for this reason, the only attack vector here being a pre-compromised client that also accepts the attacker's signature.

[v3n]

@ti-mo, absolutely, we try our best to fix things when we break them!

I'm a huge fan of the RFC1918 idea and have raised it to the rest of the team. MITM shouldn't be a huge issue, we normally download packages and fall back to loose files if there is an issue. I believe we also checksum loose files as an additional security precaution.

Is there a specific time frame you are dealing with? (I can't make any promises as towards our schedule, but I've taken a interest in getting this resolved).

[ti-mo]

@v3n, alright, then I believe RFC1918 is our best bet. Need an equivalent for IPv6 though, which should be RFC4193.

I'm running a casual event on the 18th of March, the Multiplay guys are running i57 on the 25th, so the sooner the better! ;)

Can't hold this against you, though, thanks a lot for the cooperation so far, really makes a difference! It's in all of our best interests to get as many people patched, logged on and playing during these events. All I hope for is other platforms not making the move to TLS as well.. They might be a lot harder to reach out to.

[nexusofdoom]

1.Fortigate firewall SSL Inspection
do we know if Riot LOL will work with fortigate firewall inspection since it does almost the same as
nginx revers proxy with ssl.

2.will the launcher work if i get a domain name Example gameupdates.com and get a real SSL from like godaddy
and do a cname from l3cdn.riotgames.com to gameupdates.com

3.what about pushing the cert to the client like Windows Trusted Cert Store.
or a options to tell the client to look for the cert in a folder

[ti-mo]

@nexusofdoom

1. Fortigate requires their CA to be installed onto clients, we're not doing this. I would not run packet inspection on CDN connections for the life of me!
2. This would be a huge flaw, a well-written client will run validation against its given endpoint. A stub resolver is generally not aware of any cnames; a cname is not a redirect, merely an intermediary step in a recursive resolving process.
3. Sort of the same as in point 1, this again requires manipulation to the participant's computer. A TLS client is not required to obey the OS trust store (Riot could easily provide their own trust store). Also, this works differently across operating systems. Personally, I would never install some LAN's CA on my personal machine as a visitor.

[nexusofdoom]

so for my setup where all the pc's owned by me would this work?

2.will the launcher work if i get a domain name Example gameupdates.com and get a real SSL from like godaddy
and do a cname from l3cdn.riotgames.com to gameupdates.com

then install the CRT in the windows crt store on the windows computer.

do we know what the launcher is looking for since the domain l3cdn.riotgames.com forwards to akimia servers and at that point the current ssl is akimia

[ti-mo]

@nexusofdoom you would need a self-signed cert for l3cdn.riotgames.com, not for any of its cname targets. No guarantees the launcher obeys the Windows trust store, you'll have to test that.

[nexusofdoom]

thanks

[v3n]

We validate the l3cdn cert, so attempting to spoof it with a self-trusted won't work, otherwise we're vulnerable to CA sideloading attacks.

[nexusofdoom]

if that is so then any client of ours that uses Fortigate firewall SSL Inspection or any other firewall inspections will not work also then?

[v3n]

Correct.

The only workaround right now is this:

Try adding the following line to {League install dir}/RADS/system/system.cfg:

UseTLS=false

[nexusofdoom]

would there be a option for riot to make a check box in the launcher to use https Enable/Disable for updates.
so the default option is to use https. example you have a check box for Enable/Disable for Allow Peer To Peer Transfer.

[ti-mo]

@nexusofdoom Please read the whole thread :)
@v3n Any news so far?

[nexusofdoom]

http://wiki.squid-cache.org/Features/HTTPS

Quick skim of the thread you reference brought squid & HTTPS to mind.
Squid can be configured to cache HTTPS stuff.

Maybe some local DNS trickery and squid listening on 443 or nginx talking
to it.

Basically it sounds like riot is trying to secure tranmission channel
instead of validating the package(think md5sum or gpg signature of the
patch).
On Mar 9, 2016 3:23 AM, "Timo Beckers" notifications@github.com wrote:

@nexusofdoom https://github.com/nexusofdoom Please read the whole
thread :)
@v3n https://github.com/v3n Any news so far?

—
Reply to this email directly or view it on GitHub
#8 (comment)
.

[v3n]

@nexusofdoom: We actually do validate the packages, our switch to TLS was a move to punch through bad CDNs.

@ti-mo: It's in the pipeline, unfortunately we missed the cut for this patch. Could you give me some idea of the numbers associated with these LANs for some better justification to the team?

Sorry this is affecting you guys adversely, I understand that the 'workaround' is a pretty bad one for players who aren't necessarily technically competent.

[RobertJamesMichael]

@v3n: I am from Bolivia, I am not an expert in the telecommunication aspect but I can tell you something about the problem and how this is affecting a lot of people in my city (and my country).
First of all, Bolivia doesn’t have a coast and this mean that all ISP are obligated to get to the Internet Backbones through other countries like Brasil, Chile, Peru and Argentina, this problem makes that the internet connection prices for the final user are very expensive, for example a 3 MB (3mb DOWN / 512kb UP) costs about 90$ per month and this is very expensive for the majority of the people in Bolivia (consider that the minimum salary is 237$). This is the reason why the LANs steel are a place where many people is playing.
I have two LANs (80 and 50 pcs), since I installed the solution provided, my pcs began to get the game updates instantly and this helped a lot, since then I spread the solution to many LANs in my city, today (considering only the LANs I personally contacted and helped) may be approximately about 5000 - 6000 computers that are using the solution every day. Consider also that some of the “helped” began to spread the solutions to others (I think this easily reached a 30000 computers just in my city).
Since you changed the way the game updates, and you gave the alternative to add the line (useTLS=false) in the configuration (and I did that) many estrange problems began to happened like some computers got FATAL ERROR 0001, other ones began to download the complete game again (or many MB), all the computers are changing the region to Brasil when you start the game, etc. So we had to get the computers updated using old methods like coping the updated game from a “server” again. This two weeks were a fatal headache for me (and the others).
I hope this is a good reason for you to help us to get a solution to the problem.

[v3n]

@fsalazar1977: Just want to make sure that you used the correct configuration: UseTLS = false this is case sensitive. Can we also get some copy of logs on the bad machines?

@ALL: We're actively working on a fix.

[ti-mo]

@v3n i57 (march 25th) currently has over 2000 BYOC seats open, and looks like they'll be sold out. Thanks for the update!

[riconnon]

It might be worth noting that at i57 (and insomnia in general) client machines do not have RFC1918 addresses.

[riconnon]

@stevenh
Even if they merged it today, what are the chances that users would have updated to a version containing it before the event?
Way I see it you basically have to run with no LoL content caching for i57 otherwise there'll be loads of people on 6.5 that can't update to 6.6 at the event.

[stevenh]

If that where the full client yes, but it should just need the new patcher, which we could provide as a local download?

[riconnon]

@stevenh
how would you provide the information on how to user the locally mirrored patcher to the end-users?
I could see that being a recipe for a lot of work for helpdesk

[stevenh]

Local intranet, forums and tournament info pages.

They will soon find out when the updates instantly fail ;-)

[riconnon]

@stevenh
Alternatively... can we make l3cdn.riotgames.com a separate IP alias on the lancache instance and simply DNAT to the direct Riot service from port 443 but service port 80 as usual.
This way the first stage of the client update should go direct via HTTPS then the second stage will fallback to HTTP once the launcher is updated.
Needs testing, but would make it a zero config option again.

[stevenh]

@irconan We use seperate IP aliases for each service, so no changes there, however caching doesn't go through NAT as that would mean it would have to be processed by the FW, so that wouldn't work.

[v3n]

@stevenh: Should be releasing with the 6.6 patch which is supposed to release this week, it's in mainline now. I realize the release schedule isn't ideal, sorry for that.

@irconan: Is it possible to configure nginx to directly pass all 443 traffic to the real l3cdn? Or is that exactly what you're proposing and I'm just dumb? :p (not a network engineer).

[stevenh]

@v3n this week before Friday?

[ClearCarbon]

@stevenh I'm not attending i57 but I've been attending Insomnia events for almost a decade now. I've generally found communication about these sort of issues to be frustrating, not everyone checks (or knows about) the intranet and forums and the group I attend with don't participate in the tournaments. I've not been to an insomnia at the NEC yet but if you have the gaming room projectors like at Telford adding a message about this on there would be really helpful for a lot of the attendees.

[v3n]

@stevenh: today & tomorrow for various regions.

[nexusofdoom]

is the update live?
do you know when the RFC1918 changes will be merged to the mainline branch?

[ClearCarbon]

@nexusofdoom I've tested it and the change appears to have gone live

[RobertJamesMichael]

Hi Temo, just in case, LOL is serving via HTTPS again, do you know how to solve this?

[stevenh]

There's no really fix I'm afraid.

Caching proxies and https are just at odds, https is specifically designed to avoid that type of thing as if it could be done it you could sit as a man in the middle and sniff everyones content.

[riconnon]

@fasa1977 so you're resolving the names to an IP in RFC1918 block and it's still attempting HTTPS?
This would indicate that Riot have removed the workaround that was put in place for this exact use-case.

I wonder if @v3n is able to comment?

[ClearCarbon]

from what I could see the new client doesn't have the RFC1918 workaround. This was some time ago during the new clients beta.

[riconnon]

I guess we can hope that the workaround was just lost in a refactor somewhere and can be re-added.

[RobertJamesMichael]

Right @irconan and @ClearCarbon, @stevenh is it possible to get back the workaround? , we are having hard time patching LANs, we were making this manually, but, since the beginning of the new season, we have more people that want to play (thats why I am reporting this and not before), of course we do our job, but think in Lans with more 200 computers...no matter the bandwidth you get, it's difficult to patch efficiently that ammount.
Thanks in advance for your help!

[ti-mo]

Hi folks, curious to hear @v3n's input, but to me the move to TLS seems inevitable. CDNs are all offering TLS at the edge, up to the point of (I predict) plaintext endpoints not even being an option anymore a couple of years down the line.

If Riot is willing to keep supporting this, that would be great for us, but I'm afraid this is it.

[nexusofdoom]

what version of the client are you running?

this is my config
#riot
server {
listen lancache-riot deferred default;
server_name riot _;

access_log /srv/lancache/logs/Access/riot.log main buffer=128k flush=1m;
access_log /srv/lancache/logs/Keys/riot.log keys_range buffer=128k flush=1m;
error_log /srv/lancache/logs/Errors/riot.log;

Default Node

include lancache/resolver;
include lancache/cache-key-default;

location / {
# Some downloads are very large so we cache based on
# range to keep single downloads quick and hence ensure
# interactivity is good.
set $no_cache 0;
if ($request_uri ~* "(/releases/live/solutions/.*/releases/releaselisting)") {
set $no_cache 1;
}
proxy_no_cache $no_cache;
proxy_cache_bypass $no_cache;
proxy_bind 10.0.4.2;
#proxy_ignore_headers Expires;
proxy_ignore_headers Expires Cache-Control;
#proxy_hide_header ETag;
#proxy_cache_valid 200 206 360d;
proxy_cache riot;
proxy_read_timeout 300;
include lancache/proxy-cache;
include lancache/cache-range;
}
}

[nexusofdoom]

I am running on client version patch 8.1 and it looks like it downloading over http not https and working for me.

[jblazquez]

Hi all,

I work on League of Legends. I wanted to drop a note here to mention that over the next couple of weeks we're going to start switching League to patch in a different way from a new set of CDN endpoints, which will affect the caching setup that you have implemented here.

We're moving away from l3cdn.riotgames.com to two new hostnames: lol.dyn.riotcdn.net and lol.secure.dyn.riotcdn.net. We're keeping the HTTP fallback behavior that you're relying on here for caching because we don't want to break your use case. Please read below to see how we're going to be using the two new domains going forward. Hopefully this will be an easy transition for you folks!

lol.dyn.riotcdn.net is the hostname we'll use for downloading the vast majority of patch data. We default to using HTTPS here but we will fall back to HTTP if the hostname resolves to a private IP address. We consider all of the following "private IPs":

IPv4: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 (RFC 1918 private IPs), 169.254.0.0/16 (RFC 3927 link-local IPs) and 127.0.0.0/8 (RFC 1122 localhost IPs).
IPv6: FC::/7 (RFC 4193 unique local unicast) and FE80::/10 (RFC 4862 link-local IPs)

lol.secure.dyn.riotcdn.net on the other hand is a hostname we'll use for HTTPS exclusively, with no possible fallback. We will only download a small (~5MB) release manifest from here, which contains cryptographic hashes for the rest of the patch data. This allows us to safely use HTTP in the fallback scenario with no security implications, while still making the vast majority of the data cacheable.

Your DNS server should redirect lol.dyn.riotcdn.net only.

Another thing that's changing is the nature of our HTTP requests. We will make many more individual requests compared to before, and each request will be an HTTP range request, potentially with multiple byte ranges at a time. This will allow us to retrieve just the data that we need from the CDN. Our patcher expects CDNs to honor these range requests and respond with a 206 code, but it can gracefully handle 200 responses as well, albeit with reduced efficiency.

Hope this helps! Let me know if anything is unclear or you need more information about how this will work.

[nexusofdoom]

Thank you I have updated my installer.

[mintopia]

@jblazquez I'd just like to thank you very much for letting us all know about this and providing this fallback. It's brilliant, and I just wish the other game publishers and developers would do the same thing!