New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Why backbones? #14

Closed
20zinnm opened this Issue Jul 30, 2016 · 22 comments

Comments

Projects
None yet
3 participants
@20zinnm

20zinnm commented Jul 30, 2016

I have no idea why you use "backbones" to add more API requests. You could literally use a list of proxy servers and simply route requests through them. You could even simulate Mojang's rate limiter for every proxy server so you don't go over and waste time fetching resources to get a rate limit error.

@sweetcode

This comment has been minimized.

Collaborator

sweetcode commented Jul 30, 2016

@20zinnm I am using currently Digitalocean to host my servers (SSDs, easy scalable, etc. ) but they currently don't support multiple IP addresses per server. The feature was requested back in 2012 (https://digitalocean.uservoice.com/forums/136585-digitalocean/suggestions/3224425-more-ips) but while I am writing this I recognize that DO offers floating-ips... just maybe I can assign more than one floating IP to my droplets. I will test that, but not now.

The idea of using a proxy sounds also great but I wont make that change to the current version, this will be part of 3.0. :)

@20zinnm

This comment has been minimized.

20zinnm commented Jul 30, 2016

@sweetcode

I love DigitalOcean (and I use them), but this doesn't change my proposal. In fact, my proposal could eliminate all of your backbone servers and their related expenses--here's how:

  1. Go to http://proxylist.hidemyass.com/ and pick your favorite 50 proxy servers (yes, you will have to manually copy them over into a text file--or just buy their huge list thingy).
  2. Add a system (https://github.com/akirk/php-ratelimiter) that rate limits access to the proxy servers by IP. (Instead of using $_SERVER[] use the proxy server's IP and port string.)
  3. Then, whenever you need to make a request to Mojang, just feed it through the proxy servers.

Issues:

  1. Proxy servers are not private. Anyone could potentially exhaust the API calls for a given server. This could be avoided by simply trying the next IP in case of a 429. Another, less heavy alternative to the cache is to simply have a list ["", ""] of proxies; when one returns 429, move it to the end. Just use the first proxy all the time. (This COULD lead to an infinite loop, but eventually it'd resolve, right?)
@20zinnm

This comment has been minimized.

20zinnm commented Jul 30, 2016

Floating IPs allow one IP to dynamically switch target droplets. It doesn't allow multiple IPs per droplet.

@20zinnm

This comment has been minimized.

20zinnm commented Jul 30, 2016

I wonder, would IPv6 be an option? DigitalOcean can dedicate a whole bunch of IPv6 IPs to any droplet.

@sweetcode

This comment has been minimized.

Collaborator

sweetcode commented Jul 30, 2016

@20zinnm Yep, IPv6 would be great... according to the PHP documentation of the lib this should work. The idea to use public proxys is tbh not a good idea from my point of view , because I don't like it to relay on public proxies to be available.

The IPv6 idea is great and I hope it works out! 👍

@20zinnm

This comment has been minimized.

20zinnm commented Jul 30, 2016

@sweetcode You can enable IPv6 free, but you have to configure each IP 😆.

@games647

This comment has been minimized.

games647 commented Jul 30, 2016

I think Mojang API isn't Ipv6 compatible.

@20zinnm

This comment has been minimized.

20zinnm commented Jul 30, 2016

@games647 Hm, just tested. It's not. Could we potentially tunnel it? But that would mean the endpoint of the tunnel is rate limited, which eliminates the entire point.

@sweetcode

This comment has been minimized.

Collaborator

sweetcode commented Jul 30, 2016

I don't think that DO will provide multiple IPv4 addresses in the future, because the request for this feature has so many up-votes but is also open for 4 years now (https://digitalocean.uservoice.com/forums/136585-digitalocean/suggestions/3224425-more-ips). - I found another provider called 'Linode' and they offer additional IPv4 addresses for 1$/address. This might be worth it, because it is also unknown when the Mojang API supports IPv6...

@20zinnm

This comment has been minimized.

20zinnm commented Jul 30, 2016

@sweetcode I don't think there are enough ipv4 IPs available to DO to do it.

@20zinnm

This comment has been minimized.

20zinnm commented Jul 30, 2016

I'm experimenting with Node and using proxies to get requests through.

@games647

This comment has been minimized.

games647 commented Jul 30, 2016

I think proxies aren't a good idea. Often they go down.

Although buying multiple IPv4 might work, I don't think this practise (>10 IP) applies to the RIPE guideline.

Maybe something like a master and slave instance where users could run a slave instance in the background of their VPS. (like https://github.com/unascribed/Visage)

@20zinnm

This comment has been minimized.

20zinnm commented Jul 30, 2016

@games647 I like that idea, but guaranteeing speed is an issue. Also, we have to be able to prevent data tampering.

@20zinnm

This comment has been minimized.

20zinnm commented Jul 30, 2016

I had an idea that people who want to run Slaves apply for a "slave key", which must be sent to the master when a slave starts. When a slave goes to be registered, it must send a CHECKSUM of itself (including the key) that the master can verify is legit--it's hard to fake a checksum when there is unique data.

@20zinnm

This comment has been minimized.

20zinnm commented Jul 30, 2016

If that made 0 sense, please let me know.

@sweetcode

This comment has been minimized.

Collaborator

sweetcode commented Jul 30, 2016

It makes sense, but there is actually now way to prove the integrity of a data-set that we get from slave. We could do spot tests on our side, so every nth request we compare our data to the data that we get from a third-party slave. If it fails we just ban the slave server until we manually verified that the data is compromised or not.

@20zinnm

This comment has been minimized.

20zinnm commented Jul 30, 2016

@sweetcode What if we do conformity tests? Define a Server resource shared code between Slave and Master. Then, when a master asks a slave for information, it gets scrutinized by the master using RegEx and type comparisons. For example, the protocol version must be one of the several options, and, say, the blocked status must be a boolean.

@20zinnm

This comment has been minimized.

20zinnm commented Jul 30, 2016

Because we must also consider, @sweetcode, what if the information changes between the time a slave send the data and the time the master receives it? There are also several plugins that change the MOTD dynamically every request (among other things).

@sweetcode

This comment has been minimized.

Collaborator

sweetcode commented Jul 30, 2016

I wont validate the actual values contained in those data-sets but we can validate e.g. if the slave provides the correct UUID for players etc. and as you already mentioned the syntax. How the tests are designed depends on what data we request. - The next thing to solve is updates? Do run slaves automatically an update if one is available or is this going to be a user driven process?

My idea is to have a database that keeps track on all slaves. The database knows how long the last request took, if there were any errors, if the slave has been banned, what the current version of the slave is and how many request the slave can do until he gets a cooldown etc.. Than we can always choose the best slave from this database to answer a request.

This is already so much stuff to consider that I created a new Slack room. It would be great if you can join this room @games647 you are also invited to join. If anyone of you is interested just send me an email to mcapi at krueger minus jan dot de and I will send you an invite.

@20zinnm

This comment has been minimized.

20zinnm commented Jul 30, 2016

Sent!

@20zinnm

This comment has been minimized.

20zinnm commented Jul 31, 2016

@sweetcode Please do add me.

@sweetcode

This comment has been minimized.

Collaborator

sweetcode commented Jan 5, 2017

@sweetcode sweetcode closed this Jan 5, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment