Skin lookups reverting to pre-UUID skins or falling back to Steve

[LukeHandle]

Approx 7am UTC today Mojang disabled the (legacy) lookup via Username:

http://skins.minecraft.net/MinecraftSkins/LukeHandle.png (404)

This can be seen by a sudden sharp increase in the FallbackUsernameS3 we are logging/stating.

Sneak preview of "upcoming" metrics:

Most of the code required here is written over at https://github.com/minotar/minecraft/tree/develop and was on my roadmap for the coming months (infrastructure migration was previously being prioritised here). I had been hoping to make some further changes to optimize the flow here for UUID traffic before making the switch, but that will have to wait as well.

I will look in the next 6 hours or so in pushing a smaller fix to resolve this (currently working the day job).

[Dakirutz]

Oh that come from here ! Thanks, as we use your domain to get skins at https://socialcraft.fr/stream we are waiting that minotar.net give again the good skins, thanks for taking time to correct this, bad that mojang disable this api.

Gracefully,
SocialCraft Team

[TNTUP]

I'm still using your legacy PHP code and I can't seem to find a proper way to cache the queries names->uuid->texture as I get often ratelimited... There's no up to date skin server without ratelimiting as I know, the minotar.net still serves steve skins :/

[LukeHandle]

I understand them deprecating/decommissioning the old endpoint/API, some notice may have been nice, but they don't owe that to us! They had run it way longer than I expected for the transition process.

Our issue is directly with the API for looking up Usernames from UUIDs:

https://api.mojang.com/users/profiles/minecraft/LukeHandle

The rate-limit here does not allow the volume of traffic we need to request from it. I'm trying to engage them about this.

You are more likely rate-limited by the UUID -> Texture API, see our (now out of date) skin docs here: https://github.com/minotar/skin-spec:

The sessionserver.mojang.com domain uses very aggressive rate-limiting and will return a 429 429 [sic] if the same UUID is requested from the same source IP within 30 seconds.

If you do not cache the data for this endpoint, you will be unable to use it due to the strict rate-limiting.

===

When we reach resolution here, the following would be suitable for using in your legacy application as a replacement to the deprecate Mojang API:

http://minotar.net/skin/LukeHandle or http://minotar.net/skin/LukeHandle.png

===

I will further explore some other option this afternoon to see what we can do to help alleviate the issues.

[TNTUP]

That's im doing quite now using the /skin/ endpoint from your API but though some ends with a steve... For now some of the skins still work (noticed it does fallback to the s3 one)

Though I don't know that part of PHP to cache that... Better just to stop the legacy skin API... Thanks anyway for your help ^^

[Dakirutz]

Do you think that https://minotar.net/avatar/microscraft could work again in less than 19hours ? We are launching new things at this moment and having kikoo head on all our pages is not so great haha

SocialCraft

[LukeHandle]

I honestly doubt it :( I've been spinning up more infrastructure which extends the time we can use the API, but still choke too early.

I have a few more ideas, but the time I can dedicate to this is limited. This is my second night up until 2-3am (waking up at 6:30) 😭

[TNTUP]

I'm soon to shutdown my Skin API because I couldn't find any solutions because I kept being rate limited even from another Skin API.. Guessing its either the end of public fetching skins or someone else will make a magic API which wont ratelimit idk cries

Or better just set to private for my own server/query services... I have a russian skin website which uses my URL thats generating alot of 429 :/

[LukeHandle]

@Dakirutz I retract my "doubt it". It's now a "possible".

I've doubled the server count again. We are now at 8x our previous server capacity. By my bad maths, this should cover our lookups for "most" of the day, but spikes will cause issues. I'll monitor how this works for a few days and can further evaluate then.

I am directly tracking the amount of RateLimit failures seen so will get some idea on if this is acceptable or not. We've just gone 15 minutes without an error, but this is the lowest traffic time...

[Dakirutz]

I'm sorry, I saw that working 2hours like you said but then it just get away from some (less) people.

As we are verfying user premium state, we got uuid for all our confirmed players. I then find an other api like your's that took uuid and then have no problem. We have ours head back!

But it as too a cost fors us, a request to get the uuid is made in our database, and I will be glad to set back minotar if you can find a valuable solution.

Sorry for having stressed you, thanks again for all, hoping that my little donation can excuse that I stop using your api for now :/

If you have news don't hesitate to tell us, i will follow this topic and as I said if you find a valuable solution we will come back to minotar with socialcraft.

Thanks again!

[LukeHandle]

Huh, I didn't intentionally close this ticket last night, but oh well. This is resolved in the short term. We're now operating at 13 times our previous capacity in terms of backend servers. When I have time in the coming weeks I will work on optimizing things here. We'll use the metrics from this period to then adjust the capacity as required.

[Dakirutz]

This will cost you a lot more no ? Are donations enough 0.o ? :/

[clone1018]

We had 2 donations in 2017, totaling $20.50

[Dakirutz]

Hum okey but who is "we" ? ahahah I don't know you ! :o
If you are from minotar, i'm sad to learn that... I donated 10$ 4 days ago to minotar, but I hope i will not be the only one this year !

[clone1018]

We is me and LukeHandle, we run Minotar. I received your donation, it was very appreciated!

[Dakirutz]

Oh okey, sorry didn't noticed that !
I hope so (:

[LukeHandle]

I will re-open this to keep awareness of the issue until the long-term solution is implemented. Less than 4% of our backend requests are encountering the issue currently, but we can improve on this.

[ghost]

Can I just run your imgd server and cache the skins myself for use in my application and site?

Mine seem to show Steve more often than not :|

[ghost]

Just installed imgd, skins still fallback. What am I missing? Shouldn't I simply be able to cache my own players skins?

[10:56:18.073249] INFO Failed UUID lookup: nogura (unable to GetAPIProfile: likely not found, maybe forbidden)

[10:56:18.114052] DEBU Failed Skin S3: nogura (FetchWithUsernameS3 failed: unable to Fetch Texture: likely not found, maybe forbidden)

I am confused as to why I am still accessing your servers if this is a self hosted skin fetcher? Shouldn't it be getting the UUID from the player profile via Mojang, then associating and caching the skin based on that? It appears to just query your server...

Maybe I am confused? Please advise.

[LukeHandle]

Hey @KipperedSnack, Apologies you're seeing issues here (both with Minotar directly and with imgd) :(

Running a local instance should resolve the issue in the sense that you are no longer bound by the Rate Limiting that we are occasionally facing. The FetchWithUsernameS3 would be called when the initial request failed - this is no deprecated as well though so also fails (to Steve).

The issues here today are because Mojang have further blocked our (generic Golang) User Agent - by coincidence, something I was just about to set properly and reference back to us. Further changes though and unit testing made this a more involved change though.

I guess this will be a long evening :/

[ghost]

No its fine lol, I was just confused as if I had set it up incorrectly. Much appreciate your work.

[LukeHandle]

So, I have a build kind of ready - but Mojang seems to now be having issues:

[{"minecraft.net":"green"},{"session.minecraft.net":"red"},{"account.mojang.com":"green"},{"authserver.mojang.com":"green"},{"sessionserver.mojang.com":"red"},{"api.mojang.com":"red"},{"textures.minecraft.net":"green"},{"mojang.com":"green"}]

[LukeHandle]

So, this change is now live and we are back. I've tagged a release at https://github.com/minotar/imgd/releases/tag/2.11.0 as well

[TomLewis]

Is this why I have 50% of my avatars always showing steve for months?

[LukeHandle]

@FrozenBeard afraid so. Currently working on a better caching system which will offer a graceful fallback and attempt to better protect against these issues. Been balancing some work commitments these past 2 months, but I'm back to putting in a few hours on the weekends and before/after work where possible.

[TomLewis]

@LukeHandle dope. I was at the point where I was considering writing my own as its driving me nuts.

[LukeHandle]

@FrozenBeard yesterday I had a brainwave on potentially how we can continue to keep up with the crazy amount of traffic we get. Further to this, I'm separating out all the caches (Username->UUID, UUID->SkinHash,SkinHash->Image).

We'll be able to keep a larger cache or Usernames and UUIDs, refresh them more often and with any luck the Skin is locally cached, otherwise the rate limiting on that Image request is non existent.

Unfortunately, It's turned into a fairly large rewrite and having to account for lots of different variables (eg. we have stale data, we tried to hit API but failed, fallback to old data - better than a Steve).

I'm optimistic I can be done with the core changes in the next 10 days and hopefully get tests written in that time.

[LukeHandle]

So, I have a build that passes the original tests and does what I think it should do - probably. The next steps:

1. Tidying certain bits of code up
2. Re-adding non-essential stuff I removed for testing
3. Either update existing config or switch to new library based on new cache options
4. Try and write some tests to better exercise the new bits I have added and that the logic lines up with expectations

The first 3 should be straightforward, and possibly achievable before the weekend. 4 is what I expect will eat a lot of Saturday. Hopefully I'll then be able to think about deploying this out.

[LukeHandle]

Okay, I re-prioritized on 4 as it's the best way to verify things are working as I expect. Saturday was spent on 1 and 2, then separating the mock testing server from the minecraft package. Today I used that testing package to write maybe 50% of the tests. Still pending 3. I've lost a chunk of time this afternoon/evening as something has come up. I'll try and cram in more before work, but next good chunk will be Wednesday where I'll have 5+ hours.

[LukeHandle]

Just about managed to get this written within the 10 day window I set myself. 😲 💀 💤

https://github.com/minotar/imgd/tree/develop (using develop at https://github.com/minotar/minecraft/tree/develop)

I'll look to deploying this tomorrow 🤞

[LukeHandle]

This is now deployed, I'll check-in through the day, but it seems to be okay so far. It might take a day or so to get up to speed, but once the Username->UUID cache is primed, it should be fairly well (🤞).

[LukeHandle]

Hey @FrozenBeard, from what I can see looking at our stats, we are very rarely hitting rate-limits here. We have something like 3 million usernames cached and ample space for more. We optimistically cache a working username for 60 days (further verifying the UUID still matches upon request) as well as a 404 username for 14 days so this should minimise the main issues we saw.

I will once again close this issue out 😄

[TomLewis]

This is fantastic news! Thanks for the update man, I really appreciate it.