API Error 500, Probable Cause is this Mod

[CraftedFury]

https://github.com/Moulberry/NotEnoughUpdates/blob/4369156ef9e0a96e0f789b27fa88fff9b2f0f2c7/src/main/java/io/github/moulberry/notenoughupdates/auction/APIManager.java#L283-L301

I'll break it down so people don't have to wonder what they are looking at.

• For every single user running this mod (every 5 seconds):
  • If they haven't queried the auctions API yet, query page 0.
  • If the query lastUpdated value is not identical to the saved lastApiRequest:
    • If config value quickAHUpdate is enabled (not enabled by default), and user is currently looking at the custom auction house screen, set doFullUpdate to true, and query every single page on the auction house on next call (< 5 seconds away)
    • Set all auction pages as needing an update.
  • Set lastApiRequest to lastUpdated
  • If doFullUpdate:
    • Set doFullUpdate to false
    • Call getPageFromAPI for every page in the auctions endpoint, all at once, (dozens of requests in a second)
      • If any of these requests reach the (If the query lastUpdated value is not identical to the saved lastApiRequest), then repeat the above.

Depending on how often the skyblock/auctions endpoint is updated, this is the most likely cause of the API problems currently. It was advertised on YouTube and I estimate it's got a few thousand users, and maybe hundreds of active users.

So if we take a generous estimate and assume 500 active users. Every 5 seconds, all 500 users request at least 1 auction house page, regardless of if the custom auction house screen is visible. If the custom AH screen is visible, then for every user looking at the auction house, they are also sending out several dozen requests, every 5 seconds.

Let's assume of those 500, 50% (1/2) have the quick AH feature enabled and are using the remote auction house feature at the same time. That's 250 users sending 45 requests each, all at once, for a total of ~11,250 requests every X seconds.

X is either 5 seconds (the delay in his mod), or how often the skyblock/auctions endpoint is set to update.

[jecsham]

API is not working properly right now

[Alexander3a]

it only updates when u have the gui open so the number is very probably way lower most users dont know about the ah and most dont have their api inputed

[CraftedFury]

it only updates when u have the gui open so the number is very probably way lower most users dont know about the ah and most dont have their api inputed

I accounted for that. The assumed users who may use it all at once may be incorrect, but some combination of use cases in this mod is the most likely culprit, total size or not.

Hypixel should add a way to download the entire skyblock/auctions endpoint without requesting pages, and limit it specifically to once per minute or something.

[ThatGravyBoat]

A good way for the traffic to be reduced would be if they could add the ability for the recently updated items to be sent on page 0 instead of it being updated through out all pages so that mods like this don't have to download every page they can just download 1 page to get the updated items.

[puppy0cam]

The solution to this is going to be to have a system for receiving updates to data in real-time instead of requiring clients to constantly poll for, and retrieve a large amount of unchanged data.
I would propose a public WebSocket or Kafka server for clients to be able to connect to and receive data from as it happens. In this case, the API would only need to process data once and could then distribute a small payload to all connected clients to indicate the changed resource.
The RESTful endpoint should be kept to ensure clients are able to sync their state at startup, and just listen for changes for the duration of the program instead of constantly pinging the API to receive large payloads of data.

Such a system has been used successfully before for similar purposes which you can see from https://chatwars.github.io/chatwars-api-docs/public-exchanges.html. Historically the public data from this similar use case was hosted on RabbitMQ, however it proved to be too much of a hassle hurdle and the number of users was too much for RabbitMQ's poor authentication service. As such it was migrated to Kafka which has proven to be extremely durable and useful for real-time data and even retrieving historical records of up to a month old.

[Ekwav]

Now it is down completely, I am getting the response
{"success":false,"cause":"Auction data has not yet been populated"}

[ThatGravyBoat]

yeah it seems like they shut it down, the mod that was thought to cause it has switched to their own server and system to get their ah data to put less stress on the api.

[AzureAaron]

This could actually be the cause after reading this, whilst the API was having issues the Auctions API would not retrieve any data for a specific player. It just returned {"success":true,"auctions":[]} and didn't list any ongoing auctions for the specified player even if they had multiple running auctions.

[ConnorLinfoot]

This was due to an API outage that should no longer be an issue.