Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CSGOFloat API Usage #120

Closed
Step7750 opened this issue Feb 6, 2020 · 14 comments
Closed

CSGOFloat API Usage #120

Step7750 opened this issue Feb 6, 2020 · 14 comments
Assignees
Labels

Comments

@Step7750
Copy link

@Step7750 Step7750 commented Feb 6, 2020

Hey Gergely,

I was looking at request logs for the CSGOFloat API and it seems like users of your extension are generating around 3-4 million float requests per day. Given the amount of users on your Chrome extension, this seems to be quite high per user and wanted to make sure that there weren't issues on your end.

I looked into the request metadata a bit and it seems like a lot of these requests are duplicates.

Thanks

@gergelyszabo94

This comment has been minimized.

Copy link
Owner

@gergelyszabo94 gergelyszabo94 commented Feb 6, 2020

@Step7750

Hi!

Thanks for reaching out and wow, those are big numbers!
Could you specify what you consider a duplicate so I can give you a better answer?
Every client (extension) should be caching the float data that it retrieves for at least 24 hours - if it's used at least once then the cache does will not expire for another week and every time it's used it's extended.

Let me know what details you need and thanks for providing this great service!

@Step7750

This comment has been minimized.

Copy link
Author

@Step7750 Step7750 commented Feb 6, 2020

I consider a duplicate to be a request for the exact same item from the same ip in succession. It doesn't seem like all your users are affected, but a handful are sending 400-500k+ requests where the majority are the same items over and over. It also looks like it is typically items in the same inventory that keep on getting queried.

I'd imagine this is a decent amount of bandwidth for those users. I also don't think these are bots spoofing your origin header since they wouldn't be requesting duplicates like this.

@gergelyszabo94

This comment has been minimized.

Copy link
Owner

@gergelyszabo94 gergelyszabo94 commented Feb 6, 2020

Thanks for the details, I will try to look for reasons why this could be happening. They are valid requests and the API returns a valid response in these cases, right?

@Step7750

This comment has been minimized.

Copy link
Author

@Step7750 Step7750 commented Feb 6, 2020

Yeah, they are valid and returning status 200.

@gergelyszabo94

This comment has been minimized.

Copy link
Owner

@gergelyszabo94 gergelyszabo94 commented Feb 6, 2020

So if this returns 'error' then the job is pushed back to the queue which is very likely responsible for creating an infinite loop:

fetch(getRequest).then((response) => {
if (!response.ok) {
console.log(`Error code: ${response.status} Status: ${response.statusText}`);
sendResponse('error');
}
else return response.json();
}).then((body) => {
if (body.iteminfo.floatvalue !== undefined) {
let usefulFloatInfo = extractUsefulFloatInfo(body.iteminfo);
addToFloatCache(assetID, usefulFloatInfo);
if (usefulFloatInfo.floatvalue !== 0) sendResponse({floatInfo: usefulFloatInfo});
else sendResponse('nofloat');
}
else sendResponse('error');
}).catch(err => {
console.log(err);
sendResponse('error');
});

Does the API ever return a valid response where the floatvalue property does not exist and would fail a the !== undefined check?
This is my best guess for now.

@Step7750

This comment has been minimized.

Copy link
Author

@Step7750 Step7750 commented Feb 6, 2020

Here's an example of one of the float requests that are being retried continually: https://api.csgofloat.com/?url=steam://rungame/730/76561202255233023/+csgo_econ_action_preview%20S76561197972110750A17650580014D875389816902081350

I believe that there should always be a floatvalue, but it depends on whether Valve sends it in the gc proto (which they do for things like stickers and such).

@gergelyszabo94

This comment has been minimized.

Copy link
Owner

@gergelyszabo94 gergelyszabo94 commented Feb 8, 2020

I changed the logic so failed requests don't get pushed back to the queue. It will be in the next release (most likely tomorrow night).
I think it should be clear after a few days of the rollout if it fixed the problem.

@Step7750

This comment has been minimized.

Copy link
Author

@Step7750 Step7750 commented Feb 8, 2020

Update: These requests are peaking at 5-10 million in the past 24 hours. I've had a user contact me that they had a Cloudflare rate limit IP ban when accessing db.csgofloat.com and they also used the csgo trader extension. I've asked for them to share reproduction steps here.

@putzN

This comment has been minimized.

Copy link

@putzN putzN commented Feb 8, 2020

Heii
I've experienced a ban from db.csgofloat.com because of checking too many items at the same time, thats at least what i think.
After the server maintenance todays morning i tried to buy a new low float nova, ump and mp7. so i checked like two or three skins for each weapon (may vary a bit, im terrible im memorizing stuff :D) at the same time with the chrome extension. after that some floats didnt want to load and i couldnt check the database anymore because of the error seen in attached picture

image
This hasnt been a problem for me, even with similar search requests, until today ~1:30pm CET.
after that i asked others if they experienced the same and they didnt, so i contacted @Step7750

@gergelyszabo94

This comment has been minimized.

Copy link
Owner

@gergelyszabo94 gergelyszabo94 commented Feb 8, 2020

@putzN how many listings do you have set to load on the market? do you use other similar extensions?

@putzN

This comment has been minimized.

Copy link

@putzN putzN commented Feb 8, 2020

@gergelyszabo94 100 listings per page, and besides float market checker I only use adblock

//edit: spelling and grammar

@Step7750

This comment has been minimized.

Copy link
Author

@Step7750 Step7750 commented Feb 8, 2020

I think the issue might be that on my end, these requests return status 200 from the origin, but if a user makes too many requests in a short time, Cloudflare bans their IP which causes the extension to continually retry due to network errors.

I'll apply a similar fix that doesn't auto retry failed requests into the next CSGOFloat extension release. It seems like this is an arbitrary rate limit set by Cloudflare.

@gergelyszabo94

This comment has been minimized.

Copy link
Owner

@gergelyszabo94 gergelyszabo94 commented Feb 9, 2020

I have published the new version with the fix, it's "Pending review" in the Chrome Web Store at the moment.

@Step7750

This comment has been minimized.

Copy link
Author

@Step7750 Step7750 commented Feb 10, 2020

API request distribution is looking more normal now so I'll close this issue.

@Step7750 Step7750 closed this Feb 10, 2020
@gergelyszabo94 gergelyszabo94 moved this from In progress to Done in Kanban Board Feb 10, 2020
@gergelyszabo94 gergelyszabo94 moved this from Done to Released in Kanban Board Feb 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Kanban Board
  
Released
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.