Get progress and update progress

[kingboyk]

Should get progress and update progress be merged, or is there a good reason why they are seperate commands?

[Bhargav-Rao]

update progress is a api heavy command and should not be used frequently. The bot updates the scores automatically every hour. You can get progress any number of times in between the updates and it fetches from the DB. Merging these two commands would result in the API being called everytime, and would eventually run out of quota.

[makyen]

@Bhargav-Rao Based on testing of the update progress command in chat, it appeared that the updates from the SE API, at this point, are not done in response to the update progress command, but only on the schedule. In that testing, using update progress resulted in Gemmy reporting the same remaining quota as prior to the command, and returned a "Progress has been updated!" result significantly faster than it's possible to fetch 200 responses from the SE API¹. I expect that in the past the update progress did manually initiate a fetch of SE API data, but that no longer appears to be the case.

In addition, that testing of the update progress and get progress commands indicated that an update progress was required after a scheduled fetch from the SE API in order for the information reported by Gemmy to reflect the current data. If update progress was not used, then the reported data reflected the state prior to the most recent scheduled fetch of SE API data (not the last time update progress was run; i.e. the scheduled fetch did update the progress, but it was to the progress prior to the scheduled update). See the difference between this get progress report after an automatic fetch, but before a update progress command and this get progress report after a update progress command. For confirming that it does update to the progress to prior to the scheduled fetch, not the last update progress, see this get progress report compared to the first one I linked.

---

1. Fetching 200 pages from the SE API should take an absolute minimum of about 7s, but 25–45s is much more likely. 25–45s assumes that you're requesting multiple pages in parallel and doing so at 1/3 to 1/2 the maximum request rate the SE API permits per IP (30 req/s), and that you don't get a backoff, which will depend on how fast you're requesting and server load at that time. This is based on my actual experience making that number of requests, or more, from the SE API. While I've done multiple sets of fetches like this, I've only recorded the time for the most recent one (now two), which was 38.4s at 1/3 the maximum request rate (i.e requesting at a max of 10 req/s) with a queue depth of 10 requests. With a queue depth exceeding the number of pages, it was 26.3s.

[Bhargav-Rao]

@makyen That is interested, I was informed that update would update from the API and get fetches from the local DB. Anyway, I guess we can clarify this by diving into the code. Or perhaps @g00glen00b knows about it already?