-
Notifications
You must be signed in to change notification settings - Fork 412
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
missing post count in new api #590
Comments
While going to /tags can help for single tag counts, searches with more than 1 tags would need to use posts.xml to know the count. |
Binary search over the page number combined with limit=100 seems like the only solution. |
But if the results are more than 100,000 you won't know the result number. |
There's no easy way for me to implement this without copy-pasting a lot of code. I could stuff the total page count in the post elements but that's inelegant. Is there any reason why the iterative approach doesn't work? |
In practice it is simply to slow to iterate over the posts. From my location it takes about 1 second for a request to complete, if you need to do just 5-10 requests it becomes way too slow. (Of course you could simply spam request in parallel, but that is not elegant and programming asynchronously is probably too tricky for beginners.) I'm currently using post count in two ways, first to display it to the user, and secondly as an aid to control my partial cache. Neither justifies the extra requests, it would make the UI less responsive and the point of the cache was to make fewer requests. |
One thing I can do is pass the count in the HTML header, as X-Total-Pages or some equivalent. |
my idea of total post count should only be declared once and should be included in the first server's reply as an xml element or json attribute. having to declare it in every post entry requires continues updating to maintain accuracy which could probably even slow down the process. btw, is the api access still throttled and limited to 1 request per sec per user? |
I would also prefer to have it directly in the XML and JSON data. I think the throttling was removed after a short period of time, though I haven't tested it. I don't need to provide any user information anymore though, so at most it is on the IP level. |
This might have been what RaisingK meant, but by calling the raw http (&limit=200) and extracting the number from the paginator you can get a fairly accurate result in a single call. |
Since we already have a rough count from ::Post.fast_count() attached to the paginated @posts array thanks to the implementation of total_count in the paginator module, wouldn't it be sufficient to just add a reimplementation of Array#to_xml in the paginator module to shim in an extra "total-count" attribute on the root "posts" element? (Extra bonus: you'd get a "total-count" attribute on all other paginated XML files) |
My algorithm is basically:
Searching for
|
I added it to the XML for posts. |
Idea: Don't return result count with the results themselves; instead create a separate action that returns just the result count. Ideally this would work for all pages (not just posts) and with both xml and json formats. eg. That might be neater and easier to implement than the other ideas so far. |
I can't tell if it no longer works or I'm doing something wrong. I think I remember it working a month ago... |
It still works for me. What do you see if you visit this link normally?: http://danbooru.donmai.us/counts/posts.json?tags=3girls+1boy Should be: {
"counts": {
"posts": 1359
}
} |
I was doing it wrong, thanks. |
the post count is missing in the new posts.xml ( example: http://danbooru.donmai.us/posts.xml?tags=touhou+bunny_ears ) , previously the old post/index.xml ( example: http://danbooru.donmai.us/post/index.xml?&tags=touhou+bunny_ears ) has one.
somehow, this reminds me of #207 but the counts seem to be very accurate now. and one advantage of post count is to have an estimate of how many pages to fetch, or are we left to an iterative solution until there are no elements?
The text was updated successfully, but these errors were encountered: