Query total tweet count

[casassg]

I think it would be really useful to add an option to get the total amount of tweets for a defined query. This can be done by pulling counts for that query and adding them up. It would allow checking if you have enough requests with the current tier of premium/enterprise (500 tweets/request) and calculate the total cost to run the query. Especially useful if you need to use the premium API only for specific queries (ex: analyzing tweets from users during a specific event).

This would be a similar feature to what I think GNIP PowerTrack used to have, to check the cost of the query before starting it.

I can work on a PR myself if this is considered "non-prioritary" internally, as I'm currently doing this manually and it's quite tiresome.

[arvindiitm]

@casassg Yeah I agree with you. It will really help.

[jrmontag]

Thanks for the suggestion, @casassg. It's an interesting approach, and I can imagine how it could be built on top of the existing /counts API functionality. One aspect that may be challenging for your accounting use case, is that the requests aren't guaranteed to include the max 500 activities because pagination occurs over both activity count and time (30 days). I'm not certain where in the library would make the most sense to implement this, but I'll talk with the team and report back.

[venomouscyanide]

@jrmontag Is there any update to this? Any way to determine the total tweet count for given query.

[casassg]

@jimmoffitt tagging you here just in case you have more insights here 😄

[jimmoffitt]

Hey there, I missed the mention from two weeks ago ;(

Are you suggesting that a call to the counts endpoint should also provide a 'total counts', that is a grand total across however many paginated responses are needed?

Yes, that would be a useful 'helper' function. FWIW, the TwitterDev Ruby client I wrote does that.

I recently adopted this Python library to get it ready for Labs (and soon v2). These new versions of search do not yet include a counts endpoint. The counts functionality is on the roadmap, so when that time comes, the Labs/v2 client should be updated to offer a helpful grand total.

For the enterprise/premium branch, let's hope for a pull request with those updates.

[casassg]

@jimmoffitt mind if we re-open the issue to track whenver that gets merged? Also, it may be worth adding to the roadmap a way to get the full count of the data. I used to use this w @jenningsanderson to calculate how many requests we would need to download a dataset for @Project-EPIC. This would allow us to estimate cost better.

This is similar to the historical powertrack used to do back in GNIP days