Empty data for 2020-09-01 / 2020-09-02

[mildbyte]

Hi,

Thanks for building this! We've a daily Airflow job that grabs the raw JSON files from this repo and ingests them into the https://www.splitgraph.com/splitgraph/congress_tweets dataset (making it importable into Postgres / queryable over a REST API/SQL).

I noticed that https://raw.githubusercontent.com/alexlitel/congresstweets/master/data/2020-09-01.json and https://raw.githubusercontent.com/alexlitel/congresstweets/master/data/2020-09-02.json are empty -- is this supposed to happen?

[alexlitel]

@mildbyte Seems there is some issue with the Twitter API search endpoints the app consumes to get tweets. For some reason, URL encoded query parameters are currently broken. It's not something I can fix on my end. I hope it gets resolved soon though. Whenever it is resolved, I'll propagate the files.

See this thread on Twitter API support forums for more details about the bug.

[alexlitel]

@mildbyte This is still not working so in the meantime I'm just going to be adding data manually. Will close issue whenever there's a fix.

[junwang4]

As of today (2020-09-16), tweets from 09-10 to 09-15 are empty.

[alexlitel]

@junwang4 I'm aware. Until Twitter fixes the API issue mentioned above, I have to manually run a script to add tweets manually.

[leslie-huang]

Hi @alexlitel, the underlying bug may have been fixed! Some users reported the API is working again for them, in the thread you linked above. 9-20 through 9-26 are currently empty though (as of 9-27). Thanks for maintaining this repo!

[junwang4]

@junwang4 I'm aware. Until Twitter fixes the API issue mentioned above, I have to manually run a script to add tweets manually.

Can others run your script to get new tweets manually?