Twitter Restricts Filtered Stream API to Enterprise

[EthanC]

Although it has slowly rolled out, as my access was only cut off in the last 24 hours, Twitter now restricts access to the Filtered Stream API that this project relies on to the Enterprise tier. Today, my Docker instance of this project could not authenticate...

Traceback (most recent call last):
  File "/discord_twitter_webhooks/main.py", line 244, in <module>
    start()
  File "/discord_twitter_webhooks/main.py", line 200, in start
    delete_old_rules(stream=stream)
  File "/discord_twitter_webhooks/rules.py", line 23, in delete_old_rules
    old_rules = stream.get_rules()
                ^^^^^^^^^^^^^^^^^^
  File "/opt/pysetup/.venv/lib/python3.11/site-packages/tweepy/streaming.py", line 480, in get_rules
    return self._make_request(
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/pysetup/.venv/lib/python3.11/site-packages/tweepy/client.py", line 129, in _make_request
    response = self.request(method, route, params=request_params,
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pysetup/.venv/lib/python3.11/site-packages/tweepy/client.py", line 100, in request
    raise Forbidden(response)
tweepy.errors.Forbidden: 403 Forbidden
When authenticating requests to the Twitter API v2 endpoints, you must use keys and tokens from a Twitter developer App that is attached to a Project. You can create a project via the developer portal.

I see there is a rewrite branch of this project that appears to remove the API dependency and instead uses RSS feeds. This appears to be the best solution moving forward. Is this still in active development? Could a Docker image be provided for us to use?

Regardless, thanks for your work, @TheLovinator1

[TheLovinator1]

Hi, the rewrite is 95% done, so I still need a little more time before it is complete. I will post it here when it is done. Sorry for not being ready when the API died.

[thatjk]

Also hit the issue of my docker instance could not authenticate earlier today as well, any eta of the update, not to rush just more curiosity. Thanks for your hard work

[dziban303]

My API access has been killed off as of today. Eagerly awaiting the RSS rewrite.

[jedimage]

any update ?
I'm not sure about limit reading tweet per day will be some issue next ?

[EthanC]

The rewrite appears to have relied on nitter, per the following line:

discord-twitter-webhooks/discord_twitter_webhooks/add_new_feed.py

Line 184 in 25a0e9f

feed_url: str = f"https://nitter.lovinator.space/{username}/rss"

Twitter no longer provides data to unauthenticated users, effectively killing nitter. Research is ongoing for ways around this new limitation, but @TheLovinator1's project can't progress without the upstream issue being resolved first.

[TheLovinator1]

Hi, sorry for disappearing.

As Ethan said, I am using RSS feeds from Nitter for getting tweets but that broke
when Elon shut down the API that Nitter uses.

Elon says it is temporary, but I have no idea how long that will be. Or if we can even trust him.

If this doesn't get fixed soon, I guess I will have to find another way to get tweets. (Or just give up lol)

[Vilemouse]

Just chiming in here. the restriction has been lifted so nitter should work again

[TheLovinator1]

Hi, I merged the rewrite branch with master.

It is probably a little bit rough right now.

The README has instructions how to run the bot. I have to sleep now, but feel free to reply/create a new issue for everything that is missing or broken.

Sorry again for taking so long to fix this

[dziban303]

Awesome! I couldn't get it working on the Pi4 I'd been using previously; it wanted ffmpeg, so I installed it and added the envar pointing to it. Then it threw a sql error, to wit: reader.exceptions.StorageError: error while opening database: sqlite3.OperationalError: unable to open database file shrug

I just wanted to get it running before I went to bed, so I tried it in docker on a windows machine: it worked like a charm. I'll look at setting up a nitter instance tomorrow (or in the next few days, anyway, you know how it is) to support my own use case.

Is text filtering planned?

Thank you so much for getting this back up and running. We've been missing it!

Cheers

Edit: Tried again on Pi4 with latest build, threw a permissions error for the /home/botuser/.local/share/discord_twitter_webhooks directory for some reason. I chmod'd it to 777 and then it worked.

[EthanC]

Successfully running the rewrite in Docker on Ubuntu; thanks for your hard work!

Because this project now relies on Nitter RSS feeds, there seems to be a delay between a Tweet being posted and it being posted to the Discord Webhook - which wasn't the case before. I'm wondering if the default Nitter instance could set rssMinutes to 1 (or even 0) to alleviate this issue?