Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

404 on /twitter_user_to_rss #59

Closed
alexlamsl opened this issue Oct 17, 2018 · 24 comments
Closed

404 on /twitter_user_to_rss #59

alexlamsl opened this issue Oct 17, 2018 · 24 comments
Labels

Comments

@alexlamsl
Copy link

@alexlamsl alexlamsl commented Oct 17, 2018

Looks like the same as #52 - started about 10 hours ago.

Try https://twitrss.me/twitter_user_to_rss/?user=BillGates for example.

@anarchitectures
Copy link

@anarchitectures anarchitectures commented Oct 17, 2018

Noticed the same issue. Many of my twitter feeds are now either unreachable or unscrapable.

@kazux
Copy link

@kazux kazux commented Oct 17, 2018

same issue here

@kav2k
Copy link

@kav2k kav2k commented Oct 17, 2018

The request that pulled Twitter data (through scraping with Yahoo's YQL) now returns:

"message": "html table is no longer supported. See https://policies.yahoo.com/us/en/yahoo/terms/product-atos/yql/index.htm for YQL Terms of Use"

He's dead, Jim..

@ciderpunx
Copy link
Owner

@ciderpunx ciderpunx commented Oct 17, 2018

Hi folks.

Yes it looks like twitter are now rate limiting TwitRSS.me's IP address. TBH I'm slightly surprised it has taken them so long!

You should be able to run your own version of the code with no problems assuming you're not looking at loads of users/searches.

I can still see the mobile version of twitter from that IP, but I'd need to rewrite the code to deal with that version of Twitter - not likely to happen until the weekend.

Sorry for the annoyance, but y'know, blame Twitter.

@kazux
Copy link

@kazux kazux commented Oct 17, 2018

i was able to fix my issue by using the advanced search mode if you want to get feeds from a specific user it will be like this

https://twitrss.me/twitter_search_to_rss/?term=from%253ABillGates

@AlexeyVolkov
Copy link

@AlexeyVolkov AlexeyVolkov commented Oct 17, 2018

Does anyone know other services (cannot get the Twitter api key)?

publicate.it

@miksh7
Copy link

@miksh7 miksh7 commented Oct 17, 2018

I fixed it by changing the user name from low to upper case, but the next day the uppercase is blocked while lowcase is okay. Looks like Twitter's blocking logic is case-sensitive.

@piccolbo
Copy link

@piccolbo piccolbo commented Oct 17, 2018

@piccolbo
Copy link

@piccolbo piccolbo commented Oct 17, 2018

@cookieguru
Copy link
Contributor

@cookieguru cookieguru commented Oct 19, 2018

@AlexeyVolkov:

publicate.it

Doesn't seem to work, and is loaded with CORS errors

@kimadagem
Copy link

@kimadagem kimadagem commented Oct 19, 2018

@kazux, that worked for me. Thank you.

@kiwi810
Copy link

@kiwi810 kiwi810 commented Oct 23, 2018

@kazux that worked thanks, however now i think i see EVERY single post the user is posting even replies to other users rather than the ones he broadcasts on his/her own account.
Not that it bothers me but it kinda shows you what kind of person he/she really is when you see them reply rudely to other people on twitter lol wow im gobsmacked at the things ive been reading.

@garas
Copy link

@garas garas commented Oct 24, 2018

Until problem resolved, could you respond with single static RSS item like

<item>
  <title>Twitter are rate limiting requests from TwitRSS.me</title>
  <description>Twitter are rate limiting requests from TwitRSS.me, meaning it is effectively broken until I can think of a way round it. You can still run your own instance</description>
  <pubDate>Wed, 17 Oct 2018 00:00:00 +0000</pubDate>
  <link>https://github.com/ciderpunx/twitrssme/issues/59#issuecomment-430535935</link>
</item>

I didn't notice that feed is down, and Feedly didn't show sign until checked myself today.

ronilaukkarinen added a commit to ronilaukkarinen/khonsu that referenced this issue Oct 29, 2018
@p-ho
Copy link

@p-ho p-ho commented Oct 31, 2018

Would it be viable to fetch Twitter via proxies?
Just did a quick check: The two perl-scripts seem to run fine with torsocks for example (with increased timeout).

@jessicachinafile
Copy link

@jessicachinafile jessicachinafile commented Oct 31, 2018

@kazux @kiwi810 to avoid all the replies and retweets you can just add the following to the end of the url: +exclude:replies+exclude:retweets
The Bill Gates feed is not a good example since he doesn't appear to reply to people :) . But I tested it with other feeds and it appears to be working.

@kiwi810
Copy link

@kiwi810 kiwi810 commented Nov 5, 2018

@p-ho What format for the feed did you use with tor?

@jessicachinafile thanks will try that out

@p-ho
Copy link

@p-ho p-ho commented Nov 5, 2018

@kiwi810 What exactly do you mean by format?
As for the URLs I've just used the "straightforward" ones such as https://twitter.com/billgates

@ariddell
Copy link

@ariddell ariddell commented Nov 11, 2018

Perhaps a rotating proxy service could be persuaded to donate to twitrss? For example, https://scrapinghub.com/crawlera quotes 150K monthly requests at $25/mo.

@ciderpunx
Copy link
Owner

@ciderpunx ciderpunx commented May 30, 2019

Hi,

A small update. I've added a second script mobile_twitter_to_rss.pl that replicates the behaviour of twitter_user_to_rss.pl, except using the mobile version of the twitter site, which I don't think is rate limited.

There are a couple of limitations cf:
9f48b09

If people were able to help me by throwing some requests at it and seeing what breaks, that would be handy :-)

In the long run, I'll probably do some refactoring so that the choice of using the normal or mobile twitter version is passed as a param (mobile by default, normal if you need images and can deal with the reliability issues you might face).

@mhoran
Copy link
Contributor

@mhoran mhoran commented Jul 10, 2019

This didn't work for me at first, but after the changes in #74 things are a bit better.

@nivekalara237
Copy link

@nivekalara237 nivekalara237 commented Jul 11, 2019

@glye
Copy link

@glye glye commented Aug 6, 2019

@nivekalara237 I get internal server error with queries on that page.

@chovy
Copy link

@chovy chovy commented Apr 15, 2020

You need to use a proxy rotator. A good one is https://www.scraperapi.com/?fp_ref=anthony24

@ciderpunx
Copy link
Owner

@ciderpunx ciderpunx commented Apr 15, 2020

Thanks for the tip, at present it does not look like twitter are blocking by IP, so a proxy rotator isn't needed, but may well be helpful in future.

@ciderpunx ciderpunx closed this Apr 15, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.