looks like X/twitter(?) broke something again #983
Comments
broke again zedeus/nitter#983
|
Also, the syndication api for |
|
Yes, it is not working now. I hope the Nitter people fix this soon. |
|
Is there a online/CLI tool converting |
Not really. showReplies=false shows years-old content when not logged in. |
|
Down again... |
That's because in that specific example, those tweets were years ago. Look again at the like count, notice anything? |
We can just search for the first |
Indeed #!/usr/bin/python3
import requests
import re
import urllib
url = "https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk"
with urllib.request.urlopen(url) as response:
encoding = response.info().get_param('charset', 'utf8')
html = response.read().decode(encoding)
result = re.search('script id="__NEXT_DATA__" type="application\/json">([^>]*)<\/script>', html)[1]
print(result) |
Interesting, but this doesn't return RSS with 'item', 'pubDate' etc. tags. Maybe a script using https://github.com/lkiesow/python-feedgen would do the job? |
Not sure I understand ? It expose far more informations than needed and it does expose the date and all Here's an example for one tweet only : {
"type": "tweet",
"entry_id": "tweet-1519480761749016577",
"sort_index": "1691455400412446720",
"content": {
"tweet": {
"id": 0,
"location": "",
"conversation_id_str": "1519480761749016577",
"created_at": "Thu Apr 28 00:56:58 +0000 2022",
"display_text_range": [
0,
52
],
"entities": {
"user_mentions": [],
"urls": [],
"hashtags": [],
"symbols": [],
"media": []
},
"favorite_count": 4600599,
"favorited": false,
"full_text": "Next I’m buying Coca-Cola to put the cocaine back in",
"id_str": "1519480761749016577",
"lang": "en",
"permalink": "/elonmusk/status/1519480761749016577",
"possibly_sensitive": false,
"quote_count": 171975,
"reply_count": 187438,
"retweet_count": 649833,
"retweeted": false,
"text": "Next I’m buying Coca-Cola to put the cocaine back in",
"user": {
"blocking": false,
"created_at": "Tue Jun 02 20:12:29 +0000 2009",
"default_profile": false,
"default_profile_image": false,
"description": "Blades of Glory",
"entities": {
"description": {
"urls": []
},
"url": {}
},
"fast_followers_count": 0,
"favourites_count": 30569,
"follow_request_sent": false,
"followed_by": false,
"followers_count": 153112066,
"following": false,
"friends_count": 410,
"has_custom_timelines": false,
"highlightedLabel": {
"badge": {
"url": "https://pbs.twimg.com/profile_images/1683899100922511378/5lY42eHs_bigger.jpg"
},
"description": "X",
"userLabelType": "BusinessLabel",
"userLabelDisplayType": "Badge"
},
"id": 0,
"id_str": "44196397",
"is_translator": false,
"listed_count": 126597,
"location": "𝕏Ð",
"media_count": 1659,
"name": "Elon Musk",
"normal_followers_count": 153112066,
"notifications": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/44196397/1690621312",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1683325380441128960/yRsRRjGO_normal.jpg",
"protected": false,
"screen_name": "elonmusk",
"show_all_inline_media": false,
"statuses_count": 29441,
"time_zone": "",
"translator_type": "none",
"url": "",
"utc_offset": 0,
"verified": false,
"withheld_in_countries": [],
"withheld_scope": "",
"is_blue_verified": true
}
}
}
},EDIT : Maybe you meant a directly usable solution for an end user, and of course it's not, the snippet need to be adapted by a dev. |
Ok, thank you, I'll try this. |
calling the syndication URL without being logged in twitter doesn't retrieve the most recent tweets. If I call this url in postman, I retrieve 100 tweets from 10/19/2018 to 07/31/2023; no tweets from august... |
It retrieves the tweets with the highest like count from that user, which doesnt sound good if your goal is retrieving the most recent tweets, as there's no guarantee new tweets will make it to the top 100 tweets from that user. And even if they did, it might take a considerable amount of time |
|
I've noticed that for smaller accounts that have less than 100 tweets, that syndication URL does not load any tweets. |
No. That is the case for all big accounts. I am interested in the most recent Tweets and this approach will lead to nothing. |
im using this for my bot and it working fine with cookies and headers. |
Yes. And there is at least one tweet from August with more likes (>807K) than some older tweets which are included (e.g. <680K). |
|
is there any forecast for solving this problem? |
|
Looks like https://nitter.privacydev.net/ is working |
|
That one is a fork which uses account credentials. See #830 |
|
I am aware but couldn't nitter implement a system that aurora uses with lots of accounts that rotate per user? |
That's hard to maintain and simple for twitter to ban by just filtering "if number of accounts per IP > SOME_CONSTANT: ban all of them" |
User feeds not working on this |
|
I switched to the privacydevel fork, credentials in but its still 404ing the same endpoint upstream is having problems with |
|
Strange. privacydev (without credentials) works more or less for @ElonMusk, but not for other users like for instance @BarackObama. |
try this: #1178 |
Avoid expiring the tokens for now. See: - zedeus#983 (comment) - zedeus#1155 (comment) Thanks @cmj
Avoid expiring the tokens for now. See: - #983 (comment) - #1155 (comment) Thanks @cmj
|
@somini I coincidentally merged that right before you commented |
Oddly, I can only access your instance through a VPN, despite never having used it before. Do you block certain country IPs or dynamic IPs? Or is something wrong on my end? |
if three or more IPs in a /24 range have been blocked by fail2ban the entire /24 will get blocked. you can email the first two octets of your ip to graf at poast.org and i will look but that's more than likely what it is |
Uh... That's far too advanced for me, I'm afraid... I don't even know what a "24 range" is... Should I email you my current IP? |
|
that is what i was asking for, yes |
Ah, "Graf" is your username! Still, I'm too dumb and couldn't find a way to contact you directly, so I just sent an email to Poast support. |
|
If we have reached the point where instances have to carefully ration requests: @zedeus how hard would it be to implement caching/database storage of Tweets? Most likely a significant portion of requests are for the same (popular) Tweets over and over again. If instances store or cache Tweets, they don't have to spend their request budget getting the same Tweets repeatedly. Yes I did check the other issues, the closest I could find is #288. Even more useful would be a collaborative or shared cache used by multiple Nitter instances. This would effectively pool their accounts and request budgets, without disadvantaging any instance operator or requiring them to sacrifice request budget to help others. (EDIT: nevermind, #1188 discusses this) |
|
I have been told some instances are caching certain users but I've not been told how yet, so I would be for something like this. maybe 2-3 requests in a short time would suffice to trigger some kind of caching. you could probably do it with nginx but im not sure how you would cache the data returned that way. I've been caching twitter media on a 3h expiry and it's helped a lot |
|
@animegrafmays Yeah, it seems like caching was once done more extensively but reduced for performance reasons (unfortunately I can't find the issue anymore). Now since it's a choice between slow loading and not loading, even a very large and not very performant cache is better than being unable to load Tweets. |
|
This might also help a lot in dealing with scrapers -- they are probably some of the worst offenders in requesting the same major accounts. |
|
Twstalker is doing funny things: when someone quotes a tweet, the quoted tweet repeats what the quoter says. |
|
@animegrafmays Hey, how/where can I contact you to whitelist my IP for nitter.poast.org :) ? |
|
im not in the market of 'whitelisting' IPs. you can email me at graf @ poast.org and I can see why yours is blocked, but I'm not whitelisting people |
@animegrafmays Mine is blocked too, maybe because I use the RSS feeds for some accounts (~ 10). I set them to fetch new posts once per hour on my browsers, instead of every 30 minutes. Before I e-mail you, could you confirm that it could have been a cause of “ERR_CONNECTION_REFUSED”? |
|
this github issue is not an issue tracker for your connection issues to my nitter instance. please stop shitting up this github issue by triggering an email every single time you respond and instead use the contact methods i've presented in this thread already |
I have never had an issue with the RSS feeds lately, I get them fine with Power Automate and Element Feeds. Fully agree with the fact we are clogging up but it's not like I can move this to fediverse (the servers I am on block poa.st because of allowing some undesirable people that were on certain platforms that's all). |
The solution to all your problems is to host these services yourself. One of the common SBC or an old laptop - screen not required - is more than enough to run an instance of Pleroma (for your fediverse needs), Nitter, Libreddit, Invidious (Youtube proxy) and even Peertube if you go easy on the transcodes. That way you are neither a burden to others nor beholden to them. You may get blocked by the 'purple-hair crowd' on Mastodon but that is more or less unavoidable unless you want to become part of that crowd yourself. As long as you behave sensibly - and why wouldn't you? - you should be able to federate with the world and its dog, minus the aforementioned block-happy group. Of course all this assumes you either have a stable internet connection at home to which you can attach said SBC or laptop or thin client or what have you or can splurge a few |
|
For those of us with no landline, a need to self-host a service simply to access Twitter content is the same as Twitter not being usable.
|
Wow, amazing information. If I was able to, I would have done so already. I have no experience with hosting stuff and there's no chance of me running up a cable to connect a device. Using a VPS seems expensive too, a local company wants £3 for their lowest tier a month. @lukefromdc how is a landline related to this? |
If you only would use the machine for TwiXXer->Nitter, yes that would be overkill. The idea would be to use your own host for much more than that both to avoid the mentioned blocking problems as well as to host your own services. Host your own photos instead of giving them to Google, host your own music instead of being dependent on streaming services, host your own XMPP server instead of being dependent on Telegram/Whatsapp/Signal/Matrix or ( |
|
Landline relevence is much more bandwidth available. I have de facto (not admitted to) capped cellular bandwidth via tethering only, when I am on the road it would be out of range of any home server. Forget a static IP address of course. Also they don't like tethering but blocking sophisticated ways to tether brings lots of false positiives so they don't bother. If they do I get a new carrier and a new phone number. No stable IP address, no continuously on connection in any single location, not going to carry a server everywhere I go and we are not discussing the presumably lighter load of distributed hosting (e.g diaspora) here. I do not however use Google or Meta for any purpose as I oppose their business models. I publish videos on Mastodon and on archive.org (latter good for linking especially as they want copies of all online content anyway). Twitter used to be widely used in my community so I had to be able to keyword search it and pull up known useful timelines, now neither of those work on the Twitter webapp without an account. I refuse to make an account with any ad supported corporate social media site, but the importance of Twitter is plunging as people stop posting there. Unfortunately this is fueling a migration to Instagram which is even worse than Twitter but at least still has a partially working 3ed party backend. You must know at least the username whose content you want to see to use it as far as I can tell. On the plus side, the migration to Mastodon has greatly boosted the reach of my content, but some events I would cover now go unseen as they are organized in places I am now blind to. |
Ah that makes sense. Providers here generally don't require a landline but one provider still only does ADSL service and the other is HFC. |
|
so people stop generating multiple notifications via here and email for stuff that isn't related to nitter development, if it pertains to my nitter instance please email graf at poast dot org. do not post it here. want to be critical of me? email it. have a problem with what im doing? email it. leave this thread for dev. thanks |
and others
The text was updated successfully, but these errors were encountered: