Number of Extracted Tweets #106
Comments
|
Same problem here, sometimes it stops at just 10 tweets and sometimes it goes to end. It seems to be a problem with Twitter so I suggest you to try over time, this works for me. I don't see a specific pattern of the problem at every try so it's quite hard for me to dig further into the problem, but if you get this feel free to let us know! |
|
I am seeing the issue, I will look into this further |
|
Weird, Seems right now I cannot collect anything past 2018-05-14 |
|
So is. |
|
Okay, I've just fixed this in the recent commit. I'll update the PyPI later today after I add some more stuff |
|
I've tested with |
|
Thank you very much @haccer But should I make any changes? |
|
Did you update the repo? |
|
I uploaded changes to pypi, so |
|
True. Now that works. |
|
Again, with the new updates, I can not extract all the tweets. It only allows me to extract about 3000 tweets. |
|
Sorry to return to this topic. But I'm going to detail some more data, to see if it is possible to locate the error. It's curiosity that when I use the command Can you get all the tweets with the command |
|
Hi @DavidPerea Can you provide the user so I may take a better look? |
|
The problem is that there are many tweets, about 5900 tweets and you will have to wait a long time. I'm embarrassed that you have to wait so long.
Without |
|
Yes it is quite a long time lol @DavidPerea , I'll take a look later tonight or maybe this afternoon |
|
Of course, no problem. When you have time. I tell you better another user, AytoHuelva, who has fewer tweets, 4600. However, I only get 3115 tweets.
All this happens to me since the last update of the package. You already communicate to me whether it has been solved or not. Thank you very much!! |
|
I faced the same issue also |
|
@DavidPerea A quick thought, 3115 and 3175 are very close, perhaps my second method for grabbing retweets is limited to around that number; I haven't confirmed this yet, I will later though. Question: |
|
The strange thing is that before making the last update, I could get all the tweets with And excuse me @haccer , you could repeat the question with other words. I do not understand it well. |
|
Thinking well, you may be right @haccer . Because, I'm not sure if before with |
|
I am running the query but is quite large so I will let you know if it works as soon as it finishes |
|
I still face problems getting the tweets :( there are many left https://github.com/haccer/twint/issues/141#issuecomment-394861498 |
|
Okay I looked at this tonight. Number of Tweets advertised on the profile: 4,662 Test 1
Test 2
Test 3
I'm going to make some adjustments to the code tonight and tomorrow, we'll see if these numbers change |
|
Perfect, when you finish these adjustments, let me know and we will confirm if the numbers of the tweets change with respect to that test. As always thank you very much. |
|
An update @DavidPerea, I suspect this issue is very similar to #141, in which the reason the script is stopping early is because Twitter has hit a request limit and giving a 503 error... I'm rerunning the command with the requests being logged so I can see the last request when the script stops working. I should have more information about this tomorrow. |
|
Okay, it seems that a possible cause has already been found. Please, comment on the new information you have about this. |
|
Sorry for the delay, I've been quite busy the past week. tl;dr There's no way to resolve this, Twitter has imposed a limit.-- When requesting with the last ID there are no Tweets in the response. From my URL debug logs: I then constructed a
and it always receives this HTML response (no Tweets inside): ➜ twint-dev cat twint-last-request.log
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//WAPFORUM//DTD XHTML Mobile 1.1//EN" "http://www.openmobilealliance.org/tech/DTD/xhtml-mobile11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta name="HandheldFriendly" content="True" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=0"/>
<link rel="canonical" href="https://twitter.com/aytohuelva">
<meta name="twitter-redirect-url" content="twitter://user?screen_name=aytohuelva"/>
<meta name="twitter-redirect-srcs" content="{"pwreset-iphone":true,"android":true,"email":true}"/>
<link href="https://ma.twimg.com/twitter-mobile/8f3445bd0e5eb63b939e25a6ff29981d947a4a51/images/favicon.ico" rel="icon" type="image/x-icon" />
<title>Ayuntamiento Huelva (@AytoHuelva) on Twitter</title>
<link href="https://ma.twimg.com/twitter-mobile/8f3445bd0e5eb63b939e25a6ff29981d947a4a51/assets/a.css" inline="false" media="screen" rel="stylesheet" type="text/css" />
<script src="https://ma.twimg.com/twitter-mobile/8f3445bd0e5eb63b939e25a6ff29981d947a4a51/javascripts/framebust.js" type="text/javascript"></script>
<meta name="google-site-verification" content="V0yIS0Ec_o3Ii9KThrCoMCkwTYMMJ_JYx_RSaGhFYvw" />
<meta name="deciders" content="{"m2_mmw_scribe_get_url":true}" />
</head>
<body class="images nojs users-page users-show-page">
<div id="container">
<div id="brand_bar">
<table id="top">
<tr>
<td class="left">
<a href="/" class="brandmark">
<img alt="Twitter" height="28" src="https://ma.twimg.com/twitter-mobile/8f3445bd0e5eb63b939e25a6ff29981d947a4a51/images/sprites/larry_28px.gif">
</a>
</td>
<td class="right">
<img alt="|" class="divider" height="28" src="https://ma.twimg.com/twitter-mobile/8f3445bd0e5eb63b939e25a6ff29981d947a4a51/images/brandbar_divider.gif" />
<a class="search" href="#search">
<img alt="Search" height="28" src="https://ma.twimg.com/twitter-mobile/8f3445bd0e5eb63b939e25a6ff29981d947a4a51/images/sprites/search_28px.gif">
</a>
<img alt="|" class="divider" height="28" src="https://ma.twimg.com/twitter-mobile/8f3445bd0e5eb63b939e25a6ff29981d947a4a51/images/brandbar_divider.gif" />
<a class="signin" href="/session/new"><span>Log in</span></a>
<img alt="|" class="divider" height="28" src="https://ma.twimg.com/twitter-mobile/8f3445bd0e5eb63b939e25a6ff29981d947a4a51/images/brandbar_divider.gif" />
<a href="/signup"><span class="signup">Sign up</span></a>
</td>
</tr>
</table>
</div>
<div id="main_content">
<div class="profile">
<table class="profile-details">
<tr>
<td class="avatar">
<img alt="Ayuntamiento Huelva" src="https://pbs.twimg.com/profile_images/890511236533825536/5mgw8tfP_normal.jpg" />
</td>
<td class="user-info">
<div class="fullname">Ayuntamiento Huelva
</div>
<div class="username">
<span>@</span>
<span class="screen-name">AytoHuelva</span>
</div>
<div class="location">Huelva</div>
</td>
</tr>
<tr>
<td class="details" colspan="2">
<div class="bio">
<div class="dir-ltr" dir="ltr">
Cuenta oficial del Ayuntamiento de Huelva.
</div>
</div>
<div class="url">
<div class="dir-ltr">
<a href="http://t.co/paMajpJDSk" data-url="huelva.es" class="twitter-timeline-link activeLink dir-ltr tco-link"
dir="ltr" rel="nofollow" target="_blank">huelva.es</a>
</div>
</div>
</td>
</tr>
</table>
<table class="profile-stats">
<tr>
<td class="stat">
<div class="statnum">4,730</div>
<div class="statlabel"> Tweets </div>
</td>
<td class="stat">
<a href="/AytoHuelva/following">
<div class="statnum">333</div>
<div class="statlabel"> Following </div>
</a>
</td>
<td class="stat stat-last">
<a href="/AytoHuelva/followers">
<div class="statnum">5,724</div>
<div class="statlabel"> Followers </div>
</a>
</td>
</tr>
</table>
<div class="profile-actions">
<form action="/i/guest/follow/AytoHuelva" method="post">
<span class="m2-auth-token">
<input name="authenticity_token" type="hidden" value="259db82899551d8f4a7a49c4bd2243517335eb14"/>
</span>
<span class="w-button-common w-button">
<input name="commit" type="submit" value="Follow">
</span>
</form>
<form action="/AytoHuelva/actions" method="get">
<span class="w-button-common w-button">
<input name="commit" type="submit" value="•••">
</span>
</form>
</div>
</div>
<div class="w-mediaonebox">
<table>
<tr>
<td style="width: 73px;">
<a href="/AytoHuelva/media/grid?idx=0"><img src="https://pbs.twimg.com/media/DgDdTP9X4AE3v3O.jpg:thumb" width="73" height="78"/></a>
</td>
<td style="width: 55px;">
<a href="/AytoHuelva/media/grid?idx=1"><img src="https://pbs.twimg.com/media/DgDChdiXUAATyNi.jpg:thumb" width="55" height="78"/></a>
</td>
<td> </td>
</tr>
</table>
<div class="see-more">
<a href="/AytoHuelva/media/grid">View more photos</a>
</div>
</div>
</div>
<div id="footer">
<div class="search-fields">
<div class="title">
<label for="q">Enter a topic, @name, or fullname</label>
</div>
<form action="/search" class="search-input" method="get">
<table>
<tr>
<td class="value" id="search"><div><input id="q" name="q" type="text" value=""/></div></td>
<td class="button">
<input type="hidden" name="s" value="typd" />
<input type="image" src="https://ma.twimg.com/twitter-mobile/8f3445bd0e5eb63b939e25a6ff29981d947a4a51/images/sprites/magnifying_glass.gif" alt="Search"/>
</td>
</tr>
</table>
</form>
</div>
<table class="global-actions">
<tr>
<td><a href="/settings">Settings</a></td>
<td><a href="https://support.twitter.com/"> Help</a></td>
</tr>
</table>
<div class="view-actions"><a href="#top"> Back to top</a> · <a href="/settings/profile_images">Turn images off</a></div>
</div>
</div>
<script id="scribe-configuration" type="application/json">{"page":"profile"}</script>
<script src="https://ma.twimg.com/twitter-mobile/8f3445bd0e5eb63b939e25a6ff29981d947a4a51/assets/m2_tweets.js" type="text/javascript"></script>
<img src="/i/anonymize?data=%5B%7B%22integration%22%3A%22ga%22%2C%22ref%22%3A%22%22%2C%22mobileMetricsToken%22%3A%22152948269839453175%22%7D%5D" height="0" width="0" style="opacity: 0">
</body>
</html>In conclusion the limitation is: 160 Pages/Requests |
Initial Check
I have made sure to check the following.
[] Python version is 3.6.2
[] Using the latest version of Twint.
Command
I run this command to get the total of his published tweets.
python Twint.py -u malaga -o file.csv --csvDescription of Issue
I hope this question is not duplicated. I have reviewed the closed questions and I have not seen one related to this topic directly.
I intend to extract all the tweets from a user. However, I can not extract the total number of tweets that the user has published. Also every time I execute the code I get a different tweets number.
Why does this happen?
Why is it that the number of extracted tweets varies?
OS Details
I am using Windows10.
The text was updated successfully, but these errors were encountered: