Twint stopped working out of the blue #918

fsiep · 2020-09-19T14:53:56Z

Have been using python 3.6 scripts for the last 3 months, using twint. Yesterday, I received a 404 with one of the scripts and with the other I am seeing no progress, just a countdown that indicates that the execution has been stopped.

robert-moore · 2020-09-19T16:58:45Z

Same here. It stopped 17 Sept around 8pm EST for me.

twint.run:Twint:Feed:noDataExpecting value:

See this thread: #915

dansar39 · 2020-09-20T00:10:04Z

Habra posibilidad de solucionarlo? Twiter cambio algo?

dansar39 · 2020-09-20T00:11:02Z

Conocen otro parecido para reemplazarlo, para escritorio

lukaspistelak · 2020-09-20T09:05:12Z

same here 👍

error search  Expecting value: line 1 column 1 (char 0) 0
Expecting value: line 1 column 1 (char 0) [x] run.Feed[!] if get this error but you know for sure that more tweets exist, please open an issue and we will investigate it!CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 1.0 secs
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 8.0 secs
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 27.0 secs

osamaabdalla · 2020-09-20T09:22:23Z

same here:

CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 1.0 secs
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 8.0 secs
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 27.0 secs
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 64.0 secs

any solutions !

dansar39 · 2020-09-20T13:24:22Z

Habra alguna solucion?

…

________________________________ De: lukaspistelak <notifications@github.com> Enviado: domingo, 20 de septiembre de 2020 9:05 Para: twintproject/twint <twint@noreply.github.com> Cc: dansar39 <dansar39@hotmail.com>; Comment <comment@noreply.github.com> Asunto: Re: [twintproject/twint] Twint stopped working out of the blue (#918) same here 👍 error search Expecting value: line 1 column 1 (char 0) 0 Expecting value: line 1 column 1 (char 0) [x] run.Feed[!] if get this error but you know for sure that more tweets exist, please open an issue and we will investigate it!CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) sleeping for 1.0 secs CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) sleeping for 8.0 secs CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) sleeping for 27.0 secs — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#918 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/APYWXU67CHXK3AIA5TGY6RDSGXAVJANCNFSM4RTB7HHA>.

adulau · 2020-09-20T14:11:51Z

Twitter just killed the legacy without login and JS enabled.

akuma527 · 2020-09-20T14:45:10Z

Twitter just killed the legacy without login and JS enabled.

Hi @adulau , can we get this working again?

adulau · 2020-09-20T16:32:26Z

For my understanding, the only option is to go for a complete instrumentation (Selenium) of the browser to support JS but it also means using logged accounts. It would be major rewrite of the project.

zhouhanc · 2020-09-20T18:27:42Z

same here!

guidovranken · 2020-09-20T21:41:45Z

For my understanding, the only option is to go for a complete instrumentation (Selenium) of the browser to support JS but it also means using logged accounts. It would be major rewrite of the project.

https://github.com/JustAnotherArchivist/snscrape still works just fine. It relies on HTML parsing using BeautifulSoup, but Selenium/JavaScript is not needed.

Use snscrape --jsonl twitter-user <user> to get very similar results to twint --json.

I'm not really familiar with the internals of either project, but implementing in twint whatever snscrape is doing should get us going again. @pielco11

zhixideyu · 2020-09-21T03:57:55Z

据我了解，唯一的选择是使用浏览器的完整工具（Selenium）以支持JS，但这也意味着使用登录帐户。这将是该项目的主要重写。

https://github.com/JustAnotherArchivist/snscrape仍然可以正常工作。它依赖于使用BeautifulSoup的HTML解析，但是不需要Selenium / JavaScript。

使用snscrape --jsonl twitter-user <user>变得非常相似的结果twint --json。

我对这两个项目的内部都不是很熟悉，但是无论snscrape正在做什么，都可以用twint来实现，这应该会让我们重新开始。@ pielco11

can get a tweet link, but how do you get the text content

lukaspistelak · 2020-09-21T08:44:53Z

https://github.com/mherrmann/selenium-python-helium

akuma527 · 2020-09-21T10:52:33Z

https://github.com/mherrmann/selenium-python-helium

This becomes unresponsive when I login into twitter with my user and pass. Is this happening to anyone else?

arghzero · 2020-09-22T00:21:23Z

@guidovranken @zhixideyu and anyone else looking for an alternative, looks like the --jsonl flag was added to snscrape quite recently, it's present in the github source but the version on pypi doesn't have it yet.

To get tweets in json using snscrape (including their text content) do this:

pip install --upgrade git+https://github.com/JustAnotherArchivist/snscrape@master
snscrape --jsonl twitter-user <twitter-handle-without-@-sign>

You'll get back json like this:

{
  "url": "https://twitter.com/textfiles/status/1308158822922493952",
  "date": "2020-09-21T21:39:02+00:00",
  "content": "@Sierra_OffLine Just for the record, this person is now an \"Afterlife Coach\" who writes books about how your pets will enjoy the afterlife https://t.co/GK3qnk9Pf2",
  "id": 1308158822922494000,
  "username": "textfiles",
  "outlinks": [],
  "outlinksss": "",
  "tcooutlinks": [],
  "tcooutlinksss": "",
  "retweetedTweet": null
}

@adulau can whatever method snscrape uses to get tweets be ported into twint?

zhixideyu · 2020-09-22T02:37:03Z

snscrape --jsonl twitter-user <twitter-handle-without-@-sign>

Hello, there are still some mistakes after I install it according to what you said. I hope you can introduce it clearly

zhixideyu · 2020-09-22T02:42:24Z

@guidovranken @zhixideyu和其他正在寻找替代方法的人，似乎该--jsonl标志是最近才添加到snscrape的，它存在于github源码中，但是pypi上的版本还没有它。

要使用snscrape（包括其文本内容）在json中获取推文，请执行以下操作：
pip install --upgrade git+https://github.com/JustAnotherArchivist/snscrape@master
snscrape --jsonl twitter-user <twitter-handle-without-@-sign>
您将像这样返回json：
{
   “ url ”：“ https://twitter.com/textfiles/status/1308158822922493952 ”，
   “ date ”：“ 2020-09-21T21：39：02 + 00：00 ”，
   “ content ”：“ @Sierra_OffLine仅用于根据记录，此人现在是“来世教练”，他写有关您的宠物如何享受来世的书https://t.co/GK3qnk9Pf2 “，
   ” id “：1308158822922494000，
   ”用户名“：”textfiles “，
   ” outlinks “：[]，
   ” outlinksss “：” “，
   ” tcooutlinks “：[]，
   ” tcooutlinksss “：” “，
   ” retweetedTweet “：null 
}
@adulau可以将snscrape用于获取tweet的任何方法移植到twint吗？

AttributeError: module 'functools' has no attribute 'cached_property'

arghzero · 2020-09-22T02:43:59Z

that looks like an issue with your python install. Above works for me on python 3.8

zhixideyu · 2020-09-22T02:45:01Z

that looks like an issue with your python install. Above works for me on python 3.8

ok I'm going to install 3.8

zhixideyu · 2020-09-22T04:07:27Z

看起来像您的python安装有问题。以上对我适用于python 3.8

好的，我要安装3.8

It works, but how can you run a script instead of a command line

fsiep · 2020-09-22T17:10:22Z

The 404 error seems to be triggered by Twitter finger-printing the request. The URL works in various browsers. A non-Javascript version of twitter is available through https://mobile.twitter.com. The link works without login.

rennanharo · 2020-09-22T18:26:26Z

It seems to be a generalized problem with Twitter scrapers. The GetOldTweets3 package also stopped working after the changes Twitter made to the endpoint.

dansar39 · 2020-09-22T18:54:06Z

python >>y aqui el script

Saludos

…

________________________________ De: 窒息得鱼 <notifications@github.com> Enviado: martes, 22 de septiembre de 2020 4:07 Para: twintproject/twint <twint@noreply.github.com> Cc: dansar39 <dansar39@hotmail.com>; Comment <comment@noreply.github.com> Asunto: Re: [twintproject/twint] Twint stopped working out of the blue (#918) 看起来像您的python安装有问题。以上对我适用于python 3.8 好的，我要安装3.8 It works, but how can you run a script instead of a command line ― You are receiving this because you commented. Reply to this email directly, view it on GitHub<#918 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/APYWXU6CJAMJRSROG42DNTLSHAPIZANCNFSM4RTB7HHA>.

elamje · 2020-10-01T23:51:46Z

Just to be absolutely clear - After reading the thread above, people seem to be saying everything but the project is now in the graveyard.

@adulau Is it safe to say this project is now in archive due to the change in Twitter's API?

himanshudabas · 2020-10-09T12:18:19Z

For my understanding, the only option is to go for a complete instrumentation (Selenium) of the browser to support JS but it also means using logged accounts. It would be major rewrite of the project.

I have made a PR #944, it uses new endpoints of twitter. There's no need to use Selenium or a JS enabled browser/headless browser. A lot of stuff might still be broken though.

irisdemented · 2020-10-09T18:36:51Z

For my understanding, the only option is to go for a complete instrumentation (Selenium) of the browser to support JS but it also means using logged accounts. It would be major rewrite of the project.

I have made a PR #944, it uses new endpoints of twitter. There's no need to use Selenium or a JS enabled browser/headless browser. A lot of stuff might still be broken though.

@himanshudabas firstly, thank you for your work on this! i started fresh and downloaded the twint with your fix added, but continue to get can't find '__main__' module in 'twint' when I run searches.. I'm not sure where to start troubleshooting. have tried in both 3.6 and 3.6.8. Any ideas?

himanshudabas · 2020-10-10T03:24:08Z

@irisdemented did you install it locally using pip install -e .
I'll lay down the steps.
I recommend using virtual environment.

Create a virtual environment with python 3.6.x
Activate this virtual environment and cd to the twint directory. (Which contains setup.py
Run pip install -e . (this will install the twint locally in your virtual environment)
Now you can create a .py file in you system anywhere and can simply import twint (but remember, your virtual env must be active to import this twint installation.)

Let me know if you get it working.
Note : there might be a missing dataclasses error so simply pip install dataclasses in your virtual env to fix that.

irisdemented · 2020-10-15T19:30:47Z

@himanshudabas extremely happy to report this is working!

once i figured out i needed to use it as module and not at the command line, everything clicked into place.

thanks much for your work and help.

milad14 · 2020-10-22T15:06:31Z

@himanshudabas thanks for your work
I can not retrieve tweets when I configure the user and search item concurrently for example this:

c = twint.Config()
c.Username = "realDonaldTrump"
c.Search = "great"
twint.run.Search(c)

but they work when they are apart

himanshudabas · 2020-10-22T15:09:04Z

@milad14 please check #973

milad14 · 2020-10-22T15:56:39Z

@milad14 please check #973

thanks a lot . it is solved

certrik mentioned this issue Sep 21, 2020

Seems Twitter is broken ;) ail-project/ail-feeder-twitter#3

Closed

MartinBeckUT mentioned this issue Sep 21, 2020

HTTP Error, Gives 404 but the URL is working Mottl/GetOldTweets3#98

Open

mjh-sakh mentioned this issue Sep 25, 2020

CRITICAL:root:twint.get:User:'NoneType' object has no attribute 'find' #925

Open

akshaykalucha mentioned this issue Sep 25, 2020

consistent sleeping for a long time and no response #926

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Twint stopped working out of the blue #918

Twint stopped working out of the blue #918

fsiep commented Sep 19, 2020

robert-moore commented Sep 19, 2020 •

edited

dansar39 commented Sep 20, 2020

dansar39 commented Sep 20, 2020

lukaspistelak commented Sep 20, 2020

osamaabdalla commented Sep 20, 2020 •

edited

dansar39 commented Sep 20, 2020 via email

adulau commented Sep 20, 2020

akuma527 commented Sep 20, 2020

adulau commented Sep 20, 2020

zhouhanc commented Sep 20, 2020

guidovranken commented Sep 20, 2020

zhixideyu commented Sep 21, 2020

lukaspistelak commented Sep 21, 2020

akuma527 commented Sep 21, 2020

arghzero commented Sep 22, 2020 •

edited

zhixideyu commented Sep 22, 2020

zhixideyu commented Sep 22, 2020

arghzero commented Sep 22, 2020

zhixideyu commented Sep 22, 2020

zhixideyu commented Sep 22, 2020

fsiep commented Sep 22, 2020

rennanharo commented Sep 22, 2020

dansar39 commented Sep 22, 2020 via email

elamje commented Oct 1, 2020

himanshudabas commented Oct 9, 2020

irisdemented commented Oct 9, 2020

himanshudabas commented Oct 10, 2020

irisdemented commented Oct 15, 2020

milad14 commented Oct 22, 2020 •

edited

himanshudabas commented Oct 22, 2020

milad14 commented Oct 22, 2020

Twint stopped working out of the blue #918

Twint stopped working out of the blue #918

Comments

fsiep commented Sep 19, 2020

robert-moore commented Sep 19, 2020 • edited

dansar39 commented Sep 20, 2020

dansar39 commented Sep 20, 2020

lukaspistelak commented Sep 20, 2020

osamaabdalla commented Sep 20, 2020 • edited

dansar39 commented Sep 20, 2020 via email

adulau commented Sep 20, 2020

akuma527 commented Sep 20, 2020

adulau commented Sep 20, 2020

zhouhanc commented Sep 20, 2020

guidovranken commented Sep 20, 2020

zhixideyu commented Sep 21, 2020

lukaspistelak commented Sep 21, 2020

akuma527 commented Sep 21, 2020

arghzero commented Sep 22, 2020 • edited

zhixideyu commented Sep 22, 2020

zhixideyu commented Sep 22, 2020

arghzero commented Sep 22, 2020

zhixideyu commented Sep 22, 2020

zhixideyu commented Sep 22, 2020

fsiep commented Sep 22, 2020

rennanharo commented Sep 22, 2020

dansar39 commented Sep 22, 2020 via email

elamje commented Oct 1, 2020

himanshudabas commented Oct 9, 2020

irisdemented commented Oct 9, 2020

himanshudabas commented Oct 10, 2020

irisdemented commented Oct 15, 2020

milad14 commented Oct 22, 2020 • edited

himanshudabas commented Oct 22, 2020

milad14 commented Oct 22, 2020

robert-moore commented Sep 19, 2020 •

edited

osamaabdalla commented Sep 20, 2020 •

edited

arghzero commented Sep 22, 2020 •

edited

milad14 commented Oct 22, 2020 •

edited