Error json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) (again...)

[eliottdbs]

JSONDecodeError

Initial Check

• Python version is 3.6.9;
• Updated Twint with pip3 install --user --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint;
• I have searched the issues and there ARE duplicates of this issue but they're old
  • json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) #426
  • Fix UserAgent issue #427
  • Added fake-useragent package for RandomUserAgent. #428

Command Ran

I execute a simple test to collect user's tweets and profile :

import twint

c = twint.Config()

c.Username = "USERNAME"
c.Count = True
c.Debug = True

twint.run.Profile(c)
twint.run.Lookup(c)

Description of Issue

After several days using twint without error, this error came up yesterday, I checked the existing issues (mentionned up there ^^) so it's might also be related to user-agent but I don't understand how user-agent can block the request.

Below my traceback:

Traceback (most recent call last):
  File "PATH_A\test.py", line 19, in <module>
    twint.run.Profile(c)
  File "PATH_A\twint\run.py", line 467, in Profile
    run(config)
  File "PATH_A\twint\run.py", line 398, in run
    get_event_loop().run_until_complete(Twint(config).main(callback))
  File "PATH_B\AppData\Local\Programs\Python\Python39\lib\asyncio\base_events.py", line 642, in run_until_complete
    return future.result()
  File "PATH_A\twint\run.py", line 285, in main
    await task
  File "PATH_A\twint\run.py", line 342, in run
    await self.profile()
  File "PATH_A\twint\run.py", line 260, in profile
    await self.Feed()
  File "PATH_A\twint\run.py", line 76, in Feed
    response = await get.RequestUrl(self.config, self.init)
  File "PATH_A\twint\get.py", line 141, in RequestUrl
    response = await Request(
  File "PATH_A\twint\get.py", line 177, in Request
    return await Response(session, _url, params)
  File "PATH_A\twint\get.py", line 190, in Response
    raise TokenExpiryException(loads(resp)["errors"][0]["message"])
  File "PATH_B\AppData\Local\Programs\Python\Python39\lib\json\__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "PATH_B\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "PATH_B\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I've checked the user-agent used and it changed as it is supposed to since #427 #428 random use.

Are we supposed to change the user_agent_list in get.py once in while ?

I remain at your disposal if any information is missing.

Thanks for helping.
Have a nice day 🙂

Environment Details

I'm running the script on Windows 10 Pro, in a VSCode bash terminal.

[Abdulmajeed95-zz]

same issue here

[rapkido]

I tried to change to the user agent list but that doesn't fix it, it may be because of API Token?

[eliottdbs]

I just realized that a TokenExpiryException is raised.
It is related to response code 429 Too many requests i.e. Rate Limit Exceeded as it's said in get.py.
Maybe I've just reached the rate limit for the month ?
Any information on rate limit ?

[rapkido]

async def Username(_id, bearer_token, guest_token):
    logme.debug(__name__ + ':Username')
    _dct = {'userId': _id, 'withHighlightedLabel': False}
    _url = "https://api.twitter.com/graphql/I5nvpI91ljifos1Y3Lltyg/UserByRestId?variables={}".format(dict_to_url(_dct))
    _headers = {
        'authorization': bearer_token,
        'x-guest-token': guest_token,
    }
    r = await Request(_url, headers=_headers)
    print(r)
    j_r = loads(r)
    username = j_r['data']['user']['legacy']['screen_name']
    return username

I think same with you it about the Token in _url variable

[rapkido]

I just realized that a TokenExpiryException is raised. It is related to response code 429 Too many requests i.e. Rate Limit Exceeded as it's said in get.py. Maybe I've just reached the rate limit for the month ? Any information on rate limit ?

https://developer.twitter.com/en/docs/twitter-api/rate-limits

[panichmaxim]

Any solutions?

[jtomas55]

I was checking apparently twitter doesn't let you extract information, the RT record is not available unless you log in

[peter4432]

Not sure if it is helpful in diagnosing the root cause, but when using twint via windows cmd line, error only occurs when the --timeline parameter is used e.g:
twint -u -o "Scrap.tsv" --csv --since "2022-01-01" => Expected output
twint -u -o "Scrap.tsv" --csv --timeline --since "2022-01-01" => Error and no output

Environment Details
I'm running the command on Windows 10 Pro, in a non-admin command prompt

[jtomas55]

#1335 (comment)

when you put this --timeline clause, it scrapes the user's own tweets plus the RTs they gave, since the RTs can't be extracted anymore, it gives the error

[irisdemented]

@panichmaxim When i switched to twint.run.Search(c) instead of twint.run.Profile(c) it started working again.

[jtomas55]

twint.run.Search(c) it works but you don't have the same data that you had with twint.run.Profile(c), if you look closely it only brings the tweets of that user twint.run.Search(c) not the RTs that he gave to other Users

[soccerqi]

In my environment, function Lookup is working correctly, but Profile is not. So I use Lookup instead of Profile to get user infomation. Hope to help you.

[minamotorin]

I wrote the code to solve the issue. minamotorin/twint minamotorin/twint

In this code, twint.run.Profile will work.
But twint.run.db will not work because the fix is incomplete.
So I have not created a PR.

This problem was caused by Twitter's changes.
Maybe this is a time for giving up on using twint which has not been updated for a long time.

[eliottdbs]

After some time, I've tested your code @minamotorin, it works, many thanks for your work (minamotorin/twint).

For others, as said this issue is not totally fixed, but it solves the original problem for now.