json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) #7
Comments
|
Ronnie, I was not able to reproduce this issue yet. I believed the thread "629006352329760768" was the one with 127,555 messages but it seems it was another thread. Does the thread "629006352329760768" was completely downloaded the first time with its 49,899 messages (count from your previous post here)? Could you try again with only the "-di" switch? Like this: |
|
The original thread with the 629006352329760768 with the ALL threads pull came up with 49,899 messages dating back to May 2016. when i ran it on it's own with that command line you sent me this morning, it pulled the 127,555 messages dating back to August 2015 when the conversation started. So it's perfect. And the images and such were downloaded into all the separate folders as you explained. So there are two specific message thread conversations that I wanted to back up individually that were not captured for some reason when i ran the full archive script, so I inserted the conversationID into the command you gave me and it started compiling messages, got up to about 3000 and then produced the error above. Thanks |
|
i went in to the developer resources and saw two alert icons and did screen shots of them in case this helps.
|
|
Reminder I'm tech illiterate about this coding, so please excuse my inexperience. Ronnie |
|
Ok, so let me sum up to check if I'm correct.
I think I may have an idea why all the threads have not been downloaded the first time. I've counted all the threads in your previous message and found exactly 50 conversations. Currently, to find "all" the conversation IDs, the script loads the conversations available on the "first" "Messages" page but do not simulate scrolling to load more. I though that all the conversations were listed directly. My guess is that when you scroll down through all the conversations, at the bottom, Twitter loads the next 50 conversations. I did not identify this case because I have a lot less than 50 conversations on Twitter! But it's an interesting case and I'm going to open a new ticket to improve the "all threads" mode which is in fact a "latest 50 conversations" mode it seems. Kudos for finding this bug! For the error in the thread 655882638012518400, it seems to be parsing problem in the thread itself. Have you already tried to run the command again? I've added a "debug" mode since to find out the tweet with issues but it's not in the current version. However, I think it will be the only possibility to find the "special" tweet which make the tool fail. I will keep you posted once I make the updated version available. It's complicated for me to create Mac versions because I do not own a Mac. |
|
Yes that is exactly correct. I will see if I can scroll through my DMs and see how many conversation threads I do in fact have. I will also try to run the command again on the one with error. you are incredible Julien, thanks so much!! Ronnie |
|
Going to paste the list as best i can pulling from my DM list. #TeamErin Zoey the Bulldog + 16 Nasty Women Lucy @BulldawgLucy Gus @stephbump Atlanta Trip Matilda @BulldogMatilda Monroe da Handsum @MonroeBulldog EmmyRose (G💗's sis) @gloveritchey Ham, Earl of Sammich @HamBulldog amc9000 @amc5848 Bubbles the Bulldog @ErinVanRyn sam dellinger @ArtisticBulldog Sammy @sammythebulldog Stella Davis @PpawDavis Princess Meyda @MeydaTheBoston Allyson Morris @AllysonMorris 100%ChanceOfSexy @MamaTookOne Lulu & Zeus + 11 London Bulldogs @thebumblers Rugrat @nottaslimjim Bruce @danilordbraga Bunty Watson @thepugsmummy LuLu & Joey @luvpug25 ❤️ RIP Sweet Glover @warner_bear Margot 🐶💖 @MargotTheBully SammydaBulldog @SammydaBulldog Barbara Tiesi @BTiesi PudgeMan 4 Georgia❤️ @Pudgeman901 Lynyrd @lynyrdsbackyard realSteveSchindler @ElectroGeezer Melanie /Ex-#GOP @Lonestarmomcom Bulldozer Bawss @BulldozerBawss Something To Chew On @something2chew Lord Nelson & Jones @LordNelson2007 Ozzy @OzzySharpe Les @lpmoorerocks deuce and tooie. @deuceandtooie deuce @Deuce4Mayor Mandy @MandyMcWilli Beckett the Bulldog @LadyBeckett Liz Bee Gee @LizBeeGee Mr.Chips @dugan_fahey Humphrey the Pug 🦃, Momz_and_Mooch Momz_and_Mooch @adaalling816 Henry Ford @HenryFordBD Sadie miss Denali @3phibotticelli Fenway ❤️ Georgia @Lil_Fen Cleopatra Eng Bully @Catsemail SIDECAR SAVANNAH® @SavannahSidecar Dawn Keller @Petunia4Disney Bogart @LizS76 Zoey the Bulldog @ZoeytheBulldog Storm #love4georgia, Mandy RockyTheDogg Mommsie @RockyTheDogg Whopper'sMom @besamemlr Sugar @RileyDevilDog Jiro @Tiny__boy Jethro The Bulldog @BulldogJethro Churchill, Otis Cooper Fartacus @stinkyshmoo Storm #love4georgia @ebucher1204 Benson'sMomma♡ @itsbensontime Otis @otisbulldog MAGOO @MagooCrew Jodi Pepper @jpep530 Cara Hergenroether @clhergenroether Gracie Sunshine @GracieSunshine1 Guido RIP Georgia ❤️ @GuidoLock Lucy Belle @PrincessLucyB ~ LORRAINE ~ @ismaelricardo22 LudKee @lu_dkee Jane Bloom @oceanjane13 Otis, Cooper Fartacus Cara Hergenroether, Erin Lundmark LOUIS.J @LOUISJPUG Nancy Thornton Churchill, Gracie Sunshine Jane Bloom + 13 D @dougbulldog Jeff Musk @JeffMusk AUNTBEA53 @AuntBEA53 Toby Bulldog @TobyBully Melanie /Ex-#GOP + 12 Melanie /Ex-#GOP + 3 Naughty Gossip @NaughtyNiceRob Melanie /Ex-#GOP, EmmyRose (G💗's sis) Brett Smiley @brettsmiley Melanie /Ex-#GOP + 4 Jon McC @fatdaddybulldog The Mighty Thor @MightyThor617 Sophie @SophieBully Marley @BeautyBull Delilah & Zoey @delilahgrace8 Coco + 9 Chaucer @BulldogChaucer Zeus @zeusGreywind Miss Dixie Dumplin' @FloridaPhyl Melanie /Ex-#GOP and You Melanie /Ex-#GOP and You Griffen The Bulldog @griff_bulldog Cooper Fartacus, Gracie Sunshine Melanie /Ex-#GOP and You Jane Bloom, Bogart Gracie Sunshine and You carrie @trooperMoo Melanie /Ex-#GOP and You Lord Roscoe @lord_roscoe Melanie /Ex-#GOP and You Melanie /Ex-#GOP, EmmyRose (G💗's sis) Anna Willett @AnnaWillett GEBR @gabullierescue sam dellinger, MAGOO Valerie Haines @ValerieHaines Mandy + 4 sam dellinger + 12 Deanna Meadowcroft @DeannaMeady Gracie Sunshine and You Gunner @BullDogGunner Melanie /Ex-#GOP, EmmyRose (G💗's sis) Churchill @Churchiebulldog Jenny @jdwalkerohmy Melanie /Ex-#GOP, EmmyRose (G💗's sis) Melanie /Ex-#GOP, EmmyRose (G💗's sis) Boomer Sprinkle @BoomerSprinkle Birthday Question?? FirstThereWasHarmony @FirstWasHarmony Lady Maggie @Magsabully Paloma J Undertooth™ @PalomaTheBoston Lynsey Robinson @LoveBulldog Ollie @olliegator57 Turkey Schmerky @casie190 Titus @jspring152 Stormy @EllaBulldogWI My Cat Gang @archie_the Rudozem Street Dogs @RSDR Sway Bears & Jack + 9 Cleopatra Eng Bully, Storm #love4georgia Dogs of Des Moines + 3 sam dellinger + 4 Scarlett ❤️ Georgia @PeachyKeen57 Pumpkin & Georgia @LadyPumpkinLove Mia The Frenchie @ThePudgeMM Tao of Dolly @TaoOfDolly Pam Lang @cheersandcrowns Jen @Bogelicious1 Goober @sincitybordeaux Charlie and Renae @NaeNae_1204 Sara Keller @ILuv2nap Snorf Industries @snorfindustries Brave Bulldogs @CueFoils
Oswald'sPack @OswaldsPack Sir Angus B. @SirPawbulous Lola @poopachoo MrBully, Mia & T.J. @BullyMiaTJ Helen Yates @HelenYatesArt mcgrubbs @mcgrubbs 🍀Emmett_dog 🍀 @Emmett_dog Jackson the Bully @shan6872 Marlene The Frenchie @FrenchieMarlene PEIA Bella @BabyB_Bulldog Lulu & Zeus @pam71579 Izzy & Louise @asamp213 Meesha The Maltese @sweetlilmeesha Harry's Niece Purdey @PurdeyHollett Edgar Eraspuss @EdgarEraspuss Hooch @TicketyBooMedia No Iowa Puppy Mills @IAFriends |
|
oops that 170 separate conversation threads I have! Yikes!! |
|
😃 Indeed, that's a lot. You may have notice Twitter is loading additional conversations when you scroll in the "Messages" list on Twitter. What would help me is to know the request that is done by Twitter but I don't want to bother you with that because it is a bit complicated. It would require to check the web requests done in the "Network" tab of the "Developer resources" you've open previously and check what is happening when you scroll in the "Messages" tab (just after you've clicked on the "Messages" button). |
|
I really have nothing to hide do you want to just login as me and scroll On Nov 8, 2016 5:42 PM, "Julien Ehrhart" notifications@github.com wrote:
|
|
Otherwise I can try what you suggest if you give me some instructions on On Nov 8, 2016 5:42 PM, "Julien Ehrhart" notifications@github.com wrote:
|
|
I've sent you a follow request on Twitter, if you follow me back, we will be able to talk in private via Twitter. |
|
The initial error message |
|
Hey Julien,
Not sure how to create a new message on your website, but this is the
errors i got when loading the revised program today
Last login: Thu Feb 16 09:49:10 on ttys000
Ronnies-MacBook-Pro:~ ronniesussman$ cd downloads
Ronnies-MacBook-Pro:downloads ronniesussman$ ./dmarchiver -di
Enter your username or email: beckbulldognj
Enter your password (characters will not be displayed):
Error: Your username or password was invalid.
Exiting.
Failed to execute script cmdline
Ronnies-MacBook-Pro:downloads ronniesussman$ ./dmarchiver -di
Enter your username or email: beckybulldognj
Enter your password (characters will not be displayed):
Authentication succeedeed.
Conversation ID not specified. Retrieving all the threads.
Starting crawl of '629006352329760768'
Unexpected error for tweet '824997990851080200', raw HTML will be used for
the tweet.
Unexpected error for tweet '816135843626831875', raw HTML will be used for
the tweet.
Unexpected error for tweet '815389148622241795', raw HTML will be used for
the tweet.
Unexpected error for tweet '812079166459904003', raw HTML will be used for
the tweet.
Unexpected error for tweet '811391729601486851', raw HTML will be used for
the tweet.
Unexpected error for tweet '809200106889093123', raw HTML will be used for
the tweet.
Begin of thread reached0
Total processed tweets: 166850
Writing conversation to 629006352329760768.txt
Starting crawl of '1214469206-3229265936'
Failed to execute script cmdline
Traceback (most recent call last):
File "requests/packages/urllib3/connectionpool.py", line 595, in urlopen
File "requests/packages/urllib3/connectionpool.py", line 393, in
_make_request
File "<string>", line 2, in raise_from
File "requests/packages/urllib3/connectionpool.py", line 389, in
_make_request
File "http/client.py", line 1197, in getresponse
File "http/client.py", line 297, in begin
File "http/client.py", line 258, in _read_status
File "socket.py", line 575, in readinto
File "ssl.py", line 929, in recv_into
File "ssl.py", line 791, in read
File "ssl.py", line 575, in read
ConnectionResetError: [Errno 54] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "requests/adapters.py", line 423, in send
File "requests/packages/urllib3/connectionpool.py", line 640, in urlopen
File "requests/packages/urllib3/util/retry.py", line 261, in increment
File "requests/packages/urllib3/packages/six.py", line 685, in reraise
File "requests/packages/urllib3/connectionpool.py", line 595, in urlopen
File "requests/packages/urllib3/connectionpool.py", line 393, in
_make_request
File "<string>", line 2, in raise_from
File "requests/packages/urllib3/connectionpool.py", line 389, in
_make_request
File "http/client.py", line 1197, in getresponse
File "http/client.py", line 297, in begin
File "http/client.py", line 258, in _read_status
File "socket.py", line 575, in readinto
File "ssl.py", line 929, in recv_into
File "ssl.py", line 791, in read
File "ssl.py", line 575, in read
requests.packages.urllib3.exceptions.ProtocolError: ('Connection aborted.',
ConnectionResetError(54, 'Connection reset by peer'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "dmarchiver/cmdline.py", line 70, in <module>
File "dmarchiver/cmdline.py", line 67, in main
File "dmarchiver/core.py", line 466, in crawl
File "requests/sessions.py", line 488, in get
File "requests/sessions.py", line 475, in request
File "requests/sessions.py", line 596, in send
File "requests/adapters.py", line 473, in send
requests.exceptions.ConnectionError: ('Connection aborted.',
ConnectionResetError(54, 'Connection reset by peer'))
Ronnies-MacBook-Pro:downloads ronniesussman$
…On Wed, Nov 9, 2016 at 5:14 PM, Julien Ehrhart ***@***.***> wrote:
The initial error message json.decoder.JSONDecodeError: Expecting value:
line 1 column 1 (char 0) could not been reproduced on the same thread.
Closing this issue for now.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#7 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AVxObrCnNNFK5UCE6uaLS5RppOFotNNLks5q8kWvgaJpZM4Ks2lb>
.
|
Edited by Mincka on August 10th 2017:
The error message was due in this case to invalid json data. It seemed to be related to a connection issue and it was not possible to reproduce it. Other causes can be found here: https://stackoverflow.com/a/18460958
Original post:
New ticket created from
#1 (comment)
$ /Users/xxx/Downloads/dmarchiver -id "YYY" -di -dg
Enter your username or email: zzz
Enter your password (characters will not be displayed):
Authentication succeedeed.
Conversation ID specified (YYY). Retrieving only one thread.
Starting crawl of 'YYY'
Failed to execute script cmdline
Traceback (most recent call last):
File "dmarchiver/cmdline.py", line 70, in
File "dmarchiver/cmdline.py", line 62, in main
File "dmarchiver/core.py", line 468, in crawl
File "requests/models.py", line 826, in json
File "json/init.py", line 319, in loads
File "json/decoder.py", line 339, in decode
File "json/decoder.py", line 357, in raw_decode
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The text was updated successfully, but these errors were encountered: