Converted code to utilize new twitter archive file format

[dannydover]

Twitter changed the file format they use for their data export/archive function from CSV to JSON. (Details available here: https://kyleconroy.com/your-twitter-data) This pull request updates the original code to parse JSON rather than CSV.

[denis-trofimov]

Hi!
You are wonderful guy @dannydover!
Your code has worked for me just now.
P.S. I am sad to see devs in Twitter have added this new obvious bug "ID as a float JS number" in the export data feature.

[Lewiscowles1986]

I got a CSV today. Not sure if it's all users or some, but yeah probably needs support for both.

[denis-trofimov]

@Lewiscowles1986 Is it a good idea is to add command line parameter -json or - csv?
1.probably a different local server, not all get updated at the same time.
2.Maybe they found a bug in JSON and rollback to CSV format?

[lalaithan]

I got a JSON file the other day and a CSV file today, so I guess they went back. Anyone know why it would show an unknown syntax error on line 17 or 30 for the JSON version?

[Lewiscowles1986]

Guys, I've read up more on this. Video record the two sessions. You're going to two similar screens which product different things. Check the URL

[koenrh]

It seems that Twitter now indeed offers two ways to export your data:

1. "Your Tweet archive" (from your account page) (contains tweets.csv)
2. "Your Twitter data" (from your Your Twitter data page) (contains tweet.js)

The first one is the one that is described in the README, and returns an archive that includes a simple CSV file containing all your tweets. The latter one returns a lot more (personal) data, and was likely introduced recently in response to new EU regulatory requirements (GDPR). I'm not sure if that option is even available to non-EU-accounts.

Could anyone of you (non-Europeans) confirm that the "Your Twitter data" feature is accessible outside the EU?

Alternatively, we could change the script so that it takes either a path to tweets.csv or tweet.js as argument, and internally either use a CSV or JSON reader, respectively (e.g.python deletetweets.py -d 2014-01-01 tweet.js). What do you think?

[lalaithan]

I am outside the EU and can receive the extra data.

…

On Mon, Jul 30, 2018, 10:48 AM Koen Rouwhorst ***@***.***> wrote: It seems that Twitter now indeed offers two ways to export your data: 1. "Your Tweet archive" (from your account page <https://twitter.com/settings/account>) (contains tweets.csv) 2. "Your Twitter data" (from your Your Twitter data page <https://twitter.com/settings/your_twitter_data>) (contains tweet.js) The first one is the one that is described in the README <https://github.com/koenrh/delete-tweets/blob/master/README.md>, and returns an archive that includes a simple CSV file containing all your tweets. The latter one returns a lot more (personal) data, and was likely introduced recently in response to new EU regulatory requirements (GDPR). I'm not sure if that option is even available to non-EU-accounts. Could anyone of you (non-Europeans) confirm that the "Your Twitter data <https://twitter.com/settings/your_twitter_data>" feature is accessible outside the EU? Alternatively, we could change the script so that it takes either a path to tweets.csv or tweet.js as argument, and internally either use a CSV or JSON reader, respectively (e.g.python deletetweets.py -d 2014-01-01 tweet.js). What do you think? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFKcGojZB1j6prHrdMI8R76LDQkYpyl0ks5uLxy7gaJpZM4VZ3EF> .

[koenrh]

I have updated the README to make sure people download the right archive (#10).

I prefer to keep this script as simple as possible, and only support the CSV file.