ERROR: 4 - fails to transform a large JSONL file

[reichaves]

Hello
Please, I have a very large .jsonl file (870MB) and I am using python 3.8 on Ubuntu
But twarc csv fails to transform to CSVs:

twarc2 csv results.jsonl tweets_with_attacks_journalists.csv

💔 ERROR: 4 Unexpected items in data!
Are you sure you specified the correct --input-data-type?
If the object type is correct, add extra columns with:
--extra-input-columns "edit_controls.is_edit_eligible,edit_controls.editable_until,edit_history_tweet_ids,edit_controls.edits_remaining"
Skipping entire batch of 9944 tweets!

Is there any other way to convert to CSV?

[SamHames]

Those columns are the very recently added fields related to tweet editing, twarc-csv doesn't support them yet.

Does using the suggested extra flags work? It should look like this:

twarc2 csv results.jsonl tweets_with_attacks_journalists.csv --extra-input-columns "edit_controls.is_edit_eligible,edit_controls.editable_until,edit_history_tweet_ids,edit_controls.edits_remaining"

[reichaves]

Thanks, that way it worked

[igorbrigadir]

Fixed in 0.6.0

Please update with:

pip install --upgrade twarc-csv