Purpose

A collection of very simple scripts to handle text-based datasets, probably scraped data in .csv or .json formats.

convert_csv2markdown

Convert csv-tables to human readable markdown. Execute script with input file as an argument Default delimiter is ";" and may be changed in line 9 in the script.

fileCat.py

Merges together files you specify.

remove_duplicate_rows.py

Simple script to remove duplicate rows in a dataset, ideally in .csv-Format.

1. opening a file in the read mode. This is the file that has the duplicates.
2. Then in a loop that runs till the file is over, we check if the line has already encountered.
3. If it has been encountered than we don't write it to the output file.
4. If not we will write it to the output file and add it to the list of records that have been encountered already

replace_linebreaks.py

Replace linebreaks in a text file, for example after OCR. Execute script with input file as an argument.