Skip to content
This repository was archived by the owner on Jun 28, 2021. It is now read-only.

draftcode/fastcsv

Repository files navigation

TLDR: This is deprecated. Use Python3 and the standard csv module.

  • This library was written in 2013 and meant for Python2. Unless you're still using Python2 and need to read non-ASCII CSV files, there's no reason not to use the standard csv module.
  • The performance was measured in 2013 with Python2. Python3 doesn't have the encoding issue I wanted to address in the first place, and thus there's no performance issue. I this Python3 itself has numerous performance improvements, so that old measurement is not trustful.
  • The complexity is anyway O(N).

So, I don't recommend using this library.

More Context

There was a reason to use Python2 in 2013. In my case, it was comtypes and py2exe. I work for a Japanese company and I had to write software to process CSV files written in Japanese and the platform was Windows. The CSV files were mostly written by Microsoft Excel, and it used non-UTF-8 character encodings. In Python2's csv module document, there's a note for non-ASCII CSV files.

Note This version of the csv module doesn't support Unicode input. Also, there are currently some issues regarding ASCII NUL characters. Accordingly, all input should be UTF-8 or printable ASCII to be safe; see the examples in section Examples.

https://docs.python.org/2/library/csv.html

This means I had to use decode -> encode -> CSV parsing -> decode proccess to read CSV files in unicode. This slowed down the software. To address this issue, I wrote this library.

Now, I don't work for that company anymore and I don't maintain this library. I don't think there's a benefit in using a deprecated library for this small performance benefit for the modern environment. Even if the benchmark result is still relevant, the benefit would be small anyway.

When the data is so large that it makes the difference, it's better not to use Python and CSV. Use C++ or Java, MapReduce or FlumeJava (or some equivalent I don't know), ColumnIO or something, and then the data is naturally sharded and the data read/write speed is negligible.

Packages

No packages published