This application does simplified categorization of CSV rows based on defined rules. It is rather than quick and dirty implementation than complete and foolproof package.
Categorization rules are stored in file rules.txt
US Northeast Garbage Renters
addr_state IS ANY PA,NY,NJ,VT,RI,NH,MA,ME,CT
grade IS NOT A,B,C
home_ownership IS ANY RENT,MORTGAGE
First row is name of rule
Followed by rows of rule definitions
Empty line defines end of rule
**rule**
\t <field_name> \t <keyword> \t <value or values>
Multiple values are separated by comma
IS value - verifies that field value equals to defined value
EQUALS value - same as above
IS ANY values - verifies that field value is within list of values
IS NOT value or values - verifies that field value is not within list of values
ALL EXCEPT value or values - same as above
- C# / Single thread / Array based run: 13.52sec
- Python / Single thread / Array based run: 24.01sec
- Python / Single thread / String based run: 24.04sec
- Python / Multithreaded 8 cores: 7.87sec
C# probably has some issue with IS NOT, but it is definitely faster
Python is slower, however more nice to work with. Array based run showed same performance as checking parts of string
Probably C# multithreaded solution might be faster, but so far results are VERY good with Python multithreading
For #4 - 3.04s was process_time() which does not include waiting for stuff, so still has room for optimization