CSV Database filter script

[sanketgarade]

Program to take a database (csv format currently) as input, keep only the necessary data (as per a filter criteria which is another input), and output this data in the same format as the input database.

[sanketgarade]

purpose -

to filter the input csv data as per the provided arguments

filter types

• invalid data
• all words
• specific topic
• specific alphabet (which will be initial of the english word)
• TBD (more can be added)

about invalid data filter

the invalid data filter is to be run always before running any other filter, as it eliminates those data elements which have insufficient or invalid data

• insufficient data - any of the english or marathi word is missing

• invalid data - if english word contains non english characters (this can be thought of later, and is low priority right now), same for marathi word.

IMP

• make separate functions for each filter type
• as per the passed argument of the "filter type" call the relavant function

steps

1. take passed csv data and filter type as argument
2. generate a truncated csv data structure as per the target filter

• call the specific filter function here

3. return this truncated data to the calling function

[zarbod]

Hey so when you say "insufficient data" do you mean missing English or Marathi words exclusively, or does it also include missing examples and tags?

[sanketgarade]

only the main 2 words. 1 en and 1 mr.

[zarbod]

Thanks. Also could you explain what the "all words" filter is supposed to do?

[sanketgarade]

Thanks. Also could you explain what the "all words" filter is supposed to do?

"All words" basically means no filtering (other than the invalid/insufficient data, of course).

[zarbod]

So I would just call the invalid/insufficient data scripts when the filter type is "All words"?

[sanketgarade]

Yes. Pretty much.

[zarbod]

Also, the filter by topic function will require the topic as an argument. Do you want me to add an optional topic argument to the main filter function?

[sanketgarade]

Yes you can do it in whichever way that makes the functions easy to use and also reusable.

What I've written in the gen-out.py file is just a basic example.
You can fill in the details and missing gaps.

[sanketgarade]

pending issues from PR #32

among this priority ones are the # 1 and # 3