Allow /src scripts to receive data files as command line arguments

[Irio]

The majority of scripts located in process datasets in hard coded locations, like data/cnpj-info.xz in src/fetch_cnpj_info.py#L10. Given a lot has changed since their creation, we expect them to receive data paths as command line arguments, as in python src/fetch_cnpj_info.py data/2016-12-06-cnpj-info.xz.

[marcusrehm]

Hi @Irio ! I'm working exactly that. I spoke with @cuducos about issue #67 that I also need to use fetch_cnpj_info with the amendments' dataset .

Actually, I was thinking about passing the column with the CNPJ's as a parameter also, so the call would be src/fetch_cnpj_info.py './data/2016-12-06-cnpj-info.xz' 'cnpj'. What do you think?

[marcusrehm]

Hi @Irio

I did the refactoring in fetch_cnpj_info.py now we can call it passing filename and column with CNPJ's.

[cuducos]

Sorry if I missed anything… but I don't think passing the column we need is what I had in mind. When I read OP's description I think he meant the file to be load (for example 2016-08-08-companies.xz or 2016-12-06-companies.xz). Also I'm afraid that's not what I meant in #67, @marcusrehm (but I'm gonna clarify that on the issue page).

[marcusrehm]

@Irio @cuducos I think this one was resolved with PR #185 as in https://github.com/datasciencebr/serenata-de-amor/blob/master/src/fetch_cnpj_info.py#L159 .

[cuducos]

@marcusrehm PR #185 addresses this issue when it come to company scripts, but not for all scripts inside src/ directory ; )

[marcusrehm]

yes, @cuducos , you're right! =)

[martini97]

I would like to know specifically what is the idea in this issue. I've looked through some files in source, and some of then just download data, would you like the script to specify the path where the data is to be saved? Or would you like only for the scripts that read files to receive arguments? Anyway I think it wouuld be nice to post a roadmap with the files that you'd like to change. Thanks.

[cuducos]

Hi @martini97,

would you like the script to specify the path where the data is to be saved?

No, actually what would be interesting is to specify via command line the files they read (as I commented earlier).

Anyway I think it wouuld be nice to post a roadmap with the files that you'd like to change.

Well… files from src/ that read other file:

• clean_cnpj_info_dataset.py (WIP Companies partners list in companies dataset - Issue #16 #218)
• fetch_civil_names.py
• fetch_cnpj_info.py (WIP Companies partners list in companies dataset - Issue #16 #218)
• fetch_federal_budget_datasets.py
• fetch_federal_sanctions.py
• fetch_foursquare_info.py
• fetch_purchase_suppliers.py
• fetch_receipts.py
• fetch_sex_places.py
• fetch_yelp_info.py
• geocode_addresses.py
• get_family_names.py
• group_receipts.py
• translation_table.py

Or another way to put every script but:

• backup_data.py
• utils.py
• And fetch_sex_places.py (already written this way)

[marcusrehm]

Hi @martini97 ! I think what @cuducos suggested was commented here. He's asking to create a sort of mapping like it:

# not functional, just a example
cols = {'amendments': 'beneficiary', 'other_dataset': 'something_else}
cnpj_col = cols.get(base_file_name, 'cnpj')

So when the script receive a file as argument it can grab data (CNPJs) from the referenced column and to the job.

It is already done in fetch_cnpj_info.py for the following datasets:

datasets_cols = {'reimbursements': 'cnpj_cpf',
                 'current-year': 'cnpj_cpf',
                 'last-year': 'cnpj_cpf',
                 'previous-years': 'cnpj_cpf',
                 'amendments': 'amendment_beneficiary'}

Is that right @cuducos ?

[cuducos]

👍

[willianpaixao]

@Irio @cuducos is this still an issue?

[cuducos]

Closed because this src/ folder is not in use anymore.