New dataset with companies information

[Irio]

Comes to solve our need (#14) of addresses and coordinates of all companies listed in the expenses. The initial version is the result a collaboration between me, @filipelinhares and @andrepinho.

Here's a short summary of what the current dataset has (not sure if this notebook fits well in the repository, probably not): https://gist.github.com/Irio/002a1d3842466aacd0027c6f382a2d9b
Once this pull request gets approved, I will uploaded it to Amazon S3.

Current version of the dataset contains a list of 59,985 companies; when counting just the Brazilian ones, 3,453 are missing given timeouts returned by the API used.

Missing

• Get coordinates of addresses
• Translate dataset
• Upload file to Amazon S3 and add its URL to src/fetch_datasets.py

[cuducos]

itertools.chain 💕

[cuducos]

I added some comments in a in line basis. Nothing really important, but my intent was to make the code easier to understand and more optimized (when needed). Most of the comments are more stylistic than technical and accepting the suggestions or not I feel like this is good to merge already.

I'll let it up to you to merge, because I'm not sure about the 3rd item from the TODO list (Upload file to Amazon S3) — so feel free to go through the comments, calling me names for being such a dick, uploading the files and finally merging ; )

[cuducos]

Rereading I found another issue — maybe more important than all the other ones.

Defining file paths like 'data/cnpj_info.xz' might raise an error in non-Unix platforms (aka Windows). The proper way to do it is to use os.path.join('data', 'cnpj_info.xz') so Python uses the proper slash (/ or \) according to the OS.

We might also need review our code base because this might have passed unnoticed in other PRs…

[Irio]

Squashing few commits and merging this branch now. The work hasn't completely finished, since there are 3,391 CNPJs remaining to be fetched (they keep returning 504 HTTP status). There are 57,644 geolocated companies, from a total of 60K. Some seem to be incorrectly pointed by Google Maps, but I'm happy with this first version for now.

https://nbviewer.jupyter.org/gist/Irio/22026574ebc597bdedfbdb13989ffe98

To get the dataset, already available on Amazon S3, run the following:

$ python src/fetch_datasets.py