Skip to content
Extract / Transform / Load Scripts for databases used in Dataviva Project
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
censo_escolar
comtrade
datasus
ies/extract
lib
rais/extract
secex/extract
static_tables
.gitignore
README.md
magic
magic.mgc
magic1.dll
regex2.dll
requirements.txt
zlib1.dll

README.md

DataViva ETL

Extract / Transform / Load Scripts for databases used in Dataviva Project.

The sources of Dataviva can be found at Dataviva GitHub.

How it Works

Databases are released by Brazil government as raw data and there is some analysis to extract the information needed for the project.

For each database we have a Wiki documentation that explain all the ETL process.

Setup

  1. You'll need libmagic for this project:

Mac installation

brew install libmagic

Ubuntu (BSD Linux) installation

apt-get install libmagic
  1. Install requirements
pip install -r requirements.txt

See the Wiki for more information.

You can’t perform that action at this time.