Open Data Crawler

A tool to crawl data to your projects from open data portals
Report Bug · Request Feature

Table of Contents

1. About The Project
2. Getting Started
   • requirements
   • Installation
3. Usage
4. Roadmap
5. License
6. Contact

About The Project

Open Data Crawler is a tool to extract data from open data portals and statistics portals. The community can contribute adding support to other data portals or adding new features.

Features:

• Download datasets from open data portals
• Download metadata from resources
• Filter by data type

(back to top)

Getting Started

This is an example of how you may give instructions on setting up your project locally. To get a local copy up and running follow these simple example steps.

Requirements

• You need at least python 3.9 installed

• Clone the repo

  git clone https://github.com/aberenguerpas/opendatacrawler.git

• Move to root directory

  cd opendatacrawler

• Install the requirements from requirements.txt

  pip install -r requirements.txt

• Socrata portals requiere an app token to avoid throttling limits, you can obtain an api key here and set on config.ini

Installation

1. Run from the project root

   python3 setup.py install 

(back to top)

Usage

Use this tool is very simple, you only need to specify the data source and the tool automatically detect the portal type and starts to dowload the data.

Examples

Dowload all data and metadata from a portal:

python opendatacrawler -d data.europa.eu

Download only metadata:

python opendatacrawler -d data.europa.eu -m

Dowload specific fromat data. For example xls and csv:

python opendatacrawler -d data.europa.eu -f xls csv

For more examples, please refer to the Documentation

(back to top)

Currently supported portals and sites

• data.europa.eu

See the open issues for a full list of proposed features (and known issues).

(back to top)

License

Distributed under the MIT License. See LICENSE for more information.

(back to top)

## Colaborators

🙋‍♂️ Javier Ríos Cerdán 🙋‍♂️ Alberto Berenguer Pastor

Contact

✉️ alberto.berenguer@ua.es

(back to top)