Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
⇔ wq's io library, an interoperability tool for importing and exporting tabular and time series data, e.g. from citizen science, crowdsourcing, and VGI.
Python
branch: master
Failed to load latest commit information.
gis
parsers ensure extra_data is an instance property
tests account for unicode normalization in python 3
.gitignore ignore more stuff
.travis.yml
LICENSE update copyright year
README.md issue tracker link [skip ci]
__init__.py wq cat arbitrary IO classes (see #8)
base.py handle data with different fields in each row
commands.py
exceptions.py
loaders.py wq cat arbitrary IO classes (see #8)
mappers.py account for unicode normalization in python 3
setup.py
util.py wq cat arbitrary IO classes (see #8)
version.py

README.md

wq.io

wq.io is a Pythonic library for consuming (input), iterating over, and generating (output) external data resources in various formats. wq.io facilitates interoperability between the wq framework and other systems and formats.

wq.io is designed to be customized, with a base class and modular mixin classes that handle loading, parsing, and mapping external data to a convenient API.

Release Notes | Installation | Documentation | Issue Tracker

Build Status PyPI Package

Tested on Python 2.7 and 3.4.

Somewhat coincidentally, https://wq.io is also the URL for the website describing the wq framework as a whole. The documentation for wq.io (the library) is available on wq.io (the website) at https://wq.io/wq.io.

Getting Started

# Basic install
pip3 install wq.io

# Alternatively, install the wq metapackage if using together with wq.app and/or wq.db:
pip3 install wq

# To enable wq.io's GIS support
pip3 install geopandas # includes Shapely & Fiona

# To enable wq.io's Excel write support
pip3 install xlwt # xls support (use xlwt-future for Python 3)
pip3 install xlsxwriter # xlsx support
# (xls/xlsx read support is enabled by default)

See the wq documentation for more information.

Features

wq.io provides a general purpose API for loading, iterating over, and writing tabular datasets. The basic idea is to avoid needing to remember the unique usage of e.g. csv, xlrd, or xml.etree every time one needs to work with external data. Instead, wq.io abstracts these libraries into a consistent interface that works as an iterable of namedtuples. Whenever possible, the field names for a dataset are automatically determined from the source file, e.g. the column headers in an Excel spreadsheet.

from wq.io import ExcelFileIO
data = ExcelFileIO(filename='example.xls')
for row in data:
    print row.name, row.date

wq.io provides a number of built-in classes like the above, including a CsvFileIO, XmlFileIO, and JsonFileIO. There is also a convenience function, load_file(), that attempts to automatically determine which class to use for a given file.

from wq.io import load_file
data = load_file('example.csv')
for row in data:
    print row.name, row.date

All of the included *FileIO classes support both reading and writing to external files, though write support for Excel files requires additional libraries (xlwt and xlsxwriter) that aren't listed as dependencies.

Network Client

wq.io also provides network-capable equivalents of each of the above classes, to facilitate loading data from third party webservices.

from wq.io import JsonNetIO
class WebServiceIO(JsonNetIO):
    url = "http://example.com/api"

data = WebServiceIO(params={'type': 'all'})
for row in data:
    print row.timestamp, row.value

The powerful requests library is used internally to load data over HTTP.

Pandas Analysis

When Pandas is installed, the as_dataframe() method on wq.io classes can be used to create a DataFrame, enabling more extensive analysis possibilities.

instance = WebServiceIO(params={'type': 'all'})
df = instance.as_dataframe()
print df.value.mean()

GIS Support

When Fiona and Shapely are installed, wq.io can also open and create shapefiles and other OGR-compatible geographic data formats.

from wq.io import ShapeIO
data = ShapeIO(filename='sites.shp')
for id, site in data.items():
    print id, site.geometry.wkt

Extending wq.io

Each IO class is composed of mixin classes (loaders, parsers, and mappers) that handle the various steps of the process. By extending these mixin or the pre-mixed classes above, it is straightforward to extend wq.io to support arbitrary formats. The climata library provides a number of examples of custom IO classes for loading climate and hydrology data.

Something went wrong with that request. Please try again.