datopy (da-toh-pie) is a Python library for people who work with unstructured data, providing a simple workflow for building data models and ETL (extract, transform, load) pipelines.
This package also includes utilities for:
- Data retrieval (web scraping and API-based data retrieval)
- Input/output processes (loading and inspecting data)
- Jupyter Notebook workflows
Note
This project is under active development.
To use datopy, first install it using pip:
$ pip install "git+https://github.com/bainmatt/datopy.git#egg=datopy"Step 1. Clone the repo:
$ git clone https://github.com/bainmatt/datopy.git
$ cd datopyStep 2. Install dependencies:
$ conda env create -f environment.yml
$ conda activate datopyWIP.
Instructions on typing/testing/documentation, CI/CD, and conventions for developers.
API reference: :mod:`datopy.inspection`
Produce multiple parallel, informative displays of Pandas data frames and NumPy arrays for data exploration and inspection.
>>> import numpy as np
>>> import pandas as pd
>>> from datopy.inspection import display, make_df
>>> df1 = make_df('AB', [1, 2]); df2 = make_df('AB', [3, 4])
>>> display('df1', 'df2', 'pd.concat([df1, df2])', globs=globals(), bold=False)
df1
--- (2, 2) ---
A B
1 A1 B1
2 A2 B2
df2
--- (2, 2) ---
A B
3 A3 B3
4 A4 B4
pd.concat([df1, df2])
--- (4, 2) ---
A B
1 A1 B1
2 A2 B2
3 A3 B3
4 A4 B4API reference: :mod:`datopy._media_scrape`
WIP.
More usage examples to come.
Retrieve media-related data from Spotify, IMDb, and Wikipedia.
This package is powered by:
- mypy type checking
- pytest unit testing
- Flake8 linting
- Sphinx documentation
- numpydoc docstrings
- PyData theming
- Read the Docs hosting
- GitHub Actions continuous integration
- PyPI packaging
- Pydantic data validation
This project is licensed under the MIT License.
Project Link: https://github.com/bainmatt/datopy