The Python package Nyctibius is designed to streamline the complex task of gathering and consolidating sociodemographic data from various sources into a cohesive relational database. Nyctibius empowers users to effortlessly unify custom data sets from diverse socio-demographic sources, ensuring that they can work with up-to-date and comprehensive information in a seamless manner. This package facilitates the process of creating a harmonized repository of socio-demographic data, simplifying data management and analysis for users across various domains.
-
Extraction:
- Seamlessly retrieve data from online data sources through web scraping, as well as from local files.
- Support for various data formats, including
.csv
,.xlsx
,.xls
,.txt
,.sav
, and compressed files, ensuring versatility in sourcing information.
-
Transformation:
- Consolidating extracted data into pandas DataFrame.
- Optimizing the transformation process of large files.
- Implement parallel processing for large files.
- Use efficient data structures to reduce memory footprint.
- Effectively manage data inconsistencies and discrepancies for enhanced accuracy.
- Apply anomaly detection algorithms.
-
Load:
- Consolidating transformed data into a cohesive relational database.
-
Query:
- Conduct precise queries and apply transformations to meet specific criteria.
-
AI Query & Visualization:
- Using natural language input to query data (Answers from values to subsets)
- Using natural language input to create simple visualizations of data
Nyctibius is ideal for data analysts, scientists, and researchers who frequently handle large volumes of data from varied sources and are looking for a streamlined way to consolidate, query, and visualize their data. It's also a great tool for developers working on projects that require the integration of disparate data sets into a single, manageable format. Additionally, business intelligence professionals and decision-makers will find Nyctibius invaluable for generating insights through natural language queries and visualizations, making complex data more accessible and actionable. In essence, anyone looking to simplify their data workflows, from extraction to visualization, and leverage AI for natural language querying will benefit greatly from using Nyctibius.
For full documentation, please refer to the Nyctibius documentation.
You can install the Nyctibius package using pip. Make sure you have Python 3.x installed on your system; the package requires Python version 3.7 or higher.
pip install nyctibius
To use the Nyctibius package, follow these steps:
-
Import the package in your Python script:
from nyctibius import Harmonizer
-
Create an instance of the
Harmonizer
class:harmonizer = Harmonizer()
-
Extract data from online sources and create a list of data information:
url = 'https://www.example.com' depth = 0 ext = 'csv' list_datainfo = harmonizer.extract(url=url, depth=depth, ext=ext) harmonizer = Harmonizer(list_datainfo)
-
Load the data from the list of data information and merge it into a relational database:
results = harmonizer.load()
-
Import the modifier module and create an instance of the
Modifier
class:from nyctibius.db.modifier import Modifier modifier = Modifier(db_path='../../data/output/nyctibius.db')
-
Perfom modifications:
tables = modifier.get_tables() print(tables)
-
Import the querier module and create an instance of the
Querier
class:from nyctibius.db.querier import Querier querier = Querier(db_path='data/output/nyctibius.db')
-
Perform queries:
df = querier.select(table="Estructura CHC_2017").execute() print(df)
The package supports the following sources:
- Colombian microdata links from National Administrative Department of Statistics (DANE)
- Local files
- Open data sources
Please note that accessing data from these organizations may require authentication or specific credentials. Make sure you have the necessary permissions before using the library.
The Nyctibius package is open-source and released under the MIT License. Feel free to use, modify, and distribute this library in accordance with the terms of the license.
We would like to thank the following entities for providing the data used and the economic financial support for the development of this package:
- National Administrative Department of Statistics (DANE)
- Barcelona Supercomputing Center (BSC)
- Universidad de los Andes
For any questions, suggestions, or feedback regarding the package please contact:
Erick lozano, Email: es.lozano@uniandes.edu.co
Diego Irreño, Email: dirreno@unal.edu.co
This library is not officially affiliated with or endorsed by any of the mentioned official organizations. The data provided by this library is sourced from publicly available information and may not always reflect the most current or accurate data. Please verify the information with the respective official sources for critical use cases.