Here you find a convinient solution of this problem. class AccumulativeData provides a simple interface to store data step by step. The data can be consisted of:
- Numbers
- Lists / arrays
- Objects
You can store it as pickled object or Pandas Dataframe.
The module can be installed from pip
pip install accudata
You have a social data collecting process. You must collect on every step heterogeneous data:
- Name of a person
- Age
- Interests
- Preferences by categories: food, pets, sport, politics
You can make a class:
from accudata import AccumulativeData
class PeopleAccData(AccumulativeData):
def __init__(self):
lists = ['name', 'age', 'interests']
dicts = {'pref': ['food', 'pets', 'sport', 'politics]}
super().__init__(lists=lists, dicts=dicts)
After that you can make an iterative collecting process as follows:
Data = PeopleAccData()
for item in raw_data:
Data.next()
# \\\ A complicated code to extract data
name, age, interests, food, pets, sport, politics, _ = extract_data(item)
Data.append(name, age, interests,
pref=[food, pets, sport, politics])
It is simple to get data:
names = Data.name
# Make the dataframe
dataframe = Data.todf()
print(dataframe.name)