Data extraction based on specified params

[mehrdadmms]

Goal

• The script can be scheduled to fetch the information periodically form the specified channels
• The script will fetch the information, minimize the data size by removing unnecessary information and store it in the database

how

• Get the list of specified servers and channels from the settings collection alongside the extraction period
• Check if there's data already available in database for this server
• Fetch the missing data based on the period and previous data extractions
• Clean the raw data and minimaize the amount of data we're going to store
• Create a new database for the server
• Store raw data in the raw data collection

More data

For checking the data schema you can go here

[mehrdadmms]

This is a high level task which needs break down, currently it's broken down to #50 and #49 but more issues will be added