You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Uses the existing SQLite intermediate databases for all stations.
Drops existing mongodb database.
Computes data states for all stations and all dates (from 01/01/1994 to today).
Inserts these values into a new mongodb database.
The current process takes upwards of 24 hours to process approximately upto data states 10,000 GPS Stations. Daily re-computation of Historical data states is redundant and can be avoided.
A simple performance improvement would be compute only the end_date's data state and attach it to the mongodb collection everyday, instead of dropping and creating from scratch.
The text was updated successfully, but these errors were encountered:
…emoved historical data state calculation. Commented code which wouldn't be required to be run daily e.g. Meta data inserts, db drop, index computation
Check if database exists. If it does not, then run the old create_mongodb.py script which processes all stations and computes historical data states.
For each Station, check if station data is present. If it is not, then this means station is new and so insert station's metadata into other collections as well as compute this stations historical data states.
Reverted create_mongodb.py to its previous state to perform entire historical date computation and inserts from scratch.
The mongodb_load_delta.py now does following additional checks:
1. Checks if Station is encountered for the first time.
2. Refreshes meta_network collection to reflect latest load meta data.
The current mongodb implementation does the following things daily -
The current process takes upwards of 24 hours to process approximately upto data states 10,000 GPS Stations. Daily re-computation of Historical data states is redundant and can be avoided.
A simple performance improvement would be compute only the end_date's data state and attach it to the mongodb collection everyday, instead of dropping and creating from scratch.
The text was updated successfully, but these errors were encountered: