Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update emilia's mongo to support new location representation #103

philipmjohnson opened this issue Apr 15, 2018 · 4 comments

Update emilia's mongo to support new location representation #103

philipmjohnson opened this issue Apr 15, 2018 · 4 comments


Copy link

@philipmjohnson philipmjohnson commented Apr 15, 2018

Please do work for this task in a branch called issue-103.

Update emilia's Mongo database to associate location slugs with Trend and Event data.

Update all existing data to provide a location slug, and change processing to associate the location slug for all future data.

Copy link

@anthonyjchriste anthonyjchriste commented Apr 18, 2018

I can do this as soon as @sergey-negrashov starts saving the new location slugs in the DB. That way, when we convert all the exisiting data, there is no gap between existing data and new data where the slug doesn't exists.

I have a couple of questions:

  1. In the context of existing data, it seems that we should only add a slug when "legacy location" data (i.e. the "locations" field in "opq_boxes" is known correct? By that, I mean, we have trends and events associated with time ranges that don't have a legacy location associated with them. It seems to me that it would be academically dishonest to associate a location slug with a document when we don't really have any data supporting that that document came from a particular location.

  2. Are we removing the "locations" field from "opq_boxes", or will we simply store location slugs and their start times here moving forward? It seems to me that it would be nice to have a history of the boxes location over time without having to query trends or events for that information.

  3. Where is the proper place to store the zipcode field? Should that maybe be added to the locations collection?

Copy link
Contributor Author

@philipmjohnson philipmjohnson commented Apr 18, 2018

Here are my thoughts:

First, the new data model for OPQ_Boxes has a location_archive field:

This is where we can store legacy location information.

Second, yes, we are removing the locations field. Instead, we'll represent the current location via two fields: location and location_start_time_ms. Then, we'll use the location_archive field for legacy location data. This seems much faster and better than the old way of looking at the last element of an array to find the current location. Here's the location data model description:

Third, I am proposing that we no longer explicitly represent zip codes. They are kind of artificial. Instead, I am proposing that all location definitions must have an array containing [longitude, latitude]. If you want zip codes, you can either create a "region" manually, or use some crazy mongo plugin to map from [longitude, latitude] to zip codes. I think we can use the "region" entity to do zip codes, towns, neighborhoods, anything we like.

Here's the region data model description:

@philipmjohnson philipmjohnson moved this from Backlog to In Progress in UH Agile Power Monitoring Project Apr 20, 2018
@sergey-negrashov sergey-negrashov self-assigned this Apr 26, 2018
Copy link
Contributor Author

@philipmjohnson philipmjohnson commented May 10, 2018

  1. Mongo will be updated with historical info at the same time that oplogs are put online.
  2. Locations are not associated with Measurements, but only with Trends, Anomalies, etc.
  3. Access the DB to find the current location of the OPQ Box each minute when you create the Trend.
  4. Add Locations to Anomalies.
  5. Where the heck is Watanabe Hall.
  6. Philip: fix the bug in unplugged.
Copy link
Contributor Author

@philipmjohnson philipmjohnson commented May 31, 2018

There remains some old measurements, events, and trends with old location representation. But who cares. We're done.

@philipmjohnson philipmjohnson moved this from In Progress to Completed in UH Agile Power Monitoring Project May 31, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.