Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Encode keys for records in mongoDB #492

Closed
pld opened this issue Mar 15, 2013 · 1 comment
Closed

Encode keys for records in mongoDB #492

pld opened this issue Mar 15, 2013 · 1 comment
Assignees
Milestone

Comments

@pld
Copy link

pld commented Mar 15, 2013

In the interests of saving space, we can encode the observation record keys instead of "[some column name]" we could use "1", etc.

This would also require mapping queries, to the coded keys, and mapping the coded keys to the column names on their way into a dframe (we can pass columns= in the dframe constructor).

This assumes we are not planning on moving to a column store (which seems to be true).

@ghost ghost assigned pld Mar 24, 2013
@pld
Copy link
Author

pld commented Mar 24, 2013

A couple more things:

  1. this is a change to the way we store data that will require migrating or clearing any current databases
  2. in observations we can remove the reserved key dataset_observation_id, it fulfills the same function as dataset_id
  3. we can tie all our batch* functions closer to the Observation model, we do not expect anyone else to use these (with the potential exclusion of batch_read)
  4. any encoding/decoding should happen at the batch level, to isolate it from the rest of the system (feels like a completely independent mix-in could be in order)

WIP here, modilabs:master...modilabs:encode-492

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant