Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/dual engines #51

Merged
merged 4 commits into from Apr 2, 2019

Conversation

Projects
None yet
3 participants
@mildbyte
Copy link
Contributor

mildbyte commented Apr 2, 2019

Allow using two engines throughout the API:

  • metadata engine: manages the images/objects/tables/tags tables (Splitgraph image metadata). Essentially, all writes/reads to splitgraph_meta tables go to the metadata engine (apart from object_cache_status and cached objects)
  • object engine: is responsible for object storage, checking fragments out, creating fragments, audit triggers and layered querying.

By default, both engines are the same but can be overridden at Repository object creation. The rough places where each engine is used:

  • ObjectManager mostly uses the object engine (including for cache occupancy calculations where it uses the physical table sizes of cached objects rather than querying the objects table)
  • FragmentManager uses the metadata engine to resolve/register fragments and the object engine to store/apply them
  • MetadataManager uses the metadata engine throughout
  • Repository mostly uses the object engine (e.g. to find out which tables have changed schemas etc) apart from actually recording new image metadata/importing images.

Only reading with two engines (e.g. running an LQ with the metadata engine + a different object engine) is supported and tested; writing for now isn't.

Also, speed up object fetching by maintaining a table with cache occupancy instead of recalculating it at every download.

@coveralls

This comment has been minimized.

Copy link

coveralls commented Apr 2, 2019

Coverage Status

Coverage increased (+0.02%) to 94.781% when pulling 1d9e305 on feature/dual-engines into 4be78a1 on master.

@mildbyte mildbyte force-pushed the feature/dual-engines branch from c8ccaac to ac12b68 Apr 2, 2019

@mildbyte mildbyte requested a review from milesrichardson Apr 2, 2019

mildbyte added some commits Apr 2, 2019

Allow using two engines throughout the API:
  * metadata engine: manages the images/objects/tables/tags tables (Splitgraph image metadata). Essentially, all writes/reads to splitgraph_meta tables go to the metadata engine (apart from object_cache_status and cached objects)
  * object engine: is responsible for object storage, checking fragments out, creating fragments, audit triggers and layered querying.

By default, both engines are the same but can be overridden at Repository object creation. The rough places where each engine is used:

  * ObjectManager mostly uses the object engine (including for cache occupancy calculations where it uses the physical table sizes of cached objects rather than querying the objects table)
  * FragmentManager uses the metadata engine to resolve/register fragments and the object engine to store/apply them
  * MetadataManager uses the metadata engine throughout
  * Repository mostly uses the object engine (e.g. to find out which tables have changed schemas etc) apart from actually recording new image metadata/importing images.

Only reading with two engines (e.g. running an LQ with the metadata engine + a different object engine) is supported and tested; writing for now isn't.
Fix multiengine LQ tests by making sure we don't hold open transactio…
…ns (object cache occupancy update can lock up)

@mildbyte mildbyte force-pushed the feature/dual-engines branch from ac12b68 to 1d9e305 Apr 2, 2019

@mildbyte mildbyte merged commit 9304aad into master Apr 2, 2019

3 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
coverage/coveralls Coverage increased (+0.02%) to 94.781%
Details

@mildbyte mildbyte deleted the feature/dual-engines branch Apr 14, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.