Layered checkout write support #662
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…en the present and latest data Still need to investigate edge cases, e.g. deletion of just inserted rows not working.
In particular ensure that deletion followed by insertion does not result in return of any previous insertions, i.e. only update the merged view with records originating after the latest delete command (if any).
…nd plain tables at the same time
Also, remove the obsolete context manager in singer ingestion.
Also make all_tables optionally omit overlay component tables
We don't allow altering table schemas in layered querying mode.
Due to the missing column defaults info in our images we can't replicate these on the upper table, and so must explicitly fill each column in INSERT statements, otherwise they'll be a silent no-op.
Since the offending section was obsolete and removed in the latest version of black, for more details see: psf/black#2964.
…om information_schema
In addition, do not owerwrite FDW server when instantiating LQ checkout
In this way we can decouple LQ servers for checkout of different images in the same repo.
mildbyte
reviewed
Apr 1, 2022
- move the central init method to a separate file in core - add None return type to a helper protected method - fix imports
Make sure to analyse the latest object in the asserts.
mildbyte
reviewed
Apr 4, 2022
- Skip temo table in favor of object + rename - Remove CASCADE from deletion of foreign tables in engine
By making sure that for deleted rows the non-PK fields are NULL-ified.
Ensure table records/objects remain intact after commit if they were not written to.
|
LGTM! great work :) |
mildbyte
approved these changes
Apr 5, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds support for writes to the layered querying checkout mode, as a basis for providing writable DDN.
The founding idea is to construct a overlay view with two supporting tables:
The users interact with the view directly through read/write statements, and then for reads the view merges existing, non-changed data (from the lower, foreign table as before), with the changes stored in the new "staging" (aka upper) table where all pending writes are written.
Upon committing, new object/image is created, the lower fdw table is bumped to the new image, while the upper table is truncated, thus effectively persisting all pending changes at that moment.
CU-21phdxv