Add metadata time traveling #83
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Recap now supports time traveling when browsing the catalog. Users can provide
an `--as-of` timestamp to see what databases, tables, schemas, and metadata
looked like at a point in time. Let's look at some CLI examples!
List tables as of 2022-12-01:
recap catalog list \
/databases/postgresql/instances/localhost/schemas/public/tables \
--aa-of '2022-12-01'
Read the metadata for `table` as it looked on 2022-12-10 01:01:01:
recap catalog read \
/databases/postgresql/instances/localhost/schemas/public/tables/table \
--as-of '2022-12-10 01:01:01'
Search for all table metadata in `some_db` as it looked on 2022-10-12 04:23:43:
recap catalog search \
"json_extract(metadata, '$.location.schema') = 'some_db'" \
--as-of '2022-10-12 04:23:43'
Of course, these changes flow through to AbstractCatalog, DatabaseCatalog, and
RecapCatalog as well. I also had to update the FastAPI server to support
`as_of` query parameters.
The FilesystemCatalog is well and busted right now. I'll need to fix that or
ditch it. Saving that for a subsequent PR.
criccomini
added a commit
that referenced
this pull request
Jan 10, 2023
I've removed FilesystemCatalog. The time travel changes in #83 have complicated things on the filesystem. I thought of a few ways to implement this using clever directory structures, but I don't think it's worth it right now. I'd rather spend time on other things. The benefit of removing this is it means even fewer dependencies. No `fsspec` or `jq` needed.
Merged
criccomini
added a commit
that referenced
this pull request
Jan 10, 2023
I've removed FilesystemCatalog. The time travel changes in #83 have complicated things on the filesystem. I thought of a few ways to implement this using clever directory structures, but I don't think it's worth it right now. I'd rather spend time on other things. The benefit of removing this is it means even fewer dependencies. No `fsspec` or `jq` needed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Recap now supports time traveling when browsing the catalog. Users can provide an
--as-oftimestamp to see what databases, tables, schemas, and metadata looked like at a point in time. Let's look at some CLI examples!List tables as of 2022-12-01:
Read the metadata for
tableas it looked on 2022-12-10 01:01:01:Search for all table metadata in
some_dbas it looked on 2022-10-12 04:23:43:Of course, these changes flow through to AbstractCatalog, DatabaseCatalog, and RecapCatalog as well. I also had to update the FastAPI server to support
as_ofquery parameters.The FilesystemCatalog is well and busted right now. I'll need to fix that or ditch it. Saving that for a subsequent PR.