Change array schema format for schema evolution #2258
Conversation
…it test with ratio 0.7523
…it test with ratio 0.7523
| RETURN_NOT_OK(get_array_schema_uris(array_uri, &uris)); | ||
| if (uris.size() == 0) { | ||
| return LOG_STATUS(Status::StorageManagerError( | ||
| "Can not get the latest array schema; Empty array schemas.")); |
There was a problem hiding this comment.
Please change comment to Cannot get the latest array schema; No array schemas found.
| // Check if the schema directory exists or not | ||
| bool is_dir = false; | ||
| // Since is_dir could return NOT Ok status, we will not use RETURN_NOT_OK here | ||
| vfs_->is_dir(uri.join_path(constants::array_schema_folder_name), &is_dir); |
There was a problem hiding this comment.
catch the return status and modify if (is_dir) { to if (!st.ok() || is_dir) {
There was a problem hiding this comment.
changed to if(!st.ok() || is_dir). but not very sure if we should return true when status is not ok,
There was a problem hiding this comment.
LGTM, thanks. I'd like @Shelnutt2 to take another look as well and see how the changes might impact the REST performance in TileDB Cloud (e.g., in the various places where we retrieve the array schema).
re @Shelnutt2 |
There was a problem hiding this comment.
@bdeng-xt I've left some comments here. Lets also schema a few minutes to discuss the next steps. This is a great change to add the initial format changes. Next I believe we need to support loading multiple schemas and passing the appropriate one to each fragment in a read based on the fragment metadata's listed array schema.
|
@bdeng-xt if you can rebase this so the new CI checks run then we can get this merged! |
Schema evolution is a feature for allowing an array schema to charge over time. To implement that, we will use the same system used for fragments and metadata. The array schemas will be moved into the folder __schema with the format of timestamp_timestamp_uuid. The details can be found in our "Schema Evolution" design paper.
TYPE: FEATURE
DESC: Store array schemas under __schema directory