Export Import sst files #5135

vpallipadi · 2019-04-02T07:43:57Z

Refresh of the earlier change here - #3822
With the same basic idea. But, import API changed to import during a CF creation instead of arbitrary time (based on earlier feedback). Comments please @siying @ajkr

This is a review request for code change needed for - #3469
"Add support for taking snapshot of a column family and creating column family from a given CF snapshot"

We have an implementation for this that we have been testing internally. We have two new APIs that together provide this functionality.

(1) ExportColumnFamily() - This API is modelled after CreateCheckpoint() as below.
// Gets the live set of SST files for a particular Column Family in
// export_dir and metadata about the files in metadata.
// The SST files will be created through hard links when the directory is in
// the same partition as the db, copied otherwise.
// The directory should not already exist and will be created by this API.
// The directory will be an absolute path. Triggers a flush.
virtual Status ExportColumnFamilyFiles(
ColumnFamilyHandle* column_family,
std::vector* metadata,
const std::string& export_dir);

Internally, the API will DisableFileDeletions(), GetColumnFamilyMetaData(), Parse through
metadata, creating links/copies of all the sst files, EnableFileDeletions() and complete the call by
returning the list of file metadata.

(2) CreateColumnFamilyWithImport() - This API is modeled after IngestExternalFile(), but invoked only during a CF creation as below.
// CreateColumnFamilyWithImport() will import external SST files into a newly created CF as specified.
//
// (1) External SST files can be created using SstFileWriter.
// (2) External SST files can be exported from a particular column family in
// an existing DB.
virtual Status CreateColumnFamilyWithImport(
const ColumnFamilyOptions& /options/,
const std::string& /column_family_name/,
const ImportColumnFamilyOptions& /import_options/,
const std::vector& /metadata/,
ColumnFamilyHandle** /handle/);

Internally, this API creates a new CF, parses all the sst files and adds it to the specified column family, at the same level and with same sequence number as in the metadata. Also performs safety checks with respect to overlaps between the sst files being imported.

If incoming sequence number is higher than current local sequence number, local sequence
number is updated to reflect this.

Note, as the sst files is are being moved across Column Families, Column Family name in sst file
will no longer match the actual column family on destination DB. The API does not modify Column
Family name or id in the sst files being imported.

Introduce a new API ExportColumnFamily to export all the live sst files of a particular column family and metadata information to be used to import these sst files. This is part 1 of the change for #3469 " Add support for taking snapshot of a CF and creating a CF from a given CF snapshot" Partially Solves #3469

This change adds a new API to create a column family and importing sst files into the column family. The files imported can come from ExportColumnFamily API or can be created with SST file writer. This is part 2 of the change for #3469 " Add support for taking snapshot of a CF and creating a CF from a given CF snapshot" Partially Solves #3469

Add new unit tests for ExportColumnFamily() and CreateColumnFamilyWithImport() APIs. This is part 3 of the change for #3469 " Add support for taking snapshot of a CF and creating a CF from a given CF snapshot" Partially Solves #3469

siying

Oh wow, such a great job! It's so impressive. I gave a quick look and the basic workflow looks good to me and the quality of the code is great. I'll take a closer a look soon. Thank you for your contribution!

siying · 2019-06-04T19:10:13Z

We only accept PR to master branch. Can you rebase against the master and sent it to there? If you have concern about doing rebasing multiple times, you can do it after I gave it a careful pass.

siying

I think it looks very good to me. I look forward to seeing this one rebased against master and merge it there!

siying · 2019-06-04T20:49:01Z

include/rocksdb/db.h

+  // (1) External SST files can be created using SstFileWriter.
+  // (2) External SST files can be exported from a particular column family in
+  //     an existing DB.
+  // External files are deleted on a successful return.


Need to mention that some failure in the middle may end up with the column family created but is empty. But it will never end up with partial data loaded.

Also need to mention that the SST files are tried to be hardlinked (right? document whatever it is). Also mention that a failure can end up with part of files being renamed, and what data will be GC by DB reopening and the caller needs to be responsible to what files.

Idea was to cleanup column family in case of any errors. Earlier change was only doing DropCF on failure. Added DestroyCF as well now and that should ensure that there should not be any CF leftover. Added test to cover this as well.

Documented the external set file handling on success/failure cases.

siying · 2019-06-04T20:52:27Z

include/rocksdb/utilities/checkpoint.h

+  // - export_dir should be specified with its absolute path.
+  // - Always triggers a flush.
+  virtual Status ExportColumnFamily(ColumnFamilyHandle* handle,
+                                    std::vector<LiveFileMetaData>* metadata,


I believe Google C++ Style which we are following suggests the output argument to the end: https://google.github.io/styleguide/cppguide.html#Output_Parameters

siying · 2019-06-04T20:57:23Z

utilities/checkpoint/checkpoint_impl.cc

+  }
+
+  const auto final_nonslash_idx = export_dir.find_last_not_of('/');
+  if (export_dir[0] != '/' || final_nonslash_idx == std::string::npos) {


I don't remember whether we do it in other places or not. Do we always expect the leading char to be "/"? If not, I suggest we not take assumption here either.

siying · 2019-06-04T21:07:38Z

utilities/checkpoint/checkpoint_impl.cc

+        live_file_metadata.smallestkey = std::move(file_metadata.smallestkey);
+        live_file_metadata.largestkey = std::move(file_metadata.largestkey);
+        live_file_metadata.level = level_metadata.level;
+        metadata->emplace_back(live_file_metadata);


I don't think it makes a difference in performance in this case, but emplace_back() with the element type feels a little bit odd to me. If you don't mind, I suggest we change it to push_back().

siying · 2019-06-04T21:09:49Z

utilities/checkpoint/checkpoint_impl.cc

+  s = db_->GetEnv()->CreateDir(tmp_export_dir);
+
+  if (s.ok()) {
+    s = db_->Flush(rocksdb::FlushOptions(), handle);


Set FlushOptions.allow_write_stall = false in this case?

allow_write_stall = false is the default.

siying · 2019-06-04T21:19:53Z

utilities/checkpoint/checkpoint_test.cc

+    ASSERT_OK(checkpoint->ExportColumnFamily(db_->DefaultColumnFamily(),
+                                             &metadata, export_path));
+    verify_files_exported(2);
+    test::DestroyDir(env_, export_path);


I know we don't always follow this but a better place to do the cleaning up is in the destructor of the test class. The code here will not be executed if we hit an assert failure above. If it's not too hard for you to move to the destructor, it's worth doing.

siying · 2019-06-04T21:21:08Z

db/import_column_family_job.cc

+
+#ifndef __STDC_FORMAT_MACROS
+#define __STDC_FORMAT_MACROS
+#endif


I think we cleaned this up and included #include <cinttypes.h> in master now.

siying · 2019-06-04T21:24:35Z

include/rocksdb/db.h

+      const ColumnFamilyOptions& options, const std::string& column_family_name,
+      const ImportColumnFamilyOptions& import_options,
+      const std::vector<LiveFileMetaData>& metadata,
+      ColumnFamilyHandle** handle) = 0;


It's not a big deal but I'm thinking of whether there is a way to further prevent users from shooting their own shoe. The biggest risk of misuse of the API is that they set up a different comparator from the source data. If there a way to pass out the comparator or name of the comparator while exporting the data and pass it in here and do some sanity check?

Good idea. Added comparator name in export and a check for that on import.

siying · 2019-06-04T21:28:15Z

include/rocksdb/db.h

+      const ColumnFamilyOptions& options, const std::string& column_family_name,
+      const ImportColumnFamilyOptions& import_options,
+      const std::vector<LiveFileMetaData>& metadata,
+      ColumnFamilyHandle** handle) = 0;


It doesn't have to be included here but is there a way we support import the data to default column family, assuming it is empty? This feels an interesting idea too.

Thanks for the feedback. Will update and push against the master in next few days.

Yes. This can be useful. Will handle this separately.

Summary: Refresh of the earlier change here - #5135 This is a review request for code change needed for - #3469 "Add support for taking snapshot of a column family and creating column family from a given CF snapshot" We have an implementation for this that we have been testing internally. We have two new APIs that together provide this functionality. (1) ExportColumnFamily() - This API is modelled after CreateCheckpoint() as below. // Exports all live SST files of a specified Column Family onto export_dir, // returning SST files information in metadata. // - SST files will be created as hard links when the directory specified // is in the same partition as the db directory, copied otherwise. // - export_dir should not already exist and will be created by this API. // - Always triggers a flush. virtual Status ExportColumnFamily(ColumnFamilyHandle* handle, const std::string& export_dir, ExportImportFilesMetaData** metadata); Internally, the API will DisableFileDeletions(), GetColumnFamilyMetaData(), Parse through metadata, creating links/copies of all the sst files, EnableFileDeletions() and complete the call by returning the list of file metadata. (2) CreateColumnFamilyWithImport() - This API is modeled after IngestExternalFile(), but invoked only during a CF creation as below. // CreateColumnFamilyWithImport() will create a new column family with // column_family_name and import external SST files specified in metadata into // this column family. // (1) External SST files can be created using SstFileWriter. // (2) External SST files can be exported from a particular column family in // an existing DB. // Option in import_options specifies whether the external files are copied or // moved (default is copy). When option specifies copy, managing files at // external_file_path is caller's responsibility. When option specifies a // move, the call ensures that the specified files at external_file_path are // deleted on successful return and files are not modified on any error // return. // On error return, column family handle returned will be nullptr. // ColumnFamily will be present on successful return and will not be present // on error return. ColumnFamily may be present on any crash during this call. virtual Status CreateColumnFamilyWithImport( const ColumnFamilyOptions& options, const std::string& column_family_name, const ImportColumnFamilyOptions& import_options, const ExportImportFilesMetaData& metadata, ColumnFamilyHandle** handle); Internally, this API creates a new CF, parses all the sst files and adds it to the specified column family, at the same level and with same sequence number as in the metadata. Also performs safety checks with respect to overlaps between the sst files being imported. If incoming sequence number is higher than current local sequence number, local sequence number is updated to reflect this. Note, as the sst files is are being moved across Column Families, Column Family name in sst file will no longer match the actual column family on destination DB. The API does not modify Column Family name or id in the sst files being imported. Pull Request resolved: #5495 Differential Revision: D16018881 fbshipit-source-id: 9ae2251025d5916d35a9fc4ea4d6707f6be16ff9

siying · 2019-09-06T00:38:07Z

#5495 is already merged.

Summary: Refresh of the earlier change here - facebook#5135 This is a review request for code change needed for - facebook#3469 "Add support for taking snapshot of a column family and creating column family from a given CF snapshot" We have an implementation for this that we have been testing internally. We have two new APIs that together provide this functionality. (1) ExportColumnFamily() - This API is modelled after CreateCheckpoint() as below. // Exports all live SST files of a specified Column Family onto export_dir, // returning SST files information in metadata. // - SST files will be created as hard links when the directory specified // is in the same partition as the db directory, copied otherwise. // - export_dir should not already exist and will be created by this API. // - Always triggers a flush. virtual Status ExportColumnFamily(ColumnFamilyHandle* handle, const std::string& export_dir, ExportImportFilesMetaData** metadata); Internally, the API will DisableFileDeletions(), GetColumnFamilyMetaData(), Parse through metadata, creating links/copies of all the sst files, EnableFileDeletions() and complete the call by returning the list of file metadata. (2) CreateColumnFamilyWithImport() - This API is modeled after IngestExternalFile(), but invoked only during a CF creation as below. // CreateColumnFamilyWithImport() will create a new column family with // column_family_name and import external SST files specified in metadata into // this column family. // (1) External SST files can be created using SstFileWriter. // (2) External SST files can be exported from a particular column family in // an existing DB. // Option in import_options specifies whether the external files are copied or // moved (default is copy). When option specifies copy, managing files at // external_file_path is caller's responsibility. When option specifies a // move, the call ensures that the specified files at external_file_path are // deleted on successful return and files are not modified on any error // return. // On error return, column family handle returned will be nullptr. // ColumnFamily will be present on successful return and will not be present // on error return. ColumnFamily may be present on any crash during this call. virtual Status CreateColumnFamilyWithImport( const ColumnFamilyOptions& options, const std::string& column_family_name, const ImportColumnFamilyOptions& import_options, const ExportImportFilesMetaData& metadata, ColumnFamilyHandle** handle); Internally, this API creates a new CF, parses all the sst files and adds it to the specified column family, at the same level and with same sequence number as in the metadata. Also performs safety checks with respect to overlaps between the sst files being imported. If incoming sequence number is higher than current local sequence number, local sequence number is updated to reflect this. Note, as the sst files is are being moved across Column Families, Column Family name in sst file will no longer match the actual column family on destination DB. The API does not modify Column Family name or id in the sst files being imported. Pull Request resolved: facebook#5495 Differential Revision: D16018881 fbshipit-source-id: 9ae2251025d5916d35a9fc4ea4d6707f6be16ff9

Venki Pallipadi added 3 commits April 2, 2019 00:08

facebook-github-bot added the CLA Signed label Apr 2, 2019

siying reviewed Jun 4, 2019

View reviewed changes

vpallipadi mentioned this pull request Jun 21, 2019

Export Import sst files #5495

Closed

siying closed this Sep 6, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Export Import sst files #5135

Export Import sst files #5135

vpallipadi commented Apr 2, 2019

siying left a comment

siying commented Jun 4, 2019

siying left a comment

siying Jun 4, 2019

vpallipadi Jun 20, 2019

siying Jun 4, 2019

vpallipadi Jun 20, 2019

siying Jun 4, 2019

vpallipadi Jun 20, 2019

siying Jun 4, 2019

vpallipadi Jun 20, 2019

siying Jun 4, 2019

vpallipadi Jun 20, 2019

siying Jun 4, 2019

vpallipadi Jun 21, 2019

siying Jun 4, 2019

vpallipadi Jun 20, 2019

siying Jun 4, 2019

vpallipadi Jun 20, 2019

siying Jun 4, 2019

vpallipadi Jun 5, 2019

vpallipadi Jun 21, 2019

siying commented Sep 6, 2019

Export Import sst files #5135

Export Import sst files #5135

Conversation

vpallipadi commented Apr 2, 2019

siying left a comment

Choose a reason for hiding this comment

siying commented Jun 4, 2019

siying left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

siying commented Sep 6, 2019