Ingest Predefine Object

[中文版 (Chinese)]

Inject predefined objects

We described how to register a custom CompactionFilterFactory with the plugin system in CompactionFilterFactory As SidePlugin, however this is a bit cumbersome in many cases.

simplified method

At the beginning of the design of the side plugin configuration system, we considered this problem and designed the corresponding interface:

class SidePluginRepo {
public:
  // more ...
  void Put(const std::string& name, json spec, const std::shared_ptr<CompactionFilterFactory>&);
  void Put(const std::string& name, json spec, const std::shared_ptr<RateLimiter>&);
  void Put(const std::string& name, json spec, const std::shared_ptr<EventListener>&);
  void Put(const std::string& name, const char* spec, const std::shared_ptr<CompactionFilterFactory>&);
  void Put(const std::string& name, const char* spec, const std::shared_ptr<RateLimiter>&);
  void Put(const std::string& name, const char* spec, const std::shared_ptr<EventListener>&);
  // more ...
};

Using Put, users don't need to register their own CompactionFilterFactory/RateLimiter to the plug-in system, but directly inject the created CompactionFilterFactory/RateLimiter and other objects into the configuration library. In MyTopling, this Put is widely used, such as ha_rocksdb.cc:

    auto listener = std::make_shared<Rdb_event_listener>(&ddl_manager);
    g_repo.Put("rdb_listener", json{
      {"class", "Rdb_event_listener"},
      {"params", {"empty", "param"}},
    }, listener);
    db_opt->listeners.push_back(listener);
//---------------------
    rocksdb_db_options->rate_limiter = rocksdb_rate_limiter;
    g_repo.Put("rate_limiter", json{
      {"class", "GenericRateLimiter"},
      {"params", {"rate_bytes_per_sec", rocksdb_rate_limiter_bytes_per_sec}}
    }, rocksdb_db_options->rate_limiter);

The json spec parameter in Put is optional, its use is only more readable for web presentation. It must be noted that this injected object is eventually manually assigned to other configuration objects, and finally takes effect through direct or indirect references to it through db_opt or cf_opt, such as in the above code:

    // These two places are directly referenced by db_opt. Examples of indirect references are BlockCache referenced by BlockBasedTable
    db_opt->listeners.push_back(listener); // db_opt is rocksdb_db_options
    rocksdb_db_options->rate_limiter = rocksdb_rate_limiter;

MyTopling's Rdb_compact_filter_factory does not use this method to inject the configuration library, not because it cannot use this method, but because Rdb_compact_filter_factory needs to support serialization. To support serialization, it must be registered. Now that it is registered, use the standard configuration Way.

Express json using strings

As can be seen from the previous Put overload, in addition to passing json objects, json strings can also be passed. In kvrocks ToplingDB, we use this overload:

repo_.Put("metadata",
  R"({"class": "MetadataFilterFactory", "params": {"storage": "this"}})",
  shared_ptr<CompactionFilterFactory>(make_shared<MetadataFilterFactory>(this)));
repo_.Put("subkey",
  R"({"class": "SubKeyFilterFactory", "params": {"storage": "this"}})",
  shared_ptr<CompactionFilterFactory>(make_shared<SubKeyFilterFactory>(this)));
repo_.Put("listener",
  R"({"class": "EventListener", "params": {"storage": "this"}})",
  shared_ptr<rocksdb::EventListener>(make_shared<EventListener>(this)));

Using this overload has two advantages:

1. The json expressed by the string is a standard json syntax, which is more readable than the json expressed by the C++ initializer list
   • The downside is that json syntax checks are delayed until runtime, and exceptions need to be caught when necessary
2. You can avoid #include json header files, just #include <topling/side_plugin_repo.h>
   • At this time, class json is just a pre-declaration, not defined, and the compilation speed can be accelerated by a little bit.

Applicable scene

Code Migration: Migrating from RocksDB to ToplingDB, this way the code changes are smaller.

Object construction is too complicated: In this way, we use the object construction code as a black box.