Make some options dynamically configurable

[hermanlee]

Issue by yoshinorim
Friday May 29, 2015 at 00:18 GMT
Originally opened as MySQLOnRocksDB#74

---

Some options like rocksdb_bytes_per_sync should be configurable dynamically.

[hermanlee]

Comment by jkedgar
Wednesday Sep 09, 2015 at 17:03 GMT

---

Does this mean that you want a global variable in MyRocks so that you can configure this option by using "Set rocksdb_bytes_per_sync = "?

What other options should be this way?

[hermanlee]

Comment by yoshinorim
Wednesday Sep 09, 2015 at 17:27 GMT

---

I'd like to start from dynamically configurable DB options. Static options such as wal_dir, rocksdb_data_dir are not targets. For CF/Table options, I think we need more discussions.

[hermanlee]

Comment by jkedgar
Wednesday Sep 09, 2015 at 17:31 GMT

---

Is there a list of dynamically configurable options?

[hermanlee]

Comment by yoshinorim
Wednesday Sep 09, 2015 at 17:39 GMT

---

^^ @siying @rven1

[hermanlee]

Comment by siying
Wednesday Sep 09, 2015 at 17:41 GMT

---

Search options.h and all dynamic configurable options will write in comments "dynamic changeable".

[hermanlee]

Comment by jkedgar
Wednesday Sep 09, 2015 at 18:07 GMT

---

The SetOptions() method appears to operate on a column family - the default column family if no column family is specified. Do I need to iterate through all the column families when an option is set via MySQL or do I just set it for the default column family?

The bytes_per_sync setting does not appear to be dynamically configurable. It is not listed as such in options.h and doesn't show up in util/options_helper.cc. Am I missing something?

[jkedgar]

I didn't receive any comments on my questions. The questions are again:

1. The request is for options like rocksdb_bytes_per_sync to be configurable via MySQL, but this option is not dynamically configurable (the only dynamically configurable options appear to be specified in rocksdb/util/options_helper.cc and 'bytes_per_sync' is not listed). Should I just ignore this and do all the other dynamically configured options?

2. The SetOptions() command in RocksDB is specific to a column family. Should the option change occur on just the default column family or should it occur on all column families associated with the database? I'm guessing that is should apply to all, but would like confirmation of that.

[hermanlee]

cc @yoshinorim

[yoshinorim]

1. Some parameters like Rate Limiter is not configurable via options, but has an interface at util/rate_limiter.h::SetBytesPerSecond(). So some special handling is needed to configure Rate Limiter settings. I hope there aren't many parameters like it.

2. Is CF handle needed to change DB options too? I think we should start from dynamic DB options. We won't need dynamic CF options very much. You'd better talk with Siying and Venky if there is no reasonable api to change DB options.

[jkedgar]

2. Maybe I'm not understanding, but the DB::SetOptions() method takes a column family as the first argument. There is a version that doesn't, but it just calls the other version with DefaultColumnFamily(). I'll also try to talk to Siying and Venkey.

[rven1]

DBOptions are not dynamically changeable. eg. bytes_per_sync and rate_limiter
Dynamic options per column family have to be changed for each column family.

[jkedgar]

So there are database-wide options that are not dynamically changeable and there are some column family options which are dynamically changeable. Here's a list of the options that I see that are dynamically configurable:

Memtable options:

• write_buffer_size
• arena_block_size
• memtable_prefix_bloom_bits
• memtable_prefix_bloom_probes
• memtable_prefix_bloom_huge_page_tlb_size
• max_successive_merges
• filter_deletes
• max_write_buffer_number
• inplace_update_num_locks

Compaction options:

• disable_auto_compactions
• soft_rate_limit
• hard_pending_compaction_bytes_limit
• hard_rate_limit (deprecated)
• level0_file_num_compaction_trigger
• level0_slowdown_writes_trigger
• level0_stop_writes_trigger
• max_grandparent_overlap_factor
• expanded_compaction_factor
• source_compaction_factor
• target_file_size_base
• target_file_size_multiplier
• max_bytes_for_level_base
• max_bytes_for_level_multiplier
• max_bytes_for_level_multiplier_additional (input is an array - difficult in MySQL)
• verify_checksums_in_compaction

Miscellaneous options:

• max_sequential_skip_in_iterations
• paranoid_file_checks

How many of these make sense to change via MySQL 'SET' statements? And should they be global across the entire database or should we have some way of setting them for different column families (indexes)?

[jkedgar]

After getting confirmation that I do have to update each column family to set any of these values as well as information that the rate limiter option is set-able using a different method, I've been thinking more about how this will work.

My assumption is that calling SET <variable> = <value> would not persist beyond the life of the server (i.e. rebooting the mysqld process would cause this information to be forgotten). Does this sound correct?

Also, if I think I probably need to keep track of the current set options for the database so that if a new column family is created (via a new index), I can set the options for this column family to match the rest of them. Does that sound correct?

[mdcallag]

In MySQL-land SET options only exist for the life of the process. MySQL
operations is happy with that as they avoid a mysqld restart thanks to SET,
and in the background do their magic to update my.cnf files

On Wed, Sep 30, 2015 at 9:28 AM, Jay Edgar notifications@github.com wrote:

After getting confirmation that I do have to update each column family to
set any of these values as well as information that the rate limiter option
is set-able using a different method, I've been thinking more about how
this will work.

My assumption is that calling SET = would not persist beyond the life of
the server (i.e. rebooting the mysqld process would cause this information
to be forgotten). Does this sound correct?

Also, if I think I probably need to keep track of the current set options
for the database so that if a new column family is created (via a new
index), I can set the options for this column family to match the rest of
them. Does that sound correct?

—
Reply to this email directly or view it on GitHub
#43 (comment).

Mark Callaghan
mdcallag@gmail.com

[jkedgar]

https://reviews.facebook.net/D48339