[FLINK-15068][rocksdb] stop writing to RocksDB logs by default

[NicoK]

What is the purpose of the change

With Flink's default settings for RocksDB, it will write a LOG file (not the WAL, but pure logging statements) into the data folder. Besides periodic statistics, it will log compaction attempts, new memtable creations, flushes, etc. This file grows indefinitely and may fill the disk without this log being actually used anywhere (it will be deleted with the job anway).

With this PR, we are effectively disabling the log by default. If anyone wants to retain it, it can be re-configured at will providing an own OptionsFactory.

Brief change log

• change the default RocksDB configuration for all PredefinedOptions
  so that they use log level HEADER_LEVEL
• disable periodic statistics dumps to the LOG file
• hotfix the description of the DefaultConfigurableOptionsFactory

Verifying this change

I ran a Flink cluster with these changes and the LOG file now only contains some headers and is then never written to again. Normal behaviour is otherwise covered by existing tests.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? JavaDocs

---

This change is 

[flinkbot]

Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.

Automated Checks

Last check on commit 50314f6 (Thu Dec 05 12:01:26 UTC 2019)

Warnings:

• No documentation files were touched! Remember to keep the Flink docs up to date!

_{Mention the bot in a comment to re-run the automated checks.}

Review Progress

• ❓ 1. The [description] looks good.
• ❓ 2. There is [consensus] that the contribution should go into to Flink.
• ❓ 3. Needs [attention] from.
• ❓ 4. The change fits into the overall [architecture].
• ❓ 5. Overall code [quality] is good.

Please see the Pull Request Review Guide for a full explanation of the review process.

Details

The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot approve description to approve one or more aspects (aspects: description, consensus, architecture and quality)
• @flinkbot approve all to approve all aspects
• @flinkbot approve-until architecture to approve everything until architecture
• @flinkbot attention @username1 [@username2 ..] to require somebody's attention
• @flinkbot disapprove architecture to remove an approval you gave earlier

[StephanEwen]

Do we still want to put all options in the "predefined options"?

I would assume that from 1.10 on, the predefined options would not be used by default any more, because we configure the memory out of the box differently.

[NicoK]

From the order that I saw in current master:

• at first any PredefinedOption (if not set, then PredefinedOptions#DEFAULT), then (afterwards)
• any OptionsFactory (default: DefaultConfigurableOptionsFactory

Where does the memory management come in? This seems to be the same with #10416

[flinkbot]

CI report:

• 50314f6 : SUCCESS Build

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run travis re-run the last Travis build

[StephanEwen]

There is a discussion about adjusting this order, but for now, the change makes sense as it is.

+1, will merge this...