cfg: hot configuration reloading

[vdmit11]

We need to change some configuration options while Tempesta FW is running.

For example, we want to add a new back-end server or tune DoS protection settings, but in the same time we don't want to stop serving client connections even for a short time.

Therefore:

1. The reloading feature should be added to the Configuration Framework. It should listen a "reload" event via sysctl, and then re-parse the configuration file and push the data to Tempesta moduels without stopping the whole system.
2. Perhaps we would not like to support the feature for all configuration options. Often it is better to have immutable configuration data in runtime and don't care about synchronization. Hence, the feature must be optional, and only some of configuration options will support it.
3. All existing code that requires the feature should be updated. At least, the following three modules should be updated:
   • 'sock_backend' - need to change list of back-end servers without stopping the sytem
   • 'sched_http' - need to change load-balancing rules
   • 'req_conn_limit' - need to be able to tune limits on a running system

[krizhanovsky]

Basically, the task can be easily done by the same workflow as currently with several adjustments. We can describe the workflow by following scenario:

1. update the configuration file;
2. run sysctl -w net.tempesta.state=reconfig
3. cfg.c parses the new config and calls modules TfwCfgSpec->handler() or a new special reconfiguration callback;
4. the modules must be responsible to compare the new TfwCfgEntry with current configuration and possible run-time reconfiguration;
5. if a module finds that a configuration entry is changed, but it's unable to reload configuration in run time, then it returns an error.

The reworked cfg.c (and/or TfwCfgSpec of all the modules) must provide variables foundation for Tempesta Language. TL variables should be easily bound with TfwCfgSpec and writing to them should just call the callbac from step (3) of the scenario above.

The feature is required now for large cloud installations with large and dynamic set of hosted VMs, so with #680 in mind at least server schedulers must be able to reconfigure in run-time. Typically, it should be done with RCU help: create a new scheduler data structures with existing server connections and change the pointers. In particularly we in this issue at least following dynamic reconfiguration must be possible:

1. add/delete/update a rule for HTTP scheduler;
2. add/delete/update a server group;
3. add/delete/update a server inside a group;

Typically host deletion should be graceful. I.e. if we delete a VM, then it's already migrated to some other place or we just need to do some maintenance on it. Thus if a server is deleted from the configuration, but is still alive, then let all current sessions to finish, but do not schedule new requests to the server. Since sessions can be almost infinite, a new configuration option grace_shutdown_time which specifies maximum time in seconds which we should wait for sessions closing for deleted server. If the time passes, then we should terminate all the server's connections`like we do so for system shutdown.

Also please separate sysctl logic from cfg.c and move it to main.c as it's mentioned in TODO in cfg.c.

The issue requires to update code of requests schedulers, so the issue should be relates to #76.

[vankoven]

closed via 3f49ada