Full/per-vhost dynamic (re)configuration: gRPC API

[krizhanovsky]

Introduction

Current file-based reconfiguration is a huge pain:

1. it's hard to add/update single server group or vhost if there are already thousands of server groups and/or vhosts - we have to traverse them all to understand what differs. File configuration update always means full configuration update and we need to understand how it differs from the previous configuration.
2. There is no good solution for a configuration update - if a part of live reconfiguration update fails, then it's hard to rollback previous changes.
3. File configuration isn't suitable for a distributed environment: apply the same configuration change to all the nodes or configure a node remotely. If we update a configuration remotely (e.g. via REST API), then we have to update the configuration file somehow and it's painful to keep comments and formatting.
4. Automatic configuration generation and validation would be a problem - the configuration language should have a grammar and allow schema-less configuration updates like Envoy.

At this point, it's clear that the whole configuration, maybe except quite a few properties (e.g. path to TDB files) necessary to start Tempesta, must be dynamically reconfigurable. Rgardles whether a particular option is dynamically reconfigurable or not, the whole configuration must be stored in TDB persistent tables, so depends on #516. No current file configuration is required.

Thus we have to move to TDB-handled configuration from the file.

Reconfiguration

We need to provide gRPC/Flatbuffer (the most efficient one) API for the configuration updates. The API must be implemented by a rest module which processes the HTTP request and fetches the reconfiguration string which is dispatched (as part of parsed HTTP message via kernel-user space transport, #77) to a user space daemon tfw_mgr.

The rest module must provide strong authentication of all the ingress requests. Initially it could be done based on ingress physical interface and IP addresses checking specified in the configuration.

The tfw_mgr daemon parses Flatbuffer and calls libtdb API to update appropriate table in TDB. The updates must be done in transactional context (#516) to be able to rollback the change on any problem. TDB must also implement events notification (callbacks), also required fro #516. TL programs must be compiled also in the user space using a separate parser. The daemon can also make any compilation of user friendly configuration into low-level representation (see for example #1105).

Different TDB tables must be used for the configuration handling. At least separate tables for vhosts and server groups are required since these are the place for multiple configuration entities. Moreover, currently server groups are stored in sg_hash - a perfect candidate for a migration to TDB HTrie.

The control flow for the configuration update is following (everything works in process context of tfw_mgr):

1. tfw_mgr starts a transaction and execute all the updates on all the tables
2. tfw_mgr commits transaction and TDB calls trx_prepare hooks on table updates
3. trx_prepare hooks call routines of all the modules owning updated configuration tables and the modules do all necessary preparatin (e.g. memory allocations) and validation of the changes. They do not affect current configuration.
4. Next trx_commit hook is called in all trx_prepare returned 0 and all the modules apply changes to their configuration.
5. If some of the trx_prepare hooks fails, then trx_rollback hooks are called which free all allicated resources.

The user space configuration functionality must be implemented in a separate library libcfg, whcih is called by tfw_mgr.

Basically, it seems we don't need transactions from Tempesta DB and just can update all the entries in RCU faschion. But the reconfiguration process on it's own really needs something like 2PC protocol (described at the above).

libcfg

The main configuration parsing and TDB actions (using libtdb) must be done in libcfg. The library is used by the configuration daemon tfw_mgr and CLI tool tfw.

The configuration parsing is done in user space, se nft also should be called from the table to tread multi-layer firewalling rules (see #862 (comment)).

Examples

Possible request to load new filter rules could look like

    POST /rest/filter?load HTTP/1.1
    Host: tempesa
    Content-Type: application/x-www-form-urlencoded
    Content-Length: N

    block=IP%3D1%2E1%2E1%2E1%26URL%7Efoo&block=URL%3Dfoo.com

This request adds two blocking rules:

1. block clients with IP=1.1.1.1 (IP%3D1%2E1%2E1%2E1) and URL containing (~) foo;
2. block all requests with URL equal (=) to foo.com

PUT, DELETE and other methods also should be used.

TL (#102) programs must be stored in separate table, probably depending on their context (e.g. req or resp processing contexts).

    GET /rest/stats?perf HTTP/1.1
    Host: tempesa

System statistics must be returned by Tempesta FW in JSON document.

Interoperability

Migration from the file

Sometimes file configuration might be handy and it makes sense to move to the new configuraion scheme smoothly. So tempesta.sh (using a helper script) should

1. parse the configuration file
2. remove options supporting the new interface
3. run Tempesta with minimal configuration
4. executes the script to translate the config file to JSON and execute the REST API for configuration
5. run tfw tool to update the configuration via libcfg

File visibility

tfw should implement more user friendly configuration interface than tdbq working with raw TDB data representation. In particular, it must implement show option which print the whole configuration with all necessary identation is single text representation.

It also must show current active configuration (#862 (comment)).

Initialization

Current tfw_init() logic remains the same, but tfw_start() abandons tfw_cfg_parse() - instead of reading and parisng the file, Tempesta reads TDB configuration tables and execute trx_prepare and trx_commit hooks on all modules and tables. Next, either if we loaded full configuration or all the tables are empty, Tempesta waits for TDB events (callback, #516) on configuration table. On the initial run, when all the tables are empty, tfw loads configuration from the configuration file, JSON, or interactive user commands.

Revision of configuration consistency

There are claims regarding consistency of current configuration options, e.g. #862 (comment) :

Historically there wasn't vhost and location directives and srv_group directive included some options that actually describe web service behaviour. This options are about HTTP processing features:

server_forward_retries
server_forward_timeout
server_retry_nonidempotent

While others describe the group of servers as single load-balaned array of server connections and 'low-level' connection and load balancer properties:

server_connect_retries
server_queue_size
sched
sticky_sessions

It seems reasonable to move options from the first group to vhost settings.

Need to review all the configuration directives and stabiles them. Future additions must extend but not change configuration format. Tempesta's upgrade to the newer version should be possible without any issues.

.htaccess

tdbfs can provide directory based per-vhost configuration with full user/group access validation (thanks to OS filesystem inteface), so we can provide configuration overriding for non-priveledged users, which is available in Apache HTTPD only at the moment and which seems the most influential reason for people to stay with the server.

References

• F5 BIG-IP configuration on top of IMDB

• Nginx Crossplane

[krizhanovsky]

This should firstly be implemented for HTTPtables only, which relates to Tempesta Language #102 and Tempesta xFW volumetric DDoS mitigation module.

[krizhanovsky]

As an update: Envoy allows both the file and on-the-fly gRPC reconfiguration, which both cover more or less the same features, but gRPC online reconfiguration isn't stored as file - any persistent configuration storage is left for a user and typically Redis is used for this. Probably we also shouldn't go with the over-engeenering to store dynamic reconfiguration in a file