ACLK new cloud architecture new TBEB

[underhood]

Summary

Parses error messages from cloud and honors backoff returned by cloud.
Implements new TBEB and uses parameters from env endpoint.

Component Name

ACLK-NG

Test Plan

• error messages can be tested by starting the agent a few times and killing it quickly. Eventually, the password endpoint will return 409 with 1-minute backoff. E.g. you will get rate limited by cloud for a minute or two.
• rest can be tested by checking parameters from cloud are used to do the bacoff

Additional Information

[vkalintiris]

Also, what does the TBEB acronym stands for?

[underhood]

TBEB is truncated binary exponential backoff although we diverted from exact definition of that algorithm :D I guess it is TEB now only or sth.

[stelfrag]

The connection is established, then after a while I kill the tcp connection and wait for the reconnection.

It seems to go into a loop

2021-04-23 10:24:27: netdata ERROR : ACLK_Query_4 : ACLK version negotiation failed. No reply to "hello" with "version" from cloud in time of 3s. Reverting to default ACLK version of 2.
2021-04-23 10:41:12: netdata ERROR : ACLK_Main : Connection Error or Dropped (errno 104, Connection reset by peer)
2021-04-23 10:41:12: netdata INFO  : ACLK_Main : Wait before attempting to reconnect in 0.000 seconds
2021-04-23 10:41:12: netdata INFO  : ACLK_Main : Attempting connection now
2021-04-23 10:41:13: netdata ERROR : ACLK_Main : SSL_read Err: Unknown!!! (errno 104, Connection reset by peer)
2021-04-23 10:41:13: netdata ERROR : ACLK_Main : Error reading or parsing response from server
2021-04-23 10:41:13: netdata ERROR : ACLK_Main : Couldn't process request
2021-04-23 10:41:13: netdata ERROR : ACLK_Main : Error trying to contact env endpoint
2021-04-23 10:41:13: netdata ERROR : ACLK_Main : Failed to Get ACLK environment
2021-04-23 10:41:13: netdata INFO  : ACLK_Main : Wait before attempting to reconnect in 1.293 seconds
2021-04-23 10:41:14: netdata INFO  : ACLK_Main : Attempting connection now
2021-04-23 10:41:14: netdata ERROR : ACLK_Main : SSL_write Err: Unknown!!! (errno 104, Connection reset by peer)
2021-04-23 10:41:14: netdata ERROR : ACLK_Main : Couldn't write HTTP request header into SSL connection
2021-04-23 10:41:14: netdata ERROR : ACLK_Main : Couldn't process request
2021-04-23 10:41:14: netdata ERROR : ACLK_Main : Error trying to contact env endpoint
 : 
 : 

trimmed message to avoid noise

[underhood]

@stelfrag I will have to investigate that.... seems not related to this PR... probably a bug in https client itself

we seem to be kicked from cloud-side right after connection for some reason:

2021-04-23 10:43:36: netdata INFO  : ACLK_Main : [mqtt_wss] I: ws_client: Websocket Connection Accepted By Server
2021-04-23 10:43:37: netdata INFO  : ACLK_Main : [mqtt_wss] I: ws_client: WebSocket server closed the connection with EC=1000. Without message.

[stelfrag]

Approving this as the reconnection error is not related to this PR.