Load test suite #819

mohierf · 2017-05-20T10:56:46Z

Add a test suite for load tests
Cleanings in the inter-daemon communication
- catch connection errors and connection timeout as separate exceptions in the HTTP client
- cleanings in the SatelliteLink with those exceptions
- add an instance id in the receiver daemon (missing to get broks from this daemon)
Cleanings in the scheduler loop
- replace the handle_requests/sockets waiting function with a sleep
- raise WARNING logs when the scheduler loop is over 1 second
- add some counters and statistics
- log counters when TEST_LOG_LOOP environment variable is set
Some small fixes
- closes Actions never come back from poller/reactionner #817
- closes Erroneous what_i_managed daemon interface #823

Initially this PR was intended to create a load test suite that runs many checks for a lot of hosts but this raised some problems in the inter-daemon communication that were needed to get fixed!

mohierf · 2017-06-04T06:39:11Z

I just rebased on develop to fix the conflicts

ddurieux · 2017-06-01T08:25:12Z

alignak/daemon.py

+        :param timeout: timeout to wait for activity
+        :type timeout: float
+        :return:Returns a 2-tuple:
+        * first value is the time spent for the time change chekc


typo: check instead chekc

ddurieux · 2017-06-01T08:27:25Z

alignak/daemon.py

+        after = time.time()
+
+        if after - before > timeout:
+            return after - before, time_changed


I prefer store the after - before in a variable to not have to calculate the same thing many times

The compiler does it for you 😉

ddurieux · 2017-06-01T08:30:19Z

alignak/daemon.py

+        if after - before > timeout:
+            return after - before, time_changed
+        # Time to sleep
+        time.sleep(timeout)


It's not better to do a sleep of timeout - (after - before) ?

It is a very slight difference, but yes...

ddurieux · 2017-06-01T08:35:50Z

alignak/daemons/brokerdaemon.py

+            con = link['con'] = HTTPClient(uri=link['uri'],
+                                           strong_ssl=link['hard_ssl_name_check'],
+                                           timeout=timeout, data_timeout=data_timeout)
+        except HTTPClientConnectionException as exp:  # pragma: no cover, simple protection


If you have exceptions in create connection, so we need define link['con'] = None in all exceptions here

You need add it to all except HTTP.... in all modifications, because we have same thing in many files in your PR

Hmmm... not sure about this. I am looking... breaking the connection when a bad connetion or time out happens is not necessarily a good idea!

ddurieux · 2017-06-01T08:43:34Z

alignak/daemons/receiverdaemon.py

+
+            is_active = sched['active']
+            if not is_active:
+                logger.warning("The scheduler '%s' is not active, it is not possible to get broks "


The message must be it's not possible to push external commands

ddurieux · 2017-06-05T19:46:08Z

alignak/satellite.py

@@ -149,6 +151,120 @@ def get_external_commands(self):
        self.external_commands = []
        return res

+    @staticmethod
+    def is_connection_try_too_close(link, delay=5):


The function name not very explicite :(

It is not me that defined this ... I kept using a former existing function :/

ddurieux · 2017-06-05T19:49:49Z

alignak/satellite.py

+
+    def do_pynag_con_init(self, s_id, s_type='scheduler'):
+        """Initialize a connection with scheduler having 'uuid'
+        Return the new connection to the scheduler if it succeeded,


The return is always true, so make description like set the connection if it succeeded, otherwise set it to None.

ddurieux · 2017-06-05T19:50:02Z

alignak/satellite.py

+
+        :param s_id: scheduler s_id to connect to
+        :type s_id: int
+        :return: scheduler connection object or None


The return is always true

ddurieux · 2017-06-05T19:53:06Z

alignak/satellite.py

+                    self.pynag_con_init(sched_id)
+                    sched['con'] = con
+
+                if con:


You can simplify by use else:

Hmm... you found out a bug 😉
It should be: con = sched['con'] ... I did not catched this one!

ddurieux · 2017-06-05T20:11:32Z

alignak/scheduler.py


        :param check: check to add
        :type check: alignak.check.Check
        :return: None
        """
        if check is None:
            return
+        if check.uuid in self.checks:
+            logger.debug("Already existing check: %s", check)


If we have it, it's not normal, so prefer use error or warning instead debug.
Same for actions, event_handler...

No, it is not a bug... if you set this at WARNING level, you will be spammed on each scheduler start/restart/new configuration ! This is why I left the DEBUG log level...

mohierf · 2017-06-09T07:26:38Z

I am preparing a small refactoring of the Satellite/BaseSatellite classes ...

mohierf · 2017-06-09T13:58:12Z

@ddurieux: I will merge this PR by myslef after having launched the tests on a more powerful server and after some more checks in the volume of logs produced by each daemon

coveralls · 2017-07-20T07:43:50Z

Coverage decreased (-0.9%) to 84.739% when pulling 9d96008 on load-test-suite into 38b183e on develop.

…anaged

coveralls · 2017-07-20T18:35:04Z

Coverage decreased (-0.8%) to 84.779% when pulling 9a14a86 on load-test-suite into 38b183e on develop.

coveralls · 2017-07-20T19:26:54Z

Coverage decreased (-0.9%) to 84.739% when pulling 9a14a86 on load-test-suite into 38b183e on develop.

coveralls · 2017-07-21T06:44:49Z

Coverage decreased (-0.9%) to 84.732% when pulling 40d0084 on load-test-suite into 38b183e on develop.

coveralls · 2017-07-24T11:30:21Z

Coverage decreased (-0.7%) to 84.91% when pulling a74eaea on load-test-suite into 38b183e on develop.

coveralls · 2017-07-24T12:32:57Z

Coverage decreased (-0.8%) to 84.769% when pulling cf1a341 on load-test-suite into 38b183e on develop.

coveralls · 2017-07-26T06:16:32Z

Coverage decreased (-0.9%) to 84.723% when pulling 82b479a on load-test-suite into 38b183e on develop.

…tion

coveralls · 2017-07-26T12:28:39Z

Coverage decreased (-0.9%) to 84.731% when pulling 13c7311 on load-test-suite into 38b183e on develop.

…mons

coveralls · 2017-07-26T16:36:10Z

Coverage decreased (-0.7%) to 84.886% when pulling 3c28635 on load-test-suite into 38b183e on develop.

coveralls · 2017-07-26T18:35:43Z

Coverage decreased (-0.7%) to 84.894% when pulling 59ab106 on load-test-suite into 38b183e on develop.

coveralls · 2017-07-26T19:05:43Z

Coverage increased (+8.2%) to 93.825% when pulling 59ab106 on load-test-suite into 38b183e on develop.

mohierf · 2017-08-01T07:12:02Z

As discussed with @ddurieux I merge this PR. Many uninterrupted days of tests with this source code on several servers make it considered as stable enough

mohierf requested a review from ddurieux May 20, 2017 10:56

mohierf changed the title ~~Load test suite~~ [WiP] Load test suite May 20, 2017

mohierf force-pushed the load-test-suite branch 9 times, most recently from beeec8e to 0eaf884 Compare May 27, 2017 07:48

mohierf force-pushed the load-test-suite branch 3 times, most recently from f31ff21 to bddc73d Compare May 29, 2017 08:10

mohierf force-pushed the load-test-suite branch from bddc73d to 7c96d9e Compare June 4, 2017 06:38

ddurieux suggested changes Jun 5, 2017

View reviewed changes

mohierf mentioned this pull request Jun 8, 2017

scheduler does not start with "enable_environment_macros=1" #845

Closed

ddurieux approved these changes Jun 9, 2017

View reviewed changes

mohierf force-pushed the load-test-suite branch from 1e440ab to 7f91c8c Compare June 9, 2017 13:10

Alignak-monitoring deleted a comment from coveralls Jun 9, 2017

mohierf added 5 commits July 20, 2017 20:14

Manage failed workers correctly

c93dd6d

Fix #861: remove unuseful code

b7fc523

Fix potential error when scheduler stops before receiver

3df5e68

Restore test with 100 hosts

ea440d5

Closes #867: no scheduler queue cleaning when no hosts/services are m…

b8fee58

…anaged

mohierf force-pushed the load-test-suite branch from 9d96008 to b8fee58 Compare July 20, 2017 18:14

Set poller at INFO log level for load tests

9a14a86

Set poller at INFO log level for load tests and run test for 5 minutes

40d0084

Fixes #869 and clean dispatching logs

cf1a341

mohierf force-pushed the load-test-suite branch from a74eaea to cf1a341 Compare July 24, 2017 11:54

Fixes #873: exception catching in put_results

82b479a

Change log when daemon got interrupted before receiving its configura…

13c7311

…tion

Fix #875: set StatsD host and port parameters for the reactionner dae…

3c28635

…mons

mohierf force-pushed the load-test-suite branch from 5c160eb to 3c28635 Compare July 26, 2017 17:40

Fix #877: fix implicit inheritance

59ab106

mohierf force-pushed the load-test-suite branch from 5496eb8 to 59ab106 Compare July 26, 2017 18:01

mohierf merged commit c05884e into develop Aug 1, 2017

mohierf deleted the load-test-suite branch August 1, 2017 07:12

Load test suite #819

Load test suite #819

Conversation

mohierf commented May 20, 2017 • edited

mohierf commented Jun 4, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mohierf commented Jun 9, 2017

mohierf commented Jun 9, 2017

coveralls commented Jul 20, 2017

coveralls commented Jul 20, 2017

coveralls commented Jul 20, 2017

coveralls commented Jul 21, 2017

coveralls commented Jul 24, 2017

coveralls commented Jul 24, 2017

coveralls commented Jul 26, 2017

coveralls commented Jul 26, 2017

coveralls commented Jul 26, 2017

coveralls commented Jul 26, 2017

coveralls commented Jul 26, 2017 • edited

mohierf commented Aug 1, 2017

mohierf commented May 20, 2017 •

edited

coveralls commented Jul 26, 2017 •

edited