Add single sensor mode to the sensor container service / process #4179

Kami · 2018-06-15T11:18:05Z

This pull request adds single sensor mode to the sensor container service (enabled via sensorcontainer.single_sensor_mode config option).

When this flag is set, sensor container process doesn't manage lifecycle of a sensor processes and respawn them if a sensor exits / crashes, but it exits itself and propagates sensor exit code to the parent sensor container process.

This comes handy in scenarios when a sensor life cycle and failover is handled by an external service such as Kubernetes / Mesos / monitoring + bash scripts / ...

Note: This flag needs to be used in combination with --sensor-ref CLI option since without it it doesn't make sense.

TODO

Tests
Documentation

…ion) to the sensor container. When running in this mode, it's assumed that sensor process life cycle and failover is handled by an external service so the sensor process container exits immediately when a server crashes. This is useful in scenarios where sensor process life cycle and failover is handled by external service such as kubernetes.

when running in single sensor mode.

arm4b · 2018-06-15T11:31:37Z

Checked, that looks like what we need from the description 👍

Also sounds like sensor_partitioner is irrelevant in this mode?
Additionally, how sensorcontainer.sensor_node_name works (clashes, ignored?) in this mode when specified in st2.conf?

st2/conf/st2.conf.sample

Lines 262 to 263 in 28129d4

    
           # name of the sensor node. 
        
           sensor_node_name = sensornode1

Since in both cases sensor ref is already specified explicitly.

I will test/play more closely with the package produced from this build later next week.

Kami · 2018-06-15T12:21:33Z

st2reactor/st2reactor/sensor/config.py

-        help='Only run sensor with the provided reference. Value is of the form pack.sensor-name.')
+    other_opts = [
+        cfg.BoolOpt(
+            'single_sensor_mode', default=False,


I'm also open to a better name for this option.

Kami · 2018-06-15T12:24:22Z

Also sounds like sensor_partitioner is irrelevant in this mode?
Additionally, how sensorcontainer.sensor_node_name works (clashes, ignored?) in this mode when specified in st2.conf?

Correct, right now when --sensor-ref option is provided, partitioner settings are ignored and SingleSensorPartitioner is used.

I can modify the code a bit to at least log a message in this case since right now we just ignore partitioner setting in such scenarios without logging anything.

partitioner is used.

--single-user-mode flag to the process.

arm4b · 2018-06-15T13:16:36Z

st2reactor/st2reactor/container/partitioner_lookup.py

@@ -41,16 +41,19 @@

 def get_sensors_partitioner():
    if cfg.CONF.sensor_ref:
+        LOG.info('Running in single sensor mode, using a single sensor partitioner...')


…kStorm/st2 into sensor_container_single_sensor_mode

integration tests and correctly set up the environment on Travis.

…ner_single_sensor_mode

Kami · 2018-06-18T13:08:02Z

@armab Tests are now in place and this PR is ready to be reviewed although I'm still battling with getting integration tests to work on Travis under Python 3.

…kStorm/st2 into sensor_container_single_sensor_mode

arm4b

Thanks for adding this!
I tested the feature in my local K8s env and it worked well 👍

arm4b · 2018-06-19T18:10:40Z

$ kubectl logs githubwebhooksensor-c9cb6b84b-xdk97 -f
Adding password for user admin
/st2-docker/bin/entrypoint.sh: ignoring /st2-docker/entrypoint.d/*

2018-06-19 16:02:36,142 DEBUG [-] Using config files: /etc/st2/st2.conf
2018-06-19 16:02:36,143 DEBUG [-] Using logging config: /etc/st2/logging.sensorcontainer.conf
2018-06-19 16:02:36,144 INFO [-] Connecting to database "st2" @ "mongo:27017" as user "None".
2018-06-19 16:02:36,316 INFO [-] Successfully connected to database "st2" @ "mongo:27017" as user "None".
2018-06-19 16:02:36,605 INFO [-] Running in single sensor mode, using a single sensor partitioner...
2018-06-19 16:02:36,607 INFO [-] Setting up container to run 1 sensors.
2018-06-19 16:02:36,607 INFO [-] 	Sensors list - [u'githubwebhook.GitHubWebhookSensor'].
2018-06-19 16:02:36,608 INFO [-] (PID:20) SensorContainer started.
2018-06-19 16:02:36,608 INFO [-] Running sensor githubwebhook.GitHubWebhookSensor
2018-06-19 16:02:36,620 AUDIT [-] Access granted to "sensors_container" with the token set to expire at "2018-06-20T16:02:36.617606Z". (username='sensors_container',token_expiration='2018-06-20T16:02:36.617606Z')
2018-06-19 16:02:36,672 INFO [-] Connected to amqp://admin:**@rabbitmq:5672//
2018-06-19 16:02:36,734 INFO [-] Sensor githubwebhook.GitHubWebhookSensor started
2018-06-19 16:02:37,568 INFO [-] Found config for sensor "GitHubWebhookSensor"
2018-06-19 16:02:37,571 INFO [-] Watcher started
2018-06-19 16:02:37,571 INFO [-] Running sensor initialization code
2018-06-19 16:02:37,573 INFO [-] Running sensor in passive mode
 * Serving Flask app "webhook_sensor" (lazy loading)
 * Environment: production
   WARNING: Do not use the development server in a production environment.
   Use a production WSGI server instead.
 * Debug mode: on
2018-06-19 16:02:37,584 INFO [-]  * Running on http://0.0.0.0:8642/ (Press CTRL+C to quit)
2018-06-19 16:02:37,585 INFO [-]  * Restarting with stat
2018-06-19 16:02:37,655 INFO [-] Connected to amqp://admin:**@rabbitmq:5672//
2018-06-19 16:02:38,489 INFO [-] Found config for sensor "GitHubWebhookSensor"
2018-06-19 16:02:38,490 INFO [-] Watcher started
2018-06-19 16:02:38,490 INFO [-] Running sensor initialization code
2018-06-19 16:02:38,492 INFO [-] Running sensor in passive mode
2018-06-19 16:02:38,497 WARNING [-]  * Debugger is active!
2018-06-19 16:02:38,616 INFO [-]  * Debugger PIN: 190-833-188
2018-06-19 16:02:38,691 INFO [-] Connected to amqp://admin:**@rabbitmq:5672//
2018-06-19 18:07:39,841 INFO [-] Stopping trigger watcher
2018-06-19 18:07:39,848 INFO [-] Invoking cleanup on sensor
2018-06-19 18:07:40,133 INFO [-] Stopping trigger watcher
2018-06-19 18:07:40,135 INFO [-] Invoking cleanup on sensor
2018-06-19 18:07:41,081 INFO [-] Process for sensor githubwebhook.GitHubWebhookSensor has exited with code 0
2018-06-19 18:07:41,086 INFO [-] Not respawning a sensor since running in single sensor mode (sensor_id=u'githubwebhook.GitHubWebhookSensor',sensor={'class_name': u'GitHubWebhookSensor', 'poll_interval': None, 'pack': u'githubwebhook', 'ref': u'githubwebhook.GitHubWebhookSensor', 'file_path': u'/opt/stackstorm/packs/githubwebhook/sensors/webhook_sensor.py', 'trigger_types': [u'githubwebhook.github_event']})
2018-06-19 18:07:42,086 ERROR [-] Process container stopped.
2018-06-19 18:07:42,086 ERROR [-] Process container quit with exit_code 0.
2018-06-19 18:07:42,086 ERROR [-] (PID:20) SensorContainer stopped.

👍

arm4b · 2018-06-19T18:12:53Z

Just a future note for myself, when I'll need to productionize that in Docker & K8s:

exec /opt/stackstorm/st2/bin/st2sensorcontainer \
  --config-file /etc/st2/st2.conf \
  --single-sensor-mode \
  --sensor-ref=githubwebhook.GitHubWebhookSensor

lakshmi-kannan · 2018-06-19T20:37:25Z

I don't think we should merge this for 2.8.0 but instead bundle all HA changes for 2.9.0.

Kami · 2018-06-20T09:14:39Z

@lakshmi-kannan It doesn't negatively affect things (or change the behavior) in any way, so I see no reason to not merge it.

At the very least, if we decide to postpone the merge we should pick out and merge Python 3 fixes and changes.

But again, as mentioned above, I see no reason for waiting with merging this (if we want to postpone including this from product / announcing perspective we can simply skip the changelog entry / add it behind the feature flag, but I think that's really an over kill for such a simple change).

Kami · 2018-06-20T09:18:01Z

@armab Yeah, you can also set single sensor mode via config option, but I personally prefer the command line flag since it's more explicit and visible compared to the config file option.

under Python 3. Input body is actually bytes under Python 3, but we except and work with text type (str / unicode) so we need to convert from bytes to unicode.

…kStorm/st2 into sensor_container_single_sensor_mode

lakshmi-kannan · 2018-06-20T13:15:11Z

@Kami If you feel strongly and you are sure it won't break anything with sensor behavior today, I am +1 to shipping this. I looked at the code and I don't expect it to break existing behavior. Please validate this and you have my +1.

However, splitting timers into own process though isn't functionally different, I will wait until after 2.8 to ship it.

Kami · 2018-06-20T13:42:07Z

@lakshmi-kannan Yeah, I agree about the timer change - it's more involved since it includes a new service and packaging + installation changes.

Kami added 2 commits June 15, 2018 13:08

Correctly propagate sensor exit code all the way to the parent process

8384eec

when running in single sensor mode.

Kami added this to the 2.8.0 milestone Jun 15, 2018

Kami requested a review from arm4b June 15, 2018 11:18

Kami commented Jun 15, 2018

View reviewed changes

Kami added 3 commits June 15, 2018 14:27

Log a message when running in a single sensor mode and a single sensor

231f7ab

partitioner is used.

Update tests config.

4ec2334

Also allow user to enable sensor container single user mode by passing

dfc57f8

--single-user-mode flag to the process.

arm4b reviewed Jun 15, 2018

View reviewed changes

Kami added 13 commits June 18, 2018 10:30

Merge branch 'master' into sensor_container_single_sensor_mode

ef0d8e0

Merge branch 'sensor_container_single_sensor_mode' of github.com:Stac…

157480f

…kStorm/st2 into sensor_container_single_sensor_mode

Add tests for sensor container single sensor mode.

cf5e7d2

Fix lint.

f732cfc

Re-generate sample configs.

c7bc2f5

Use longer sleep for more predictable tests.

2f6b8c1

Fix tests so they also work under Python 3.

1deb6ba

Run GC tests under Python 3.

d74c9d4

Split tox Python 3 task into two tasks - one for unit and one for

a16938a

integration tests and correctly set up the environment on Travis.

Update make targets, increase sleep.

93b5e11

Merge branch 'master' of github.com:StackStorm/st2 into sensor_contai…

eac6724

…ner_single_sensor_mode

Use the same python binary which is used for running the tests.

c92ac57

Use the same binary which is used for the tests process.

74e3543

Kami changed the title ~~[WIP] Add single sensor mode to the sensor container service / process~~ Add single sensor mode to the sensor container service / process Jun 18, 2018

Kami added python3 HA StackStorm in High Availability labels Jun 18, 2018

Kami added 2 commits June 18, 2018 15:56

Update exporter code so it works under Python 3.

27d40d0

Also run exporter unit and integration tests under Python 3.

5d138db

Kami added 12 commits June 18, 2018 16:00

Also run exporter unit and integration tests under Python 3.

ece9d6e

Merge branch 'sensor_container_single_sensor_mode' of github.com:Stac…

84b4068

…kStorm/st2 into sensor_container_single_sensor_mode

Fix st2debug tests so they work under Python 3.

6b3f139

Run st2debug tests under tox.

e6e0305

Also run st2common and st2api integration tests under Python 3 in tox.

f2d48e3

Fix st2actions tests so they pass under Python 3.

56a08fa

Run st2actions integration tests under Python 3 in tox.

f7abf5d

Make sure st2auth is in PYTHONPATH for tests.

c5138b1

Use the same binary which is used to run the tests.

6ea097a

Include more info to aid with debugging.

7098561

Use executable which is used to run the tests.

6fdda8c

Update tests config.

f4afc84

arm4b approved these changes Jun 19, 2018

View reviewed changes

Merge branch 'master' into sensor_container_single_sensor_mode

70d3657

Kami added 3 commits June 20, 2018 12:45

Add a fix for router so it correctly handles data of type text/plain

4d3530f

under Python 3. Input body is actually bytes under Python 3, but we except and work with text type (str / unicode) so we need to convert from bytes to unicode.

Mistral validation tests now pass under Python 3.

3ee5351

Merge branch 'sensor_container_single_sensor_mode' of github.com:Stac…

342a5ac

…kStorm/st2 into sensor_container_single_sensor_mode

Kami merged commit da6b04f into master Jun 20, 2018

Kami deleted the sensor_container_single_sensor_mode branch June 20, 2018 15:04

arm4b mentioned this pull request Sep 18, 2018

Partitioning StackStorm Sensors StackStorm/stackstorm-k8s#4

Closed

warrenvw mentioned this pull request Mar 1, 2019

Allow st2sensorcontainer to be partitioned StackStorm/stackstorm-k8s#51

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add single sensor mode to the sensor container service / process #4179

Add single sensor mode to the sensor container service / process #4179

Kami commented Jun 15, 2018 •

edited

Loading

arm4b commented Jun 15, 2018 •

edited

Loading

Kami Jun 15, 2018

Kami commented Jun 15, 2018

arm4b Jun 15, 2018

Kami commented Jun 18, 2018

arm4b left a comment •

edited

Loading

arm4b commented Jun 19, 2018

arm4b commented Jun 19, 2018

lakshmi-kannan commented Jun 19, 2018

Kami commented Jun 20, 2018 •

edited

Loading

Kami commented Jun 20, 2018

lakshmi-kannan commented Jun 20, 2018

Kami commented Jun 20, 2018

Add single sensor mode to the sensor container service / process #4179

Add single sensor mode to the sensor container service / process #4179

Conversation

Kami commented Jun 15, 2018 • edited Loading

TODO

arm4b commented Jun 15, 2018 • edited Loading

Kami Jun 15, 2018

Choose a reason for hiding this comment

Kami commented Jun 15, 2018

arm4b Jun 15, 2018

Choose a reason for hiding this comment

Kami commented Jun 18, 2018

arm4b left a comment • edited Loading

Choose a reason for hiding this comment

arm4b commented Jun 19, 2018

arm4b commented Jun 19, 2018

lakshmi-kannan commented Jun 19, 2018

Kami commented Jun 20, 2018 • edited Loading

Kami commented Jun 20, 2018

lakshmi-kannan commented Jun 20, 2018

Kami commented Jun 20, 2018

Kami commented Jun 15, 2018 •

edited

Loading

arm4b commented Jun 15, 2018 •

edited

Loading

arm4b left a comment •

edited

Loading

Kami commented Jun 20, 2018 •

edited

Loading