Issue #306 : Adding the ability to use overrides #307

cwilson21 · 2022-05-13T22:25:41Z

After working to get this implemented and tested. It appears that the changes I have added are working.

This creates the /opt/stackstorm/overrides/ mountpoint in the st2client pod so that if reloads happen in the pod it will keep the settings provided by users in the overrides. This also registers overrides when the st2 cluster is being built via the register-content job.

THis is just extending the now existing override functionality that exists in Stackstorm 3.7.0 into the HA version of Stackstorm that is now operating on version 3.7.0

2022-05-13 22:08:52,283 INFO [-] Connecting to database "st2" @ "mongodb-local-headless:27017" as user "st2-admin".
2022-05-13 22:08:52,294 INFO [-] Successfully connected to database "st2" @ "mongodb-local-headless:27017" as user "st2-admin".
2022-05-13 22:08:52,734 INFO [-] =========================================================
2022-05-13 22:08:52,734 INFO [-] ############## Registering triggers #####################
2022-05-13 22:08:52,734 INFO [-] =========================================================
2022-05-13 22:08:53,152 INFO [-] Registered 1 triggers.
2022-05-13 22:08:53,152 INFO [-] =========================================================
2022-05-13 22:08:53,152 INFO [-] ############## Registering sensors ######################
2022-05-13 22:08:53,153 INFO [-] =========================================================
2022-05-13 22:08:54,528 INFO [-] Registered 40 sensors.
2022-05-13 22:08:54,529 INFO [-] 0 sensors had their metadata overridden.
2022-05-13 22:08:54,529 INFO [-] =========================================================
2022-05-13 22:08:54,529 INFO [-] ############## Registering runners ######################
2022-05-13 22:08:54,529 INFO [-] =========================================================
2022-05-13 22:08:55,912 INFO [-] Registered 14 runners.
2022-05-13 22:08:55,912 INFO [-] =========================================================
2022-05-13 22:08:55,912 INFO [-] ############## Registering actions ######################
2022-05-13 22:08:55,913 INFO [-] =========================================================
2022-05-13 22:10:24,631 INFO [-] Registered 4473 actions.
2022-05-13 22:10:24,631 INFO [-] 0 actions had their metadata overridden.
2022-05-13 22:10:24,631 INFO [-] =========================================================
2022-05-13 22:10:24,631 INFO [-] ############## Registering rules ########################
2022-05-13 22:10:24,631 INFO [-] =========================================================
2022-05-13 22:10:24,805 INFO [-] Registered 5 rules.
2022-05-13 22:10:24,805 INFO [-] 5 rules had their metadata overridden.
2022-05-13 22:10:24,805 INFO [-] =========================================================
2022-05-13 22:10:24,805 INFO [-] ############## Registering aliases ######################
2022-05-13 22:10:24,805 INFO [-] =========================================================
2022-05-13 22:10:25,081 INFO [-] Registered 15 aliases.
2022-05-13 22:10:25,081 INFO [-] 0 aliases had their metadata overridden.
2022-05-13 22:10:25,082 INFO [-] =========================================================
2022-05-13 22:10:25,082 INFO [-] ############## Registering policy types #################
2022-05-13 22:10:25,082 INFO [-] =========================================================
2022-05-13 22:10:25,112 INFO [-] Registered 3 policy types.
2022-05-13 22:10:25,112 INFO [-] =========================================================
2022-05-13 22:10:25,112 INFO [-] ############## Registering policies #####################
2022-05-13 22:10:25,112 INFO [-] =========================================================
2022-05-13 22:10:25,117 INFO [-] Registered 0 policies.
2022-05-13 22:10:25,117 INFO [-] =========================================================
2022-05-13 22:10:25,117 INFO [-] ############## Registering configs ######################
2022-05-13 22:10:25,118 INFO [-] =========================================================
2022-05-13 22:10:25,158 INFO [-] Registered 4 configs.```

Adding override config

CLAassistant · 2022-05-13T22:25:47Z

All committers have signed the CLA.

Updating unit tests to include the new configmap that is used by jobs and deployments

…commented out

… the override node in

arm4b

Thanks for the contribution! A really clean implementation 👍

Left several comments to address and discussion around the expected interface.

values.yaml

templates/configmaps_overrides.yaml

values.yaml

…exists it enables

Adding commented example, override to overrides and new lines

…des" array

Removing theh "configs" layer in helm chart so it is just the "overri…

cognifloyd

You've only mounted the overrides under the st2client deployment. So we need this inner any other deployments like st2api? Or st2actionrunner?

And I agree that the register-content job is the only default job that needs the overrides. But, could we also add the overrides mount to the extra_hooks jobs?

Plus a few other comments.
Thanks again for working on this

templates/deployments.yaml

templates/jobs.yaml

values.yaml

templates/_helpers.tpl

templates/deployments.yaml

cwilson21 · 2022-05-15T03:47:14Z

I don't think that this needs to be in any other deployment likes st2api or st2actionrunner. I say that based on I am only adding this to client so that if a st2ctl reload is ran it does not nuke the overrides that were put in when register-content happened. As far as I see from my testing being in just st2-client is fine, I have a job in my local I was working on previously for our implementation for backing up mongo to gcs and that job has not ran since I enacted the global override on rules to set them to enabled: false . I appreciate being able to work on this, I am fairly new to working with k8s so this was a fun adventure and I will read through your suggestions and correct what was missed and see about implementing the changes to use my helper template.

Co-authored-by: Jacob Floyd <cognifloyd@gmail.com>

Updating things from override to overrides

cwilson21 · 2022-05-17T17:16:14Z

Ah I think I missed a few places I will address them shortly

tests/unit/overrides_test.yaml

Co-authored-by: Jacob Floyd <cognifloyd@gmail.com>

Adjusting test

cognifloyd

Excellent. Thanks for dealing with all my nitpicks.

tests/unit/overrides_test.yaml

cognifloyd · 2022-05-17T19:25:33Z

Oh! We also need a changelog entry for this. Please and thank you!

cwilson21 · 2022-05-17T20:46:53Z

@cognifloyd what version would we call this? 0.100.0 or is this going to be apart of 0.90.0?

cwilson21 · 2022-05-17T20:50:27Z

nvm. Just throwing it in the In Development section

Adjusting changelog

cognifloyd · 2022-05-17T21:13:00Z

@armab wdyt? I think this is good to merge.

Changing mountpath to remove trailing slash

cwilson21 · 2022-05-17T21:42:18Z

Missed one change request for the trailing slash on the mountpoint and got that updated :)

arm4b · 2022-05-17T21:54:15Z

values.yaml

+  #    rules:
+  #      defaults:
+  #        enabled: true
+  #  packA.yaml: |


Looks like a wrong indentation here for packA.yaml, compared to _global.yaml.

The indentation should be fixed now.

values.yaml

arm4b

Left a couple small comments,

besides that LGTM 👍

…with proper spacing

Updating values.yaml to have the exmaple more consistent iwth others …

arm4b · 2022-05-18T08:52:17Z

Nice!
That's a high-quality contribution Thanks, everyone 👍

cwilson21 added 5 commits May 12, 2022 17:28

Adding override config

9447f94

Changing path

0a361d2

Updating with working version of overrides

e936cad

Adding to values

1d1d68c

Merge pull request #1 from cwilson21/testing

250bbec

Adding override config

pull-request-size bot added the size/M PR that changes 30-99 lines. Good size to review. label May 13, 2022

cwilson21 mentioned this pull request May 13, 2022

Overrides for Rules/Sensors #306

Closed

cwilson21 added 9 commits May 13, 2022 17:37

Removing duplicate entry

ae5a170

Testing what needs to be done with unit tests

8a63c4d

Adding the configmap for overrides to all unit tests that will need it

8812a54

Removing some of the bits from my original testing

5282abb

Merge pull request #3 from cwilson21/testing

f402cbc

Updating unit tests to include the new configmap that is used by jobs and deployments

Adding link to documentation to values file and giving crude example …

553bdc6

…commented out

Trying to fix values so it is valid

8f68ab1

Fixing values to just link to document for overrides and putting just…

6633a4c

… the override node in

Setting to disabled

6d9a6e2

arm4b suggested changes May 14, 2022

View reviewed changes

values.yaml Outdated Show resolved Hide resolved

values.yaml Outdated Show resolved Hide resolved

templates/configmaps_overrides.yaml Outdated Show resolved Hide resolved

values.yaml Outdated Show resolved Hide resolved

values.yaml Outdated Show resolved Hide resolved

arm4b added the feature label May 14, 2022

cwilson21 added 6 commits May 14, 2022 12:33

Adding commented example, override to overrides and new lines

0bc0a66

Override to overrides in a few more places.

95ccd40

Changing so there is no enabled switch, if .Values.overrides.configs …

35f7955

…exists it enables

Merge pull request #4 from cwilson21/testing

8c8217a

Adding commented example, override to overrides and new lines

Removing theh "configs" layer in helm chart so it is just the "overri…

3e87f0e

…des" array

Merge pull request #5 from cwilson21/testing

da5d11d

Removing theh "configs" layer in helm chart so it is just the "overri…

cognifloyd reviewed May 15, 2022

View reviewed changes

cwilson21 and others added 3 commits May 14, 2022 22:47

Update templates/_helpers.tpl

7824586

Co-authored-by: Jacob Floyd <cognifloyd@gmail.com>

Update values.yaml

4d34bfe

Co-authored-by: Jacob Floyd <cognifloyd@gmail.com>

Update templates/_helpers.tpl

01f6f7f

Co-authored-by: Jacob Floyd <cognifloyd@gmail.com>

Merge pull request #7 from cwilson21/testing

32e4848

Updating things from override to overrides

cognifloyd reviewed May 17, 2022

View reviewed changes

tests/unit/overrides_test.yaml Outdated Show resolved Hide resolved

cwilson21 and others added 5 commits May 17, 2022 12:43

Update tests/unit/overrides_test.yaml

2dc8d40

Co-authored-by: Jacob Floyd <cognifloyd@gmail.com>

Adjusting test

b46ba92

Adjusting helper

c1b1561

Adding trailing slash to mountpoint

4e19d80

Merge pull request #9 from cwilson21/adjusting

412bb6c

Adjusting test

cognifloyd approved these changes May 17, 2022

View reviewed changes

tests/unit/overrides_test.yaml Outdated Show resolved Hide resolved

tests/unit/overrides_test.yaml Outdated Show resolved Hide resolved

tests/unit/overrides_test.yaml Outdated Show resolved Hide resolved

Apply suggestions from code review

89e505a

cognifloyd enabled auto-merge May 17, 2022 19:20

cwilson21 added 2 commits May 17, 2022 15:53

Adjusting changelog

bf981fb

Merge pull request #10 from cwilson21/adjusting

4235999

Adjusting changelog

auto-merge was automatically disabled May 17, 2022 21:04
Head branch was pushed to by a user without write access

cognifloyd requested a review from arm4b May 17, 2022 21:12

cwilson21 added 2 commits May 17, 2022 16:28

Changing mountpath to remove trailing slash

c5cfc2c

Merge pull request #11 from cwilson21/adjusting

bd0c39a

Changing mountpath to remove trailing slash

arm4b reviewed May 17, 2022

View reviewed changes

values.yaml Outdated Show resolved Hide resolved

arm4b approved these changes May 17, 2022

View reviewed changes

cwilson21 added 3 commits May 17, 2022 17:03

Updating values.yaml to have the exmaple more consistent iwth others …

785d48f

…with proper spacing

Fixing spacing

903b155

Merge pull request #12 from cwilson21/adjusting

fb10e50

Updating values.yaml to have the exmaple more consistent iwth others …

cognifloyd merged commit dceceed into StackStorm:master May 17, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue #306 : Adding the ability to use overrides #307

Issue #306 : Adding the ability to use overrides #307

cwilson21 commented May 13, 2022 •

edited

Loading

CLAassistant commented May 13, 2022 •

edited

Loading

arm4b left a comment

cognifloyd left a comment

cwilson21 commented May 15, 2022

cwilson21 commented May 17, 2022

cognifloyd left a comment

cognifloyd commented May 17, 2022

cwilson21 commented May 17, 2022

cwilson21 commented May 17, 2022

cognifloyd commented May 17, 2022

cwilson21 commented May 17, 2022

arm4b May 17, 2022

cwilson21 May 17, 2022

arm4b left a comment

arm4b commented May 18, 2022

Issue #306 : Adding the ability to use overrides #307

Issue #306 : Adding the ability to use overrides #307

Conversation

cwilson21 commented May 13, 2022 • edited Loading

CLAassistant commented May 13, 2022 • edited Loading

arm4b left a comment

Choose a reason for hiding this comment

cognifloyd left a comment

Choose a reason for hiding this comment

cwilson21 commented May 15, 2022

cwilson21 commented May 17, 2022

cognifloyd left a comment

Choose a reason for hiding this comment

cognifloyd commented May 17, 2022

cwilson21 commented May 17, 2022

cwilson21 commented May 17, 2022

cognifloyd commented May 17, 2022

cwilson21 commented May 17, 2022

arm4b May 17, 2022

Choose a reason for hiding this comment

cwilson21 May 17, 2022

Choose a reason for hiding this comment

arm4b left a comment

Choose a reason for hiding this comment

arm4b commented May 18, 2022

cwilson21 commented May 13, 2022 •

edited

Loading

CLAassistant commented May 13, 2022 •

edited

Loading