Skipping internal setup of ES and PG in case of External config #6606

shanmugapriya-tr · 2022-01-21T05:40:06Z

🔩 Description: What code changed, and why?

This PR contains changes for skipping the setup of internal Elasticsearch and Postgres in case External ES or PG configuration is enabled. The following changes were made to achieve this :

Made automate-elasticsearch an optional dependency in es-sidecar-service
While building the desired state for deployment, the deployment state of automate-elasticsearch and automate-postgresql services are set to Removed
While performing Health check, automate-elasticsearch and automate-postgresql services are skipped in case external configuration is set

⛓️ Related Resources

👍 Definition of Done

Tested by disabling external PG and ES. Internal Services are getting set up properly
Tested by enabling external PG and ES. Internal services are unloaded and are not running.
Tested various commands like patch (moving from external to internal and vice versa), backup and restore.
Tested Org and User creation with external PG
Ran compliance scan and client run with External services enabled (tested via via the load-generation lib)

Known issue

[RESOLVED] Certain services not restarting with changes in the ES config

Resolution : Adding ESGateway and PGGateway to the configWriteRequiresReload function, to resolve the issue with the service restarts while switching from internal to external and vice-versa.
Although, the services were restarting for PG, it has been added here as a false-safe so that in the future if the services that have stored the PG information in the _a2_platform.json file and have a direct dependency on PG will restart even if we remove the dependency in the future.
Adding the GW's to the list in the configWriteRequiresReload will ensure that the mitigation is executed for the GW as well, therefore restarting all the dependent services.

👟 How to Build and Test the Change

✅ Checklist

All PRs must tick these:

I have read the CONTRIBUTING document.
All commits signed-off for the Developer Certification of Origin.

With occasional exceptions, all PRs from Progress employees must tick these:

Is the code clear? (complicated code or lots of comments--subdivide and use well-named methods, meaningful variable names, etc.)
Consistency checked? (user notifications, user prompts, visual patterns, code patterns, variable names)
Repeated code blocks eliminated? (adapt and reuse existing components, blocks, functions, etc.)
Spelling, grammar, typos checked? (at a minimum use make spell in any component directory)
Code well-formatted? (indents, line breaks, etc. improve rather than hinder readability)

All PRs from Progress employees should tick these if appropriate:

Tests added/updated? (all new code needs new tests)
Docs added/updated? (all customer-facing changes)

📷 Screenshots, if applicable

Signed-off-by: shanmugapriya-tr <shanmugapriya.tiruchengoderamanathan@progress.com>

1. Added the code to skip the automate-elasticsearch service if External is enabled 2. Decoupled the es-sidecar service from automate-elasticsearch 3. Moved the code to skip the status check for removed services in ensureStatus Signed-off-by: sandhi <sagarwal@progress.com>

Signed-off-by: shanmugapriya-tr <shanmugapriya.tiruchengoderamanathan@progress.com>

into saas/managed_services

netlify · 2022-01-21T05:40:12Z

✔️ Deploy Preview for chef-automate ready!

🔨 Explore the source changes: fc11bc0

🔍 Inspect the deploy log: https://app.netlify.com/sites/chef-automate/deploys/62317a693a8507000890617d

😎 Browse the preview: https://deploy-preview-6606--chef-automate.netlify.app

Signed-off-by: shanmugapriya-tr <shanmugapriya.tiruchengoderamanathan@progress.com>

1. Added the code to skip the automate-elasticsearch service if External is enabled 2. Decoupled the es-sidecar service from automate-elasticsearch 3. Moved the code to skip the status check for removed services in ensureStatus Signed-off-by: sandhi <sagarwal@progress.com>

Signed-off-by: shanmugapriya-tr <shanmugapriya.tiruchengoderamanathan@progress.com>

into saas/managed_services

components/automate-deployment/pkg/server/server.go

punitmundra

LGTM. just use const instead of stings.

Signed-off-by: shanmugapriya-tr <shanmugapriya.tiruchengoderamanathan@progress.com>

components/automate-deployment/pkg/server/server.go

Signed-off-by: shanmugapriya-tr <shanmugapriya.tiruchengoderamanathan@progress.com>

into saas/managed_services

shanmugapriya-tr · 2022-03-10T08:14:53Z

@kalroy @karunesh-tech - I have rebased the branch with main. The build failures are not related to the changes done as a part of this PR. Please approve the PR if it is good to be merged

* Fixing License Scout Signed-off-by: Durga Sarat Chandra Maddu <durgasaratchandra.maddu@progress.com> * Fixing License Scout for msgp Signed-off-by: Durga Sarat Chandra Maddu <durgasaratchandra.maddu@progress.com>

* added Todo(milestone) for the servers Signed-off-by: Kallol Roy <karoy@progress.com> * added more Todo(milestone) for the servers Signed-off-by: Kallol Roy <karoy@progress.com> * adding to-dos Signed-off-by: Durga Sarat Chandra Maddu <durgasaratchandra.maddu@progress.com> * upgrading to minor/patch version (#6621) * upgrade to minor\patch Signed-off-by: Durga Sarat Chandra Maddu <durgasaratchandra.maddu@progress.com> * supporting sem version Signed-off-by: Durga Sarat Chandra Maddu <durgasaratchandra.maddu@progress.com> * considering version flag in upgrade Signed-off-by: Durga Sarat Chandra Maddu <durgasaratchandra.maddu@progress.com> * compute min current version for airgap bundle create- second pr (#6665) * adding min current version to manifest in bundle creation Signed-off-by: Rick Marry <rick.marry@gmail.com> * adding min current version to manifest in bundle creation Signed-off-by: Durga Sarat Chandra Maddu <durgasaratchandra.maddu@progress.com> Co-authored-by: dmaddu <durgasaratchandra.maddu@progress.com> * major upgrade with checklist (#6660) Signed-off-by: Durga Sarat Chandra Maddu <durgasaratchandra.maddu@progress.com> * fixing buildkite (#6674) Signed-off-by: Durga Sarat Chandra Maddu <durgasaratchandra.maddu@progress.com> * airgap bundle upgrade (#6682) * airgap upgrade Signed-off-by: Durga Sarat Chandra Maddu <durgasaratchandra.maddu@progress.com> * upgrade status for airgap Signed-off-by: Durga Sarat Chandra Maddu <durgasaratchandra.maddu@progress.com> * upgrade status for airgap instance Signed-off-by: Durga Sarat Chandra Maddu <durgasaratchandra.maddu@progress.com> * addressing review comments Signed-off-by: Durga Sarat Chandra Maddu <durgasaratchandra.maddu@progress.com> * Show Major Version Release Banner on UI (#6663) * banner-manual-upgrade changes Signed-off-by: Venkatesh-rengasamy <vrengasa@progress.com> * add banner logo Signed-off-by: Venkatesh-rengasamy <vrengasa@progress.com> * updating error messages to be shown to end user (#6714) Signed-off-by: Durga Sarat Chandra Maddu <durgasaratchandra.maddu@progress.com> * ignoring semgrep rule Signed-off-by: Durga Sarat Chandra Maddu <durgasaratchandra.maddu@progress.com> * fixing pipeline issue (#6717) Signed-off-by: Durga Sarat Chandra Maddu <durgasaratchandra.maddu@progress.com> * supporting external versions file (#6748) * supporting external versions file Signed-off-by: Durga Sarat Chandra Maddu <durgasaratchandra.maddu@progress.com> * fixing pipeline Signed-off-by: Durga Sarat Chandra Maddu <durgasaratchandra.maddu@progress.com> * showing pre post Checklist for milestone release (#6706) * postchecklist Signed-off-by: shaik80 <smudassi@progress.com> * json file Signed-off-by: shaik80 <smudassi@progress.com> * readPostCheckList Signed-off-by: shaik80 <smudassi@progress.com> * read json file for postchecklist Signed-off-by: shaik80 <smudassi@progress.com> * upgrade to major and and print postchecklist steps Signed-off-by: shaik80 <smudassi@progress.com> * test cases and basic changes Signed-off-by: shaik80 <smudassi@progress.com> * dynamic version Signed-off-by: shaik80 <smudassi@progress.com> * old manifests url Signed-off-by: shaik80 <smudassi@progress.com> * error handling in upgrade status if we don't hav compatible version Signed-off-by: shaik80 <smudassi@progress.com> * update json file func changes Signed-off-by: shaik80 <smudassi@progress.com> * add version in json file Signed-off-by: shaik80 <smudassi@progress.com> * Pre and post checklist for embedded and external PG (#6705) * changes for pre post checklist Signed-off-by: Vivek Shankar <vshankar@progress.com> * minor changes Signed-off-by: Vivek Shankar <vshankar@progress.com> * changed formatting Signed-off-by: Vivek Shankar <vshankar@progress.com> * id changes Signed-off-by: Vivek Shankar <vshankar@progress.com> * Pg external Signed-off-by: shaik80 <smudassi@progress.com> * minor fmt changes Signed-off-by: Vivek Shankar <vshankar@progress.com> * spell check Signed-off-by: shaik80 <smudassi@progress.com> * variable change Signed-off-by: shaik80 <smudassi@progress.com> * remove log Signed-off-by: shaik80 <smudassi@progress.com> * postchecklist new structure Signed-off-by: shaik80 <smudassi@progress.com> * added new interface function for post checklist Signed-off-by: Vivek Yadav <vivek.yadav@progress.com> * get post checklist implementation added Signed-off-by: Vivek Yadav <vivek.yadav@progress.com> * feed back changes Signed-off-by: shaik80 <smudassi@progress.com> * code modification Signed-off-by: shaik80 <smudassi@progress.com> * if file not found return nil Signed-off-by: shaik80 <smudassi@progress.com> * if file not found don't throw an error Signed-off-by: shaik80 <smudassi@progress.com> * remove log and lint fix Signed-off-by: shaik80 <smudassi@progress.com> * NewChecklistManager remove major version param Signed-off-by: shaik80 <smudassi@progress.com> * Fixing pipeline issues Signed-off-by: shaik80 <smudassi@progress.com> * title for post checklist in upgrade status Signed-off-by: shaik80 <smudassi@progress.com> * making global constant Signed-off-by: shaik80 <smudassi@progress.com> Co-authored-by: vivekshankar1 <59958706+vivekshankar1@users.noreply.github.com> Co-authored-by: Vivek Shankar <vshankar@progress.com> Co-authored-by: Vivek Yadav <vivek.yadav@progress.com> Co-authored-by: dmaddu <durgasaratchandra.maddu@progress.com> * partial changes to fix pipeline Signed-off-by: Durga Sarat Chandra Maddu <durgasaratchandra.maddu@progress.com> * hiding versions-file flag to end users Signed-off-by: Durga Sarat Chandra Maddu <durgasaratchandra.maddu@progress.com> * removing the commented code Signed-off-by: Durga Sarat Chandra Maddu <durgasaratchandra.maddu@progress.com> * hiding the banner Signed-off-by: Durga Sarat Chandra Maddu <durgasaratchandra.maddu@progress.com> * Fix automate UI pipeline issue (#6790) * pipline fix Signed-off-by: shaik80 <sh.mudassir98@gmail.com> * automate pipline fix Signed-off-by: shaik80 <sh.mudassir98@gmail.com> Co-authored-by: shaik80 <sh.mudassir98@gmail.com> Co-authored-by: Kallol Roy <karoy@progress.com> Co-authored-by: Rick Marry <rmarry@chef.io> Co-authored-by: Venkatesh-rengasamy <84014872+Venkatesh-rengasamy@users.noreply.github.com> Co-authored-by: Shaik Mudassir <47217471+shaik80@users.noreply.github.com> Co-authored-by: vivekshankar1 <59958706+vivekshankar1@users.noreply.github.com> Co-authored-by: Vivek Shankar <vshankar@progress.com> Co-authored-by: Vivek Yadav <vivek.yadav@progress.com> Co-authored-by: shivangisingh <87964433+shivangi-singh1@users.noreply.github.com> Co-authored-by: shaik80 <sh.mudassir98@gmail.com>

Obvious fix; these changes are the result of automation not creative thinking.

Signed-off-by: Durga Sarat Chandra Maddu <durgasaratchandra.maddu@progress.com>

* upgrade es version to 6.8.23 in automate-ha * side car package bump to avaoid the confusion Signed-off-by: punitmundra <punitmundra2008@gmail.com>

* set git config global Signed-off-by: Vivek Yadav <vivek.yadav@progress.com> * updated commit id of epgsql Signed-off-by: Vivek Yadav <vivek.yadav@progress.com> * log git config updated in license scout Signed-off-by: Vivek Yadav <vivek.yadav@progress.com> * changed url in git config override Signed-off-by: Vivek Yadav <vivek.yadav@progress.com> * fix git:// in build and unit test of notification Signed-off-by: Vivek Yadav <vivek.yadav@progress.com>

shanmugapriya-tr · 2022-03-16T05:39:39Z

@kalroy Please review and approve the PR.

Signed-off-by: shanmugapriya-tr <shanmugapriya.tiruchengoderamanathan@progress.com>

1. Added the code to skip the automate-elasticsearch service if External is enabled 2. Decoupled the es-sidecar service from automate-elasticsearch 3. Moved the code to skip the status check for removed services in ensureStatus Signed-off-by: sandhi <sagarwal@progress.com>

Signed-off-by: shanmugapriya-tr <shanmugapriya.tiruchengoderamanathan@progress.com>

Signed-off-by: sandhi <sagarwal@progress.com>

Co-authored-by: Kallol Roy <karoy@progress.com>

Signed-off-by: shanmugapriya-tr <shanmugapriya.tiruchengoderamanathan@progress.com>

into saas/managed_services

sonarcloud · 2022-03-16T05:54:58Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
4 Code Smells

18.5% Coverage
0.0% Duplication

shanmugapriya-tr · 2022-03-17T07:03:24Z

@kalroy - Can you please help merging the PR ?

shanmugapriya-tr and others added 6 commits January 18, 2022 18:33

Changes for External PG

681d142

Signed-off-by: shanmugapriya-tr <shanmugapriya.tiruchengoderamanathan@progress.com>

Refactored skipping services health check

c7f3a51

Signed-off-by: shanmugapriya-tr <shanmugapriya.tiruchengoderamanathan@progress.com>

Added few debug logs

aff4686

Added few debug logs

3d43e7a

Signed-off-by: shanmugapriya-tr <shanmugapriya.tiruchengoderamanathan@progress.com>

Merge branch 'saas/managed_services' of https://github.com/chef/automate

69470c5

into saas/managed_services

shanmugapriya-tr requested review from rickmarry, vivek-yadav, kalroy, vivekvardwaz, sandhi18, jayvikramsharma1, punitmundra, Dmaddu and karunesh-tech January 21, 2022 05:40

shanmugapriya-tr self-assigned this Jan 21, 2022

shanmugapriya-tr changed the title ~~Saas/managed services~~ Skipping internal setup of ES and PG in case of External config Jan 21, 2022

shanmugapriya-tr and others added 7 commits January 21, 2022 12:18

Making binds optional

d5dca46

Signed-off-by: shanmugapriya-tr <shanmugapriya.tiruchengoderamanathan@progress.com>

Changes for External PG

018521d

Signed-off-by: shanmugapriya-tr <shanmugapriya.tiruchengoderamanathan@progress.com>

Refactored skipping services health check

8d1f3bf

Signed-off-by: shanmugapriya-tr <shanmugapriya.tiruchengoderamanathan@progress.com>

Added few debug logs

d59c216

Making binds optional

b0c48b9

Signed-off-by: shanmugapriya-tr <shanmugapriya.tiruchengoderamanathan@progress.com>

Merge branch 'saas/managed_services' of https://github.com/chef/automate

7347a86

into saas/managed_services

punitmundra reviewed Jan 31, 2022

View reviewed changes

components/automate-deployment/pkg/server/server.go Outdated Show resolved Hide resolved

punitmundra approved these changes Jan 31, 2022

View reviewed changes

Changed Service names to constants

6489957

Signed-off-by: shanmugapriya-tr <shanmugapriya.tiruchengoderamanathan@progress.com>

vivek-yadav reviewed Feb 2, 2022

View reviewed changes

components/automate-deployment/pkg/server/server.go Outdated Show resolved Hide resolved

vivek-yadav reviewed Feb 2, 2022

View reviewed changes

components/automate-deployment/pkg/server/server.go Outdated Show resolved Hide resolved

shanmugapriya-tr added 2 commits March 10, 2022 11:22

Changes for External PG

6838868

Signed-off-by: shanmugapriya-tr <shanmugapriya.tiruchengoderamanathan@progress.com>

Merge branch 'saas/managed_services' of https://github.com/chef/automate

96c4047

into saas/managed_services

shanmugapriya-tr requested a review from a team as a code owner March 10, 2022 05:53

github-actions bot added the documentation Anything related to the Automate docs. label Mar 10, 2022

vivek-yadav approved these changes Mar 10, 2022

View reviewed changes

Dmaddu and others added 6 commits March 10, 2022 15:35

Fixing License Scout (#6782)

f4b4aa2

* Fixing License Scout Signed-off-by: Durga Sarat Chandra Maddu <durgasaratchandra.maddu@progress.com> * Fixing License Scout for msgp Signed-off-by: Durga Sarat Chandra Maddu <durgasaratchandra.maddu@progress.com>

Executed '.expeditor/generate-automate-cli-docs.sh'

70727fe

Obvious fix; these changes are the result of automation not creative thinking.

regenerating the cert files (#6794)

1ba6e1b

Signed-off-by: Durga Sarat Chandra Maddu <durgasaratchandra.maddu@progress.com>

Upgrade automate ha packages for 6.8.6 -> 6.8.23 (#6795)

1ed6649

* upgrade es version to 6.8.23 in automate-ha * side car package bump to avaoid the confusion Signed-off-by: punitmundra <punitmundra2008@gmail.com>

shanmugapriya-tr and others added 12 commits March 16, 2022 11:16

Changes for External PG

fc02406

Signed-off-by: shanmugapriya-tr <shanmugapriya.tiruchengoderamanathan@progress.com>

Refactored skipping services health check

4621519

Signed-off-by: shanmugapriya-tr <shanmugapriya.tiruchengoderamanathan@progress.com>

Added few debug logs

f1cb707

Making binds optional

9d379d6

Signed-off-by: shanmugapriya-tr <shanmugapriya.tiruchengoderamanathan@progress.com>

Changed Service names to constants

63d82e4

Signed-off-by: shanmugapriya-tr <shanmugapriya.tiruchengoderamanathan@progress.com>

Refactored code

e831482

Signed-off-by: shanmugapriya-tr <shanmugapriya.tiruchengoderamanathan@progress.com>

Changes to resolve the issue with service restart on patch

cd1d308

Signed-off-by: sandhi <sagarwal@progress.com>

Update components/automate-deployment/pkg/server/server.go

e20ccc2

Co-authored-by: Kallol Roy <karoy@progress.com>

Changed array to map

6044b69

Signed-off-by: shanmugapriya-tr <shanmugapriya.tiruchengoderamanathan@progress.com>

Changes for External PG

c26b6e0

Signed-off-by: shanmugapriya-tr <shanmugapriya.tiruchengoderamanathan@progress.com>

Merge branch 'saas/managed_services' of https://github.com/chef/automate

fc11bc0

into saas/managed_services

kalroy approved these changes Mar 16, 2022

View reviewed changes

kalroy merged commit 1a98306 into main Mar 18, 2022

kalroy deleted the saas/managed_services branch March 18, 2022 05:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Skipping internal setup of ES and PG in case of External config #6606

Skipping internal setup of ES and PG in case of External config #6606

shanmugapriya-tr commented Jan 21, 2022 •

edited

Loading

netlify bot commented Jan 21, 2022 •

edited

Loading

punitmundra left a comment

shanmugapriya-tr commented Mar 10, 2022

shanmugapriya-tr commented Mar 16, 2022

sonarcloud bot commented Mar 16, 2022

shanmugapriya-tr commented Mar 17, 2022

Skipping internal setup of ES and PG in case of External config #6606

Skipping internal setup of ES and PG in case of External config #6606

Conversation

shanmugapriya-tr commented Jan 21, 2022 • edited Loading

🔩 Description: What code changed, and why?

⛓️ Related Resources

👍 Definition of Done

Known issue

[RESOLVED] Certain services not restarting with changes in the ES config

👟 How to Build and Test the Change

✅ Checklist

📷 Screenshots, if applicable

netlify bot commented Jan 21, 2022 • edited Loading

punitmundra left a comment

Choose a reason for hiding this comment

shanmugapriya-tr commented Mar 10, 2022

shanmugapriya-tr commented Mar 16, 2022

sonarcloud bot commented Mar 16, 2022

shanmugapriya-tr commented Mar 17, 2022

shanmugapriya-tr commented Jan 21, 2022 •

edited

Loading

netlify bot commented Jan 21, 2022 •

edited

Loading