Syncing configuration to the new node after node add

[ArvinthC3000]

🔩 Description: What code changed, and why?

As an Automate HA user, when the user adds a new node, it should also add/sync up all the configs added in the bootstrap Frontend node.

⛓️ Related Resources

Jira ticket: https://chefio.atlassian.net/browse/CHEF-2712

👍 Definition of Done

Following are the scenarios covered in testing:

• On-Prem with Chef-embedded - Add node.
• On-Prem with Chef-embedded - Remove node (x) Need to check.
• On-prem with Chef-embedded and custom cert - Add node.
• On-prem with Chef-embedded and custom cert - Remove node.
• On-prem with Manged services - Add/Remove node.
• Any Infra with Chef-embedded - Add/Remove OS and PG node.
• AWS with Chef-embedded - Add node.
• AWS with Chef-embedded - Remove the bootstrap node (x) Not working.
• AWS with Managed AWS - Add node.
• AWS with Managed AWS Remove the bootstrap node and check whether the config is still intact.

Note:

• Sync-up will happen only in frontend nodes, as the BE nodes run in hab cluster
• Sync-up will ignore the load balance tls values as that will cause issue in case the HA setup uses unique cert in each node.

Why testing on Remove node flow?

Because of the following behavior of nodes getting replaced, the sync up is done in remove node flow so that the newly added nod will also have all config as well.
Before removing nodes. Automate nodes:

Service	IP Address	Status	Opensearch
automate	10.0.192.105	OK	green (Active: 100.0)
automate	10.0.192.159	OK	green (Active: 100.0)
automate	10.0.192.222	OK	green (Active: 100.0)	<- removing this node with 'node remove' command (index:2)
automate	10.0.192.90	OK	green (Active: 100.0)
automate	10.0.192.164	OK	green (Active: 100.0)	<- This is being replaced (remove and create resource) from index:4 to index:2

After remove node:

Service	IP Address	Status	Opensearch
automate	10.0.192.105	OK	green (Active: 100.0)
automate	10.0.192.159	OK	green (Active: 100.0)
automate	10.0.192.212	OK	green (Active: 100.0)	. <- newly added node does not have dex config (this is replacement of node in index:4 before removing node )
automate	10.0.192.90	OK	green (Active: 100.0)

👟 How to Build and Test the Change

✅ Checklist

All PRs must tick these:

• I have read the CONTRIBUTING document.
• All commits signed-off for the Developer Certification of Origin.

With occasional exceptions, all PRs from Progress employees must tick these:

• Is the code clear? (complicated code or lots of comments--subdivide and use well-named methods, meaningful variable names, etc.)
• Consistency checked? (user notifications, user prompts, visual patterns, code patterns, variable names)
• Repeated code blocks eliminated? (adapt and reuse existing components, blocks, functions, etc.)
• Spelling, grammar, typos checked? (at a minimum use make spell in any component directory)
• Code well-formatted? (indents, line breaks, etc. improve rather than hinder readability)

All PRs from Progress employees should tick these if appropriate:

• Tests added/updated? (all new code needs new tests)
• Docs added/updated? (all customer-facing changes)

Please add a note next to any checkbox above if you are NOT ticking it.

📷 Screenshots, if applicable

Demo Videos:
Add node in AWS:
https://progresssoftware.sharepoint.com/:v:/s/ChefCoreC/ESX5RAGEcJxDiS8JjKc4HgQBjP9rHKRU-v_HSZ9voEG-sQ?e=Ejn8xR

Add node in On-prem Managed services:
https://progresssoftware.sharepoint.com/:v:/s/ChefCoreC/Ef7rnGtWzgtPk-WZlSSB2-kB88vaVmopRaynYB2f5kCfTw?e=WKsGbw

Docs and screenshots:
Config of Newly Added node without this PR change: https://progresssoftware-my.sharepoint.com/:u:/g/personal/archand_progress_com/EagitM3p-NtPhWfHg-ZBLh8B8-OTYU35dzn5F8e_q5vfmg?e=qy9v45

Config of Newly Added node with this PR change: https://progresssoftware-my.sharepoint.com/:u:/g/personal/archand_progress_com/EYOaHHNXdHVBmcIsIVlvolcB3vvSIqvHpKtuOl_XN-3cvw?e=lmslsy

Service status after adding node and applying the new config

Test coverage: (Overall test coverage on Sonarqube for code written in this PR: 86%)

[netlify]

👷 Deploy Preview for chef-automate processing.

Name	Link
🔨 Latest commit	fdd981a
🔍 Latest deploy log	https://app.netlify.com/sites/chef-automate/deploys/6493e9ed5dcfc2000853b83a

[vivek-yadav]

Suggested change

	func ExecuteCustomCmdInServiceNode(outputFiles []string, inputFiles []string, inputFilesPrefix string, service string, cmdString string, singleNode bool) error {
	func ExecuteCustomCmdOnEachNodeType(outputFiles []string, inputFiles []string, inputFilesPrefix string, service string, cmdString string, singleNode bool) error {

[ArvinthC3000]

Done

[vivek-yadav]

this is needed in other commands, don't remove Host IP

[ArvinthC3000]

Done

[vivek-yadav]

move this to ha_node_operations_utils.go
Also rename this, with proper explanation name

[ArvinthC3000]

Done

[vivek-yadav]

you cannot import function from lib which uses this utility

[ArvinthC3000]

Removed the dependency from other libs

[vivek-yadav]

getAutomateHAInfraDetails is not in util, don't use it here.

[ArvinthC3000]

Done.

[vivek-yadav]

Suggested change

	func ServiceNodeConstructor(nodeMap *NodeTypeAndCmd, cmdString string, outputFiles []string, singleNode bool) *Cmd {
	func NewNodeTypeCmd(nodeMap *NodeTypeAndCmd, cmdString string, outputFiles []string, singleNode bool) *Cmd {

[ArvinthC3000]

Updated

[vivek-yadav]

we should run this before error check

[ArvinthC3000]

Done

[vivekshankar1]

Wrap all errors with distinct strings

[ArvinthC3000]

Done

[sonarqube-for-infrastructure-prod]

SonarQube Quality Gate

22.7% Coverage on New Code (is less than 80%)

See analysis details on SonarQube