HA: multiple instance replicas #85

hardillb · 2023-05-25T08:29:00Z

Description

Sets the correct number of replicas on the Deployment object

Also has support for merging the logs from multiple replicas

To do:

send commands to all instances

Related Issue(s)

FlowFuse/flowfuse#2156

Checklist

I have read the contribution guidelines
Suitable unit/system level tests have been added and they pass
Documentation has been updated
- Upgrade instructions
- Configuration details
- Concepts
Changes flowforge.yml?
- Issue/PR raised on flowforge/helm to update ConfigMap Template
- Issue/PR raised on flowforge/CloudProject to update values for Staging/Production

Labels

Backport needed? -> add the backport label
Includes a DB migration? -> add the area:migration label

When HA enabled call all the endpoints to make sure all instances stay in sync

knolleary · 2023-06-02T10:02:58Z

kubernetes.js

@@ -306,6 +306,12 @@ const createDeployment = async (project, options) => {
        localPod.spec.containers[0].resources.limits.cpu = `${stack.cpu * 10}m`
    }

+    const ha = await project.getSetting('ha')
+    console.log(ha)


Suggested change

console.log(ha)

console.log(ha)

knolleary

Some minor tidy up of commented out code - and an observation on the status endpoint. Would like a follow-up task to be raised to review gathering status of all instances, not just one.

knolleary · 2023-06-02T10:08:33Z

kubernetes.js

@@ -727,6 +748,7 @@ module.exports = {
                (details.body.status?.conditions[0].type === 'Available' ||
                    (details.body.status?.conditions[0].type === 'Progressing' && details.body.status?.conditions[0].reason === 'NewReplicaSetAvailable')
                )) {
+                    // not calling all endpoints for HA as they should be the same


This feels a little risky - in case one instance has hit a hang or other unpleasantness that goes unnoticed.

For an initial iteration, we can go with this, but will need to look at next week.

kubernetes.js

knolleary · 2023-06-02T10:09:18Z

kubernetes.js

+        // const prefix = project.safeName.match(/^[0-9]/) ? 'srv-' : ''
+        // if (await project.getSetting('ha')) {
+        //     const endpoints = await this._k8sApi.readNamespacedEndpoints(`${prefix}${project.safeName}`, this._namespace)
+        //     const addresses = endpoints.body.subsets[0].addresses.map(a => { return a.ip })
+        //     const commands = []
+        //     for (const address in addresses) {
+        //         commands.push(got.post(`http://${addresses[address]}:2880/flowforge/command`, {
+        //             json: {
+        //                 cmd: 'stop'
+        //             }
+        //         }))
+        //     }
+        //     await Promise.all(commands)
+        // } else {
+        //     await got.post(`http://${prefix}${project.safeName}.${this._namespace}:2880/flowforge/command`, {
+        //         json: {
+        //             cmd: 'stop'
+        //         }
+        //     })
+        // }


Suggested change

// const prefix = project.safeName.match(/^[0-9]/) ? 'srv-' : ''

// if (await project.getSetting('ha')) {

// const endpoints = await this._k8sApi.readNamespacedEndpoints(`${prefix}${project.safeName}`, this._namespace)

// const addresses = endpoints.body.subsets[0].addresses.map(a => { return a.ip })

// const commands = []

// for (const address in addresses) {

// commands.push(got.post(`http://${addresses[address]}:2880/flowforge/command`, {

// json: {

// cmd: 'stop'

// }

// }))

// }

// await Promise.all(commands)

// } else {

// await got.post(`http://${prefix}${project.safeName}.${this._namespace}:2880/flowforge/command`, {

// json: {

// cmd: 'stop'

// }

// })

// }

knolleary · 2023-06-02T10:09:53Z

kubernetes.js

+        // const prefix = project.safeName.match(/^[0-9]/) ? 'srv-' : ''
+        // if (project.getSetting('ha')) {
+        //     const endpoints = await this._k8sApi.readNamespacedEndpoints(`${prefix}${project.safeName}`, this._namespace)
+        //     const addresses = endpoints.body.subsets[0].addresses.map(a => { return a.ip })
+        //     const commands = []
+        //     for (const address in addresses) {
+        //         commands.push(got.post(`http://${addresses[address]}:2880/flowforge/command`, {
+        //             json: {
+        //                 cmd: 'restart'
+        //             }
+        //         }))
+        //     }
+        //     await Promise.all(commands)
+        // } else {
+        //     await got.post(`http://${prefix}${project.safeName}.${this._namespace}:2880/flowforge/command`, {
+        //         json: {
+        //             cmd: 'restart'
+        //         }
+        //     })
+        // }


Suggested change

// const prefix = project.safeName.match(/^[0-9]/) ? 'srv-' : ''

// if (project.getSetting('ha')) {

// const endpoints = await this._k8sApi.readNamespacedEndpoints(`${prefix}${project.safeName}`, this._namespace)

// const addresses = endpoints.body.subsets[0].addresses.map(a => { return a.ip })

// const commands = []

// for (const address in addresses) {

// commands.push(got.post(`http://${addresses[address]}:2880/flowforge/command`, {

// json: {

// cmd: 'restart'

// }

// }))

// }

// await Promise.all(commands)

// } else {

// await got.post(`http://${prefix}${project.safeName}.${this._namespace}:2880/flowforge/command`, {

// json: {

// cmd: 'restart'

// }

// })

// }

knolleary · 2023-06-02T10:10:08Z

kubernetes.js

+        // const prefix = project.safeName.match(/^[0-9]/) ? 'srv-' : ''
+        // if (project.getSetting('ha')) {
+        //     const endpoints = await this._k8sApi.readNamespacedEndpoints(`${prefix}${project.safeName}`, this._namespace)
+        //     const addresses = endpoints.body.subsets[0].addresses.map(a => { return a.ip })
+        //     const commands = []
+        //     for (const address in addresses) {
+        //         commands.push(got.post(`http://${addresses[address]}:2880/flowforge/command`, { // logout:nodered(step-4)
+        //             json: {
+        //                 cmd: 'logout',
+        //                 token
+        //             }
+        //         }))
+        //     }
+        //     Promise.all(commands).catch(error => {
+        //         this._app.log.error(`[k8s] Project ${project.id} - error in 'revokeUserToken': ${error.stack}`)
+        //     })
+        // } else {
+        //     try {
+        //         await got.post(`http://${prefix}${project.safeName}.${this._namespace}:2880/flowforge/command`, { // logout:nodered(step-4)
+        //             json: {
+        //                 cmd: 'logout',
+        //                 token
+        //             }
+        //         })
+        //     } catch (error) {
+        //         this._app.log.error(`[k8s] Project ${project.id} - error in 'revokeUserToken': ${error.stack}`)
+        //     }
+        // }


Suggested change

// const prefix = project.safeName.match(/^[0-9]/) ? 'srv-' : ''

// if (project.getSetting('ha')) {

// const endpoints = await this._k8sApi.readNamespacedEndpoints(`${prefix}${project.safeName}`, this._namespace)

// const addresses = endpoints.body.subsets[0].addresses.map(a => { return a.ip })

// const commands = []

// for (const address in addresses) {

// commands.push(got.post(`http://${addresses[address]}:2880/flowforge/command`, { // logout:nodered(step-4)

// json: {

// cmd: 'logout',

// token

// }

// }))

// }

// Promise.all(commands).catch(error => {

// this._app.log.error(`[k8s] Project ${project.id} - error in 'revokeUserToken': ${error.stack}`)

// })

// } else {

// try {

// await got.post(`http://${prefix}${project.safeName}.${this._namespace}:2880/flowforge/command`, { // logout:nodered(step-4)

// json: {

// cmd: 'logout',

// token

// }

// })

// } catch (error) {

// this._app.log.error(`[k8s] Project ${project.id} - error in 'revokeUserToken': ${error.stack}`)

// }

// }

hardillb added 9 commits May 22, 2023 15:02

Move to ha ProjectSettings key

e82fdc9

Fix await for ProjectSettings

cb46576

Move replicas tag to right place

247fd06

Logs from both pods

b5e6610

Debug

6e77265

Fix loop remove debug

dfbf28d

Flatten log array

730325b

Sort combined logs

1eaabfa

Reverse sort order

6878545

hardillb mentioned this pull request May 25, 2023

Instance Load Balancing FlowFuse/flowfuse#2156

Closed

hardillb added 2 commits May 30, 2023 14:22

Make all the command functions call all endpoints

5adc88f

When HA enabled call all the endpoints to make sure all instances stay in sync

fix lint

5c87b79

hardillb requested a review from knolleary May 31, 2023 14:13

hardillb marked this pull request as ready for review May 31, 2023 14:13

hardillb self-assigned this May 31, 2023

hardillb added 2 commits June 1, 2023 17:07

Clean up common code

a09a019

Trim logs function

8dbb906

knolleary reviewed Jun 2, 2023

View reviewed changes

knolleary requested changes Jun 2, 2023

View reviewed changes

hardillb and others added 2 commits June 2, 2023 11:14

Clean up

2a1b795

Update kubernetes.js

daa05a4

knolleary approved these changes Jun 2, 2023

View reviewed changes

knolleary merged commit c650b6a into main Jun 2, 2023
1 check passed

knolleary deleted the 2156-ha-replicas branch June 2, 2023 10:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HA: multiple instance replicas #85

HA: multiple instance replicas #85

hardillb commented May 25, 2023 •

edited

knolleary Jun 2, 2023

knolleary left a comment

knolleary Jun 2, 2023

knolleary Jun 2, 2023

knolleary Jun 2, 2023

knolleary Jun 2, 2023

HA: multiple instance replicas #85

HA: multiple instance replicas #85

Conversation

hardillb commented May 25, 2023 • edited

Description

Related Issue(s)

Checklist

Labels

knolleary Jun 2, 2023

Choose a reason for hiding this comment

knolleary left a comment

Choose a reason for hiding this comment

knolleary Jun 2, 2023

Choose a reason for hiding this comment

knolleary Jun 2, 2023

Choose a reason for hiding this comment

knolleary Jun 2, 2023

Choose a reason for hiding this comment

knolleary Jun 2, 2023

Choose a reason for hiding this comment

hardillb commented May 25, 2023 •

edited