action to update settings for rapidpro workspace #7274

kitsao · 2021-08-28T02:52:10Z

Description

Auto-deployment update for RapidPro deployments as a parameterized and re-usable GitHub action

medic/app-services-team#165

Code review checklist

Readable: Concise, well named, follows the style guide, documented if necessary.
Documented: Added to the action's readme
Tested: See here
Internationalised: All user facing text
Backwards compatible: Works with existing data and configuration or includes a migration. Any breaking changes documented in the release notes.

License

The software is provided under AGPL-3.0. Contributions to this project are accepted under the same license.

derickl

some quick feedback.

Glad to see this coming together @kitsao

derickl · 2021-08-30T06:56:06Z

.github/actions/update-rapidpro-workspace-settings/README.md

+      with:
+        directory: 'my_app_folder'
+        hostname: myapp.staging.company.org
+        username: ${{ secrets.STAGING_USERNAME }}


Consider prefixing the secret variables with RAPIDPRO_ to avoid using the wrong secrets

derickl · 2021-08-30T06:56:10Z

.github/actions/update-rapidpro-workspace-settings/action.yml

+
+  - name: update-medic-credentials
+    run: |
+        curl -X PUT https://${{ inputs.username }}:${{ inputs.password }}@${{ inputs.hostname }}/_node/couchdb@127.0.0.1/_config/medic-credentials/${{ inputs.value_key }} -d '"${{ inputs.token }}"'


CouchDB would not be running on 127.0.0.1. Did you intend to use ${{inputs.hostname}}

Nice. Should point to the local COUCH_NODE_NAME

eljhkrr

Looks good overall. Some improvement recommendations shared in-line.

eljhkrr · 2021-09-07T14:20:17Z

.github/actions/update-rapidpro-workspace-settings/action.yml

+
+  - name: update-app-settings
+    run: |
+        jq 'walk(if type == "object" and has("base_url") then .base_url = "${{ inputs.url }}" else . end) | walk(if type == "object" and has("value_key") then .value_key = "${{ inputs.value_key }}" else . end) | walk(if type == "object" and has("flow") then .flow.expr = "${{ inputs.write_patient_state_flow }}" else . end) | walk(if type == "object" and has("groups") then .groups.expr = "[${{ inputs.group }}]" else . end)' app_settings.json > app_settings.json.tmp && cp app_settings.json.tmp app_settings.json && rm app_settings.json.tmp


It's possible to write app_settings.json directly

is jq the best way to do this? i suspect you'd really improve readability if this was (for example) a node script

const appSettings = require('./app_settings.json'); .. some stuff to generate app_settings.json

eljhkrr · 2021-09-07T14:38:25Z

.github/actions/update-rapidpro-workspace-settings/action.yml

+    description: The password for user with name "inputs.username"
+    required: true
+
+  url:


Recommend renaming to rapid_pro url to avoid confusion

eljhkrr · 2021-09-07T14:47:33Z

.github/actions/update-rapidpro-workspace-settings/action.yml

+    description: The username of an administrator account on the instance at "inputs.hostname". This user will make the deployment.
+    required: true
+
+  password:


hostname, username & password can be reused from deploy-with-medic-conf

Maybe, if the two jobs are used together.

eljhkrr · 2021-09-07T14:50:42Z

.github/actions/update-rapidpro-workspace-settings/action.yml

+
+  flows: 
+    description: A flows.js file to be used in compile-app-settings
+    value: ${{ steps.generate-flows-js.outputs }}


flows.js generated in L83

kennsippell

Nice work

kennsippell · 2021-09-07T22:13:20Z

.github/actions/update-rapidpro-workspace-settings/README.md

+# update-rapidpro-workspace-settings Shared GitHub Action
+The `update-rapidpro-workspace-settings` is a parameterised reusable GitHub action that updates app-settings prior to compiling and uploading to a running CHT instance.
+
+This can be executed jointly with the deployment Github action.


Suggested change

This can be executed jointly with the deployment Github action.

This can be executed jointly with other Github actions like [deploy-with-medic-conf](https://github.com/medic/cht-core/tree/master/.github/actions/deploy-with-medic-conf)

kennsippell · 2021-09-07T22:14:11Z

.github/actions/update-rapidpro-workspace-settings/README.md

+        directory: 'my_app_folder'
+        hostname: myapp.staging.company.org
+        couch_node_name: myapp.couchdb.node.name
+        username: ${{ secrets.CHT_STAGING_USERNAME }}


Suggested change

username: ${{ secrets.CHT_STAGING_USERNAME }}

couch_username: ${{ secrets.CHT_STAGING_USERNAME }}

kennsippell · 2021-09-07T22:14:18Z

.github/actions/update-rapidpro-workspace-settings/README.md

+        hostname: myapp.staging.company.org
+        couch_node_name: myapp.couchdb.node.name
+        username: ${{ secrets.CHT_STAGING_USERNAME }}
+        password: ${{ secrets.CHT_STAGING_PASSWORD }}


Suggested change

password: ${{ secrets.CHT_STAGING_PASSWORD }}

couch_password: ${{ secrets.CHT_STAGING_PASSWORD }}

kennsippell · 2021-09-07T22:14:30Z

.github/actions/update-rapidpro-workspace-settings/README.md

+        couch_node_name: myapp.couchdb.node.name
+        username: ${{ secrets.CHT_STAGING_USERNAME }}
+        password: ${{ secrets.CHT_STAGING_PASSWORD }}
+        url: my.rapidpro.workspace.url


Suggested change

url: my.rapidpro.workspace.url

rp_hostname: my.rapidpro.workspace.url

kennsippell · 2021-09-07T22:14:39Z

.github/actions/update-rapidpro-workspace-settings/README.md

+        username: ${{ secrets.CHT_STAGING_USERNAME }}
+        password: ${{ secrets.CHT_STAGING_PASSWORD }}
+        url: my.rapidpro.workspace.url
+        token: ${{ secrets.RAPIDPRO_STAGING_TOKEN }}


Suggested change

token: ${{ secrets.RAPIDPRO_STAGING_TOKEN }}

rp_api_token: ${{ secrets.RAPIDPRO_STAGING_TOKEN }}

kennsippell · 2021-09-07T22:15:09Z

.github/actions/update-rapidpro-workspace-settings/README.md

+        url: my.rapidpro.workspace.url
+        token: ${{ secrets.RAPIDPRO_STAGING_TOKEN }}
+        value_key: medic.credentials.key
+        group: ${{ secrets.RAPIDPRO_STAGING_GROUP }}


Suggested change

group: ${{ secrets.RAPIDPRO_STAGING_GROUP }}

rp_contact_group: ${{ secrets.RAPIDPRO_STAGING_GROUP }}

kennsippell · 2021-09-07T22:15:29Z

.github/actions/update-rapidpro-workspace-settings/README.md

+        token: ${{ secrets.RAPIDPRO_STAGING_TOKEN }}
+        value_key: medic.credentials.key
+        group: ${{ secrets.RAPIDPRO_STAGING_GROUP }}
+        flows: ${{ secrets.RAPIDPRO_STAGING_FLOWS }}


Suggested change

flows: ${{ secrets.RAPIDPRO_STAGING_FLOWS }}

rp_flows: ${{ secrets.RAPIDPRO_STAGING_FLOWS }}

kennsippell · 2021-09-07T22:18:11Z

.github/actions/update-rapidpro-workspace-settings/action.yml

+
+  - name: update-app-settings
+    run: |
+        jq 'walk(if type == "object" and has("base_url") then .base_url = "${{ inputs.url }}" else . end) | walk(if type == "object" and has("value_key") then .value_key = "${{ inputs.value_key }}" else . end) | walk(if type == "object" and has("flow") then .flow.expr = "${{ inputs.write_patient_state_flow }}" else . end) | walk(if type == "object" and has("groups") then .groups.expr = "[${{ inputs.group }}]" else . end)' app_settings.json > app_settings.json.tmp && cp app_settings.json.tmp app_settings.json && rm app_settings.json.tmp


is jq the best way to do this? i suspect you'd really improve readability if this was (for example) a node script

const appSettings = require('./app_settings.json'); .. some stuff to generate app_settings.json

kennsippell · 2021-09-07T22:24:03Z

.github/actions/update-rapidpro-workspace-settings/README.md

@@ -0,0 +1,60 @@
+# update-rapidpro-workspace-settings Shared GitHub Action
+The `update-rapidpro-workspace-settings` is a parameterised reusable GitHub action that updates app-settings prior to compiling and uploading to a running CHT instance.


I don't think this summary does a particularly thorough job of explaining what is happening here.

Who is this relevant for? This is relevant only for CHT apps that have either outbound integrations with RP, or which have logic which requires RP guids in the flows. Is that right?

Introduce the problem? I've got a dev and a production rapidpro workspace. I want my dev deployment to use the dev workspace and my prod deployment to use the prod workspace. Is that accurate?

What are the steps you should go through to generate the dev flow.js (can they just run the script below?). Given a very basic rapidpro setup - can you provide a sample flow.js file?

Users work with a dev version of a "flow.js" file in their repository. This script replaces the "flow.js" file with an auto-generated file created from data returned by the RP API

What attributes are expected in your app_settings.json if any?

kennsippell · 2021-09-07T22:28:09Z

.github/actions/update-rapidpro-workspace-settings/README.md

@@ -0,0 +1,60 @@
+# update-rapidpro-workspace-settings Shared GitHub Action


Suggested change

# update-rapidpro-workspace-settings Shared GitHub Action

# dynamic-rapidpro-workspace Shared GitHub Action

Nit - maybe rename to be about the problem that is being solved rather than the action that the tool is doing?

kitsao · 2021-09-21T15:40:09Z

@kennsippell, I have following script in the same folder as the action.yml

const replace = require('replace-in-file');
const appSettings = require('app_settings.json');
const rp_hostname = process.env.rp_hostname;
const value_key = process.env.value_key;
const rp_contact_group = process.env.rp_contact_group;
const write_patient_state_flow = process.env.write_patient_state_flow;

const search = (haystack, needle) => needle in haystack ? haystack[needle] : Object.values(haystack).reduce((acc, val) => {
  if (acc !== undefined) {
    return acc;
  } 
  if (typeof val === 'object') return search(val, needle);
}, undefined);

const regex = expr => new RegExp(expr, 'g');

const options = {
  files: './app_settings.json',
  from: [regex(search(appSettings, 'base_url')), regex(search(appSettings, 'value_key')), search(appSettings, 'groups').expr, search(appSettings, 'flow').expr],
  to: [rp_hostname, value_key, `['${rp_contact_group}']`, `'${write_patient_state_flow}'`]
};

replace(options);

I tested and works. I am not sure how to run it since working-directory will not apply to the script, throwing a Error: Cannot find module ...
Thoughts?

kennsippell · 2021-09-23T05:31:30Z

@kitsao Does this help https://docs.github.com/en/actions/creating-actions/creating-a-javascript-action ?

kitsao · 2021-09-23T10:18:52Z

It does @kennsippell. From a first attempt, it requires committing node_modules here, which I think is not recommended. I'm still exploring though.

kitsao · 2021-09-23T14:08:41Z

remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.

derickl · 2021-09-23T18:38:35Z

If the commit with the node_modules is the most recent, you can run the following to unstage the folder

git reset --soft HEAD~1

Install GIt LFS from https://git-lfs.github.com

git lfs track node_modules
git add .gitattributes
git add node_modules
git commit -m 'your message'
git push

You'll may need to update your action to checkout with 'lfs' like actions/checkout#270 (comment)

kitsao · 2021-09-27T10:16:32Z

After carefully reworking this, I am blocked by Error: Cannot find module './app_settings.json'. Rightfully so, when I compile the bundle using Vercel, the expectation is that app_settings.json is accessible as provided in require('./app_settings.json'), but note that this file is available in a secondary repository that should be provided by inputs. Is there something I am missing?

See playground action here.

Looks like I so far have two options:

Retain the current code
Move the action to specific RapidPro deployment repos so that app_settings.json is available to the compiled bundle

Thoughts @kennsippell @derickl @eljhkrr

kitsao · 2021-09-28T14:38:41Z

Resolving the full path to the project root folder did the trick cc @derickl. On to the next, have to perform CURL and create flows.js file using the JavaScript file. It'll be done shortly.

kennsippell

@kitsao who are you engaging from cht-core team on this? You will need sign-off from that group.

kennsippell · 2021-10-13T17:22:19Z

.github/actions/update-rapidpro-workspace-settings/README.md

+Normally, you would want to a staging deployment use the staging workspace and a production deployment to use the corresponding production workspace. This action automates the process of updating `app_settings.json` to use relevant UUIDs.
+
+### How it works
+1. It adds the configured `value_key` to CouchDB’s config storage to securely store the [credentials](https://docs.communityhealthtoolkit.org/apps/reference/app-settings/outbound/#credentials).


i don't get why this is required... shouldn't the CouchDB credentials be setup once? why do they need to change with code deployment?

value_keys might not always be the same, could be someone's preference.

We do this to be sure that we have the right credentials for each push to production. For instance, more recently in I-TECH Aurum, we found out that the the value_key in app settings was not matching that in CouchDB and we couldn't track to when the change occurred.

i would recommend removing this from the script for two reasons.

that's a weird story with I-Tech Aurum, but i'm not sure that setting the credentials in github secrets is superior than setting it in couchdb itself. the potential for password admin problems is the same, you've just moved it to a different place. or have i missed something?

i'd prefer actions which do one thing well, rather than cluttering the interface with details required for two coupled but unrelated activities.

You might have missed something. The password given in secrets is set in CouchDB here.

.github/actions/update-rapidpro-workspace-settings/README.md

kennsippell · 2021-10-13T17:44:16Z

.github/actions/update-rapidpro-workspace-settings/index.js

+  const flowsContent = `module.exports = ${rp_flows};`;
+
+  setMedicCredentials(couch_username, couch_password, hostname, couch_node_name, value_key, rp_api_token);
+  replace(options);


this file-wide replacement seems dangerous. you're string replacing even across the compiled javascript code? how can this be more clearly targeted to only relevant areas of config. should this run on base_config before running deployment? should this be limited to specific branches of the JSON tree?

This action should be executed before other actions that is is jointly executed with such as deploy-with-medic-conf, so that dependent pieces of code are updated prior to compiling app settings. Otherwise, if executed later, we'll end up with broken updates.

so doesn't this have to operate on base_settings or the code you replace in app_settings will simply be overwritten by the next compile-app-settings during deployment?

By the time compile-app-settings runs, we'll have updated base settings. Meaning, we end up with the expected behavior. And yes, every compile-app-settings overwrites app_settings.json if there's a change in base settings. Does the following diagram help?

garethbowen · 2024-01-25T09:16:59Z

It looks like this is obsolete (solved by medic/gh-actions#1 ). Closing - please reopen if I've missed the point!

kitsao added 2 commits August 28, 2021 05:39

action to update settings for rapidpro workspace

fdf9be5

removed default group

397c10d

kitsao requested review from eljhkrr, kennsippell, abbyad and derickl August 28, 2021 02:52

derickl reviewed Aug 30, 2021

View reviewed changes

updated action, readme

8ef829f

kitsao requested a review from derickl September 3, 2021 11:54

eljhkrr approved these changes Sep 7, 2021

View reviewed changes

kennsippell requested changes Sep 7, 2021

View reviewed changes

kennsippell reviewed Sep 7, 2021

View reviewed changes

pr feedback

e81ed65

kitsao added 4 commits September 29, 2021 12:19

javascript action

3c75c7f

updated readme

0f4a0bf

regex update

486b2e8

Merge branch 'medic:master' into master

1e6f056

kitsao requested a review from kennsippell September 29, 2021 11:14

kitsao added 3 commits October 1, 2021 06:26

minor update

5c193f8

Merge branch 'master' of https://github.com/kitsao/cht-core

7e68f55

Merge branch 'medic:master' into master

b27ea74

kennsippell requested changes Oct 13, 2021

View reviewed changes

kennsippell reviewed Oct 13, 2021

View reviewed changes

kitsao marked this pull request as draft March 7, 2022 10:41

kitsao added 27 commits March 9, 2022 15:50

updates + initial test

818c814

refactoring

69f1fae

Merge branch 'master' of https://github.com/kitsao/cht-core

2661044

more refactoring

27df4ff

tests

852dbb4

refactor, tests

e310c06

log

369b7d4

licences

4b42098

remove console.log

61fd922

updates

5bf27b4

rid off path

63c58bb

minor changes

ab11e65

minor changes

188e5e1

path changes

015b9f4

module not found

e0ec7c8

use fs to read

aaaf5cd

cleanup

166f8bb

secrets not defined

0a7d1b7

undefined variable

87731d1

try

d7db057

500 statucode

a6b3bbc

refactor

0c878c7

more refactoring

300a579

pause axios

20fcdd1

console log

8104ca9

axios fix

849215e

more refactoring

30946fb

garethbowen closed this Jan 25, 2024

	This can be executed jointly with the deployment Github action.
	This can be executed jointly with other Github actions like [deploy-with-medic-conf](https://github.com/medic/cht-core/tree/master/.github/actions/deploy-with-medic-conf)

	username: ${{ secrets.CHT_STAGING_USERNAME }}
	couch_username: ${{ secrets.CHT_STAGING_USERNAME }}

	password: ${{ secrets.CHT_STAGING_PASSWORD }}
	couch_password: ${{ secrets.CHT_STAGING_PASSWORD }}

	url: my.rapidpro.workspace.url
	rp_hostname: my.rapidpro.workspace.url

	token: ${{ secrets.RAPIDPRO_STAGING_TOKEN }}
	rp_api_token: ${{ secrets.RAPIDPRO_STAGING_TOKEN }}

	group: ${{ secrets.RAPIDPRO_STAGING_GROUP }}
	rp_contact_group: ${{ secrets.RAPIDPRO_STAGING_GROUP }}

	flows: ${{ secrets.RAPIDPRO_STAGING_FLOWS }}
	rp_flows: ${{ secrets.RAPIDPRO_STAGING_FLOWS }}

		@@ -0,0 +1,60 @@
		# update-rapidpro-workspace-settings Shared GitHub Action
		The `update-rapidpro-workspace-settings` is a parameterised reusable GitHub action that updates app-settings prior to compiling and uploading to a running CHT instance.

	# update-rapidpro-workspace-settings Shared GitHub Action
	# dynamic-rapidpro-workspace Shared GitHub Action

action to update settings for rapidpro workspace #7274

action to update settings for rapidpro workspace #7274

Conversation

kitsao commented Aug 28, 2021

Description

Code review checklist

License

derickl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eljhkrr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kennsippell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kennsippell Sep 7, 2021 • edited Loading

Choose a reason for hiding this comment

kennsippell Sep 7, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kitsao commented Sep 21, 2021

kennsippell commented Sep 23, 2021

kitsao commented Sep 23, 2021

kitsao commented Sep 23, 2021

derickl commented Sep 23, 2021

kitsao commented Sep 27, 2021 • edited Loading

kitsao commented Sep 28, 2021

kennsippell left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kennsippell Oct 14, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kennsippell Oct 14, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

garethbowen commented Jan 25, 2024

kennsippell Sep 7, 2021 •

edited

Loading

kennsippell Sep 7, 2021 •

edited

Loading

kitsao commented Sep 27, 2021 •

edited

Loading

kennsippell left a comment •

edited

Loading

kennsippell Oct 14, 2021 •

edited

Loading

kennsippell Oct 14, 2021 •

edited

Loading