Skip to content

[MDS-6908] Fix PGSync DevOps Workflows#3899

Merged
alazar-aot merged 8 commits into
developfrom
MDS-6908-Fix-PGSync-Promote-Test-Promote-Prod-Workflow
May 7, 2026
Merged

[MDS-6908] Fix PGSync DevOps Workflows#3899
alazar-aot merged 8 commits into
developfrom
MDS-6908-Fix-PGSync-Promote-Test-Promote-Prod-Workflow

Conversation

@alazar-aot
Copy link
Copy Markdown
Collaborator

Objective

MDS-6908

Why are you making this change? Provide a short explanation and/or screenshots

The PGSync - Promote Test and PGSync - Promote PROD GitHub Actions workflows were both non-functional, as PGSync is deployed as part of the Permits service in ArgoCD (mds-permits-*) rather than having its own top-level GitOps directory. This created a structural mismatch with the shared scripts used by all other services. Subsequent compounding issues caused every workflow run to fail at different stages, all of which have been resolved in this PR.

Key Points:

  • The root cause was that the shared commit.sh script constructed the GitOps file paths assumed a {APP}/overlays/{ENV}/ structure, while PGSync's actual path in the GitOps repo was permits/overlays/{ENV}/pgsync/
  • Additionally, pgsync.build.dev.yaml was missing the GitOps update step present in all other dev build workflows, meaning the dev overlay was permanently stuck at git-commit-placeholder and could never be promoted to test.
  • pgsync.deploy.test.yaml was missing NEEDS_ROLLOUT conditions on the Notification step, causing it to run unconditionally and hang on every run, including when there was nothing to deploy.
  • watch-deployment.sh constructed the ArgoCD app name as mds-pgsync-test, which does not exist.
    • PGSync is managed under mds-permits-test
  • argocd app wait on the full permits app caused indefinite hangs due to pre-existing unhealthy resources unrelated to PGSync, making it appear as though PGSync did not deploy successfully when it actually did.
  • The run-if-failed job in both pgsync workflows was completely non-functional - it was missing a checkout step, calling the wrong script (watch-deployment.sh instead of notify_discord.sh), and passing insufficient arguments.
  • notify_discord.sh hardcoded the ArgoCD link as mds-{APP}-{ENV}, producing a broken link for any bundled service like PGSync.

Note that all the changes made are backwards compatible, no existing calls have been broken.

Changes Made:

  • gitops/commit.sh: Added optional argument GITOPS_APP_DIR. When provided, resolves GitOps paths as {GITOPS_APP_DIR}/overlays/{ENV}/{APP}/ instead of the default.
    • Added get_overlay_dir() and validate_patch_path() helper functions for cleaner path handling and fast, understandable failure messages.
  • gitops/watch-deployment.sh: Added optional argument ARGOCD_APP (ArgoCD app name override) and ARGOCD_WAIT (skips argocd app wait when false).
    • Added --grpc-web flag to all ArgoCD CLI calls to silence related warnings.
  • gitops/notify_discord.sh: Added optional argument ARGOCD_APP and updated the ArgoCD embed link to use it.
  • pgsync.build.dev.yaml: Added SSH agent setup and GitOps push step to write the real commit hash to the dev overlay on each build.
    • Added services/pgsync/** to the push path trigger.
  • pgsync.deploy.test.yaml: Passed permits as GITOPS_APP_DIR to commit.sh.
    • Added NEEDS_ROLLOUT conditions to Install oc, oc login, and Notification steps.
    • Bumped timeout-minutes from 10 to 15.
    • Passed mds-permits-test as ArgoCD app override and false for ARGOCD_WAIT to watch-deployment.sh.
    • Fixed run-if-failed: added checkout step, replaced watch-deployment.sh with notify_discord.sh, passed mds-permits-test as ArgoCD app override.
  • pgsync.deploy.prod.yaml: Same changes as test workflow with prod-appropriate values.

alazar-aot added 8 commits May 4, 2026 16:54
  ent to the main gitops script, allowing it to accept a parameter for the pare
  nt directory.
…he script can correctly locate the pgsync deployment in Argo
…Argo wait for the entire service - prevents hangs if another part of the permit service is unhealthy
…ns to provide an ArgoCD app for nested deployments
@alazar-aot alazar-aot requested a review from simensma-fresh May 6, 2026 17:10
@alazar-aot alazar-aot requested review from asinn134 and matbusby-fw May 6, 2026 17:11
@alazar-aot alazar-aot added the 👍 Ready for review Pull request has been double checked by the author and is ready for comments and feedback. label May 6, 2026
@alazar-aot alazar-aot merged commit 81cb72f into develop May 7, 2026
8 checks passed
@alazar-aot alazar-aot deleted the MDS-6908-Fix-PGSync-Promote-Test-Promote-Prod-Workflow branch May 7, 2026 17:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

👍 Ready for review Pull request has been double checked by the author and is ready for comments and feedback.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants