Crossview DevOps improvement

[MoeidHeidari]

Dear contributors,

Thank you for considering contributing your time and expertise to CrossView.
Your help means a lot to the community. Our goal is to continuously improve this tool to make life significantly easier for infrastructure engineers monitoring their Crossplane infrastructure in a multi-cluster Kubernetes environment.

I’ve already received a lot of positive feedback and strong interest from many DevOps engineers who are eager to join the discussions and contribute to enhancing the DevOps aspects of this project.

Let’s discuss the ideas together here and keep all eyes on these improvement points. I truly value your expertise and would love your input to help build a great product that we can all be proud of.

[khushiisystem]

GitHub Workflow Review – Problems & Solutions

1. No Standard Branch-Based Release Strategy

Problem:
No clear separation between development, testing, and production releases.

Solutions:

Define branches:

• main → Production
• pre-release → RC testing
• feature/* → Development

Enable branch protection:

• Require PR approval
• Require CI checks before merge

---

2. Releases Are Fully Automated

Problem:
Releases are triggered automatically without manual validation.

Solution:

• Use workflow_dispatch for manual trigger
• Add environment approval in GitHub Actions:
  1. Create environment: production
  2. Require reviewer approval before release
• Add input parameters:
  1. Version number
  2. Release notes

---

3. Missing Semantic Versioning Control

Problem:
Versioning is not standardized (Major/Minor/Patch not enforced).

Solution:

• Enforce version pattern: v1.0.0, v1.2.3
• Add validation step:

  if [[ ! "$VERSION" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then exit 1; fi

• Auto-generate versions using:
  • Commit messages (Conventional Commits)
  • Or version bump scripts

---

4. Weak CI Validation for Helm Charts

Problem:
Helm charts may be released without proper validation.

Solution:

Add CI steps:

• helm lint
• helm template
• helm install --dry-run

Add schema validation:

• helm schema-gen values.yaml

---

5. ArtifactHub Sync Not Guaranteed

Problem:
ArtifactHub may not reflect the latest chart.

Solutions:

• Automate chart packaging: helm package .
• Update repo index: helm repo index .
• Push to GitHub Pages or chart repo
• Ensure artifacthub-repo.yml is valid

---

6. Missing Quality & Security Checks

Problem:
No strong validation before merge.

Solutions:

Add checks:

• YAML validation: yamllint .
• Kubernetes validation: kubectl apply --dry-run=client -f
• JSON/schema validation tools

---

7. No Safeguards for Release Execution

Problem:
Accidental or duplicate releases possible.

Solutions:

• Allow releases only on tags (v*)
• Add check:

  git tag -l | grep "$VERSION" && exit 1

• Restrict workflow to:

  if: startsWith(github.ref, 'refs/tags/')

---

8. No Automated Testing

Problem:
No functional validation of application/chart.

Solutions:

• Add test stage: deploy to temporary Kubernetes (Kind / Minikube)
• Run:

  helm install test .
  kubectl get pods

---

9. No Changelog Management

Problem:
No structured release notes.

Solutions:

Auto-generate changelog using:

• git log
• Or tools like:
  • conventional-changelog

Attach changelog to GitHub release.

---

10. No Rollback Strategy

Problem:
Failed releases cannot be easily reverted.

Solutions:

• Keep previous chart versions
• Use Helm rollback: helm rollback

---

11. No Notifications / Visibility

Problem:
Team is not notified about pipeline status.

Solutions:

Integrate:

• Slack / Email notifications

Notify on:

• Success
• Failure
• Release completion

---

12. No Secrets Management Best Practices

Problem:
Secrets may be exposed or mismanaged.

Solutions:

• Use GitHub Secrets
• Avoid hardcoding credentials
• Rotate secrets periodically

---

13. No Pipeline Modularity

Problem:
Workflow is hard to maintain and scale.

Solutions:

Split workflows:

• ci.yaml
• release.yaml
• rc.yaml

Use reusable workflows (workflow_call).

---

14. No Caching (Slow Builds)

Problem:
Pipeline execution is slow.

Solutions:

Enable caching:

uses: actions/cache@v3

Cache:

• Helm dependencies
• Docker layers (if used)

from @khushiisystem

[MoeidHeidari]

thanks @khushiisystem for the great analysis. here are my comments.

1- Even though the branch versioning is weak, I believe we still have some options in place. At least we have the main and pre-release. and feature branches, which are the branches from our contributors.

regarding the branch protection, it is already protected on branch main which requires PR approval DOC checks. I guess we have some CI checks also such as helm test and security tests. but do you think we can/need to improve it?

2- Fair point. it shouldnt be automated at all.

3- fair point
4- we have a release candidate workflow. if the changes are from pre-release branch then they will be considered as rc. if you take a look at release.yaml line number 127

5- regarding helm. we have the test cases there. in addition to lint checks it also runs the helm test cases helm/scripts/test.sh. also check helm-test.yaml . starting from line 33. but maybe you have a suggestion to improve it.

6- artifacthub sync works. the issue is in order to have it synced after making the release we need to do a helm release in orther to push the release to artifacthub. And sometimes it can bring an uncynced versions between helm and packages. do you have any idea about that? we have some security tests happening but i guess you are talking about kubernetes manifests, is that right?

7- fair point. it actually happened some times. would be nice to prevent version increase when there is a version already available with the same number. and also preventing the jumps. I remember it was a time it just jumped two minor versions.

8- I guess that wouldnt be possible at this stage since it needs an infra to have the temp k8s setup.

9- fair point. we have the change log generated but it needs the content of the changelog.md to be updated. when release happens.

10- fair point

11- we used to have a slack workspace before migrating to crossplane-contrib organization. where we could get slack notifications. but then after migration we lost that integration. but I guesss if contributors have email activated as their notification channel then they would get the notifs actually, any idea?

12- but we do have our secrets in github secret, dont we?

13- there some separation for example for helm tests, security checks and main release pipeline. but I guess you ment release pipeline phase separations. correct?

14- fair point. that is what we suffer from the most.

[SukhbirSinghKhalsa]

Do we have any documents or way to tryout the project in our local environment setup?

What strategy is currently followed for the releases?

From DevOps side what is expectation here?

[MoeidHeidari]

yes. refer to these
https://github.com/crossplane-contrib/crossview/blob/main/docs/CONFIGURATION.md
https://github.com/crossplane-contrib/crossview/blob/main/docs/CONTRIBUTING.md

[jmac052002]

Thanks for this thorough breakdown this gives great structure to work from. A few of these resonate strongly with me, particularly #1 (branch strategy), #8 (automated testing with Kind/Minikube), and #14 (caching for slow builds). I've been working with GitHub Actions and Terraform-based pipelines and would like to start contributing on the CI/CD side. Happy to pick up any of these items once we align on priorities.

[MoeidHeidari]

greate @jmac052002. I have already walked trough some of the,. i will post my thoughts here shortly . but would be nice if you also take a look and compare the points we have here and the current situation in the project to find the potential points to start from.

[MoeidHeidari]

What does DevOps look like?
As you know, Crossview is a monorepo. Means we have both frontend and backend in the same repository. Good point, we have both backend and frontend in the same container. We proxy and expose the frontend through the backend. So we don't need to have separate pipelines. But the thing is we have both package releasing and helm chart releasing. we provide the helm chart in github pages and also we have published it in artifacthub which reads directly from helm builds in the github pages.

what is really important is to stay synced when we release a helm version (rc,major or minor).

also caching is a problem as for now. every time we merge a pr it starts from scratch build the multi-arch image and it takes way too long time. when making the pr it goes all the way to the release eventhough we havent merged the PR yet.

[jmac052002]

Thanks for the context on the monorepo setup that's really helpful for understanding the pipeline scope. The caching issue and premature release triggering on unmerged PRs stand out to me as high-impact starting points. I'll dig into the current workflows and compare against what's been discussed here. Looking forward to your thoughts when you post them.

[MoeidHeidari]

Sure. thanks @jmac052002 . good to have you here

[MoeidHeidari]

I posted my comments on the improvement proposal. sorry for typos. Tried to be fast.

[MoeidHeidari]

BTW, we have the Headlamp plugin here https://github.com/MoeidHeidari/crossview-headlamp
https://artifacthub.io/packages/headlamp/crossview-headlamp/crossview-headlamp

[MoeidHeidari]

Here are some issues I defined based on our discussion so far. Feel free to take any of them you like the most.
#206 #207 #208 #209 #210

[jmac052002]

Congrats on the v3.8.0 release! I see the DevOps items made it in. I'm still here and ready to pick up whatever comes next whenever you're ready.

[MoeidHeidari]

thanks @jmac052002 . Good to have you all around. We have a couple of issues left. Feel free to pick whatever you find fun to work on.

[MoeidHeidari]

Thanks everyone for the greate contribution and the improvement you made in Crossview. Feel free to join our slack channel in Crossplane workspace #crossview
https://join.slack.com/t/crossplane/shared_invite/zt-3u3767c5j-t_ApimufOO55csGnHoJ27A