[doc] can we have a complete guideline "Getting started argo CI workflow with gitlab/github"? #8813

larrycai · 2022-05-20T11:23:52Z

larrycai
May 20, 2022

Is it possible to add one getting started document for how to use argo workflow with normal github/gitlab CI project?

Suppose simple python hello world tasks like

triggered using webhook?
checkout code
static check (black/flake8)
pytest unit test

// i want to check how PVC/security is handled normal for those kind of case

security token to clone the code if it is private repo
how to arrange workspace across each steps easily

Answered by larrycai

May 28, 2022

Getting started with github golang project for CI

argo workflows is a nice cloud native workflow execution engine, surely it can work like normal CI engine as well, and it needs to be done through with different setups.

Below is a simple guideline using official documentation and example.

Installation

You need one kubernetes cluster installed with argo server and argo-cli locally, see argo workflows quick start

For client at mac, I use brew command to install directly, it default uses your kube config without extra token to access argo server

$ brew install argo

Getting started with golang project for CI

The example yaml file is ci.yaml@99359a0950, it is used to build golang example proj…

View full answer

alexec · 2022-05-21T07:32:31Z

alexec
May 21, 2022
Collaborator

Would you like to write this doc?

2 replies

larrycai May 21, 2022
Author

Not that ready yet, I'd like to contribute for the examples and polish the solution (if I can).

Argo workflow may not focus on normal CI setup, but it definitely could be a good choice. I am comparing the "gitlab ci, github actions (travis-ci), tekton, argo workflow" different cloud based CI system. And hopefully argo workflow could support those kind of setup in easy way as well.

If you already had those kind of example somewhere, I'd like to write README.md for sure ;-)

larrycai May 21, 2022
Author

like i want to see https://github.com/argoproj/argo-workflows/tree/master/.github/workflows how it works using argo workflow ;-)

larrycai · 2022-05-26T08:38:45Z

larrycai
May 26, 2022
Author

Introduction

It is just my understanding now, can be totally wrong ;-). (@alexec replied below that I had some wrong understanding, but I keep it as history log)

Here gives a short explanation for user wants to use argo workflow as alternative for your CI framework (gitlab-ci, github actions, travis-ci, jenkins)

argo workflow is not a complete CI solution, it needs other compoments to fullfil your tasks. it is cloud native workflow execution engine

If you want to have complete simple solution quickly, recommended to use "jenkins + argoworkflow"

(TODO) Getting started python project using argoworkflow

How CI works

Let's take a look at how CI general works, and how argoworkflow and other ci works in this area

trigger: when the pipeline can be triggered, it needs to monitor git repo, branch, file changes
token: how to access git repo (pull/push)
workspace: how to share the data across steps
archive (optional): how to visible the result (like unit test result)

trigger

argo-workflow doesn't cover this, you can use argo-events or webhook to trigger the flow, for different type of git repo, solution could be different.

If you are familiar with jenkins, it could be good choice as well to have jenkins in front, since it has lots of mature plugin to cover it.

anyway, it is not nature enough like github action and gitlab-ci

token to access git repo

for public repo, just git clone, see sample input-artifact-git.yaml

otherwise it needs token to access, which leads to secrets and sealedsecrets

workspace

for github action, it is VM based, shared workspace via volume sharing, for argo workflow, u need to arrange pvc to be shared

see example ci.yaml, this is a good start!

archive the result

all ci framework has different solutions for it, be prepared.

using extra service like nginx/artifactory could be good simple solution.

Example for github python project

TODO

Jenkins + argo workflow (without events)

Use jenkins plugin mechanism to get triggered and inject flow to argo workflow

Suggestion to argoproj

I do hope argoproject can create argo-ci project again to focus on providing complete CI solution

integrate needed components (argo-events / argo-workflow)
create extra service or enhance argo workflow to make key data are default without extra parameters (token, pvc)
extra UI system (ui for archives/logs/reports ..)
better "plugin" system like github action marketplace, argo-workflows-catalog still has long way to go

Reference

argo-ci (replaced by argo-events now)
webhook in argo-workflow: Trigger workflows from webhook #2667

1 reply

larrycai May 26, 2022
Author

after I provide one complete sample, I will mark this answered

alexec · 2022-05-26T15:37:24Z

alexec
May 26, 2022
Collaborator

This is not quite correct. Triggers are (a) cron schedule (b) web hook.

Argo has both: (a) CronWorkflow and (b) WorkflowEventBinding.

2 replies

larrycai May 28, 2022
Author

noticed feature https://argoproj.github.io/argo-workflows/walk-through/hardwired-artifacts/ to checkout git
and ci.yaml is good example (undocumented argo workflow)

larrycai May 29, 2022
Author

noticed a similar question in discussion #7975 (@Sudhamsh)

alexec · 2022-05-26T15:38:00Z

alexec
May 26, 2022
Collaborator

larrycai
May 28, 2022
Author

Getting started with github golang project for CI

argo workflows is a nice cloud native workflow execution engine, surely it can work like normal CI engine as well, and it needs to be done through with different setups.

Below is a simple guideline using official documentation and example.

Installation

You need one kubernetes cluster installed with argo server and argo-cli locally, see argo workflows quick start

For client at mac, I use brew command to install directly, it default uses your kube config without extra token to access argo server

$ brew install argo

Getting started with golang project for CI

The example yaml file is ci.yaml@99359a0950, it is used to build golang example project https://github.com/golang/example .

In short, do below

$ kubectl create ns argo-ci
$ argo submit -n argo-ci --watch https://raw.githubusercontent.com/argoproj/argo-workflows/master/examples/ci.yaml
Name:                ci-example-6ttdw
Namespace:           argo-ci
ServiceAccount:      unset (will run with the default ServiceAccount)
Status:              Running
Conditions:          
 PodRunning          False
Created:             Sat May 28 15:53:20 +0200 (16 seconds ago)
Started:             Sat May 28 15:53:20 +0200 (16 seconds ago)
Duration:            16 seconds
Progress:            1/4
ResourcesDuration:   3s*(1 cpu),3s*(100Mi memory)
Parameters:          
  revision:          cfe12d6

STEP                                                                                       TEMPLATE              PODNAME                      DURATION  MESSAGE
 ● ci-example-6ttdw                                                                        ci-example                                                     
 ├───✔ build                                                                               build-golang-example  ci-example-6ttdw-2944230572  9s          
 └─┬─◷ test(0:image:debian,tag:9.1)    run-hello             ci-example-6ttdw-984197245   6s          
   ├─◷ test(1:image:alpine,tag:3.6)    run-hello             ci-example-6ttdw-1191131657  6s          
   └─◷ test(2:image:ubuntu,tag:17.10)  run-hello             ci-example-6ttdw-3275768438  6s          

This workflow does not have security context set. You can run your workflow pods more securely by setting it.
Learn more at https://argoproj.github.io/argo-workflows/workflow-pod-security-context/
$ argo logs -n argo-ci @latest. # check the running log via cli
ci-example-6ttdw-2944230572: HEAD detached at cfe12d6
ci-example-6ttdw-2944230572: nothing to commit, working directory clean
ci-example-6ttdw-2944230572: github.com/golang/example/stringutil
ci-example-6ttdw-2944230572: github.com/golang/example/hello
ci-example-6ttdw-984197245: Linux ci-example-6ttdw-984197245 5.4.0-72-generic #80~18.04.1-Ubuntu SMP Mon Apr 12 23:26:25 UTC 2021 x86_64 GNU/Linux
ci-example-6ttdw-984197245: PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
ci-example-6ttdw-984197245: NAME="Debian GNU/Linux"
ci-example-6ttdw-984197245: VERSION_ID="9"
ci-example-6ttdw-984197245: VERSION="9 (stretch)"
ci-example-6ttdw-984197245: ID=debian
ci-example-6ttdw-984197245: HOME_URL="https://www.debian.org/"
ci-example-6ttdw-984197245: SUPPORT_URL="https://www.debian.org/support"
ci-example-6ttdw-984197245: BUG_REPORT_URL="https://bugs.debian.org/"
ci-example-6ttdw-984197245: Hello, Go examples!
...

Use argo-ci namespace

since argo namespace is used to install argo server, I recommend to use another namespace (like argo-ci) for pipeline execution

git checkout

It is stated in Hardwired Artifacts, git is built-in support already.

it is in ci.yaml#L55

      artifacts:
      - name: code
        path: /go/src/github.com/golang/example
        git:
          repo: https://github.com/golang/example.git
          revision: "{{inputs.parameters.revision}}"

the repo and revision can be passed with parameters as template concept, which is skipped here.

workspace

workspace is managed by volumes, which is kubernetes pvc concept, which needs to be specified in each step. depends on your storage in your cluster, it could be nfs or others as default

  volumeClaimTemplates:
  - metadata:
      name: workdir
....
      volumeMounts:
      - name: workdir
        mountPath: /go

Further check

You can further check with argo workflow templates, cron workflows and all in quick-start

Reference

from community

Good article to compare different CI engine - https://www.engineerbetter.com/blog/ci-shootout-inputs-and-outputs/ (Oct 13, 2021 | Tom Godkin)
https://www.sobyte.net/post/2022-01/expirence-of-argo-workflow/

1 reply

larrycai May 28, 2022
Author

@alexec Thanks for your great support, I found something which I don't know before. May share other samples for community (like python)

larrycai · 2022-05-29T10:06:48Z

larrycai
May 29, 2022
Author

argo workflow CI for python project@github

Here is another working example for python https://github.com/larrycai/hello-python/blob/main/.argo/workflows/ci-python.yaml from larrycai/hello-python, which u can compare with github actions

Give a try

$ kubectl create ns argo-ci
$ argo submit -n argo-ci --watch https://raw.githubusercontent.com/larrycai/hello-python/main/.argo/workflows/ci-python.yaml
Name:                ci-python-example-6pslg
Namespace:           argo-ci
ServiceAccount:      unset (will run with the default ServiceAccount)
Status:              Succeeded
Conditions:          
 PodRunning          False
 Completed           True
Created:             Sun May 29 11:52:08 +0200 (1 minute ago)
Started:             Sun May 29 11:52:08 +0200 (1 minute ago)
Finished:            Sun May 29 11:53:30 +0200 (now)
Duration:            1 minute 22 seconds
Progress:            6/6
ResourcesDuration:   42s*(1 cpu),42s*(100Mi memory)

STEP                         TEMPLATE           PODNAME                             DURATION  MESSAGE
 ✔ ci-python-example-6pslg   ci-python-example                                                  
 ├───✔ checkout              checkout           ci-python-example-6pslg-1288181826  12s         
 ├───✔ static-check          flake8             ci-python-example-6pslg-1013104642  8s          
 ├─┬─✔ unittest(0:tag:3.8)   pytest             ci-python-example-6pslg-1356423512  38s         
 │ ├─✔ unittest(1:tag:3.9)   pytest             ci-python-example-6pslg-169406608   14s         
 │ └─✔ unittest(2:tag:3.10)  pytest             ci-python-example-6pslg-2709394423  7s          
 └───✔ summary               run-hello          ci-python-example-6pslg-4250285869  8s          

This workflow does not have security context set. You can run your workflow pods more securely by setting it.
Learn more at https://argoproj.github.io/argo-workflows/workflow-pod-security-context/
````

1 reply

larrycai May 29, 2022
Author

later, it can be improved for DAG and pull request webhook trigger

tim-sendible · 2022-06-10T07:57:56Z

tim-sendible
Jun 10, 2022

I have just been made aware of this discussion. I wrote this, which doesn't quite fulfil what you asked. It aims to demonstrate an Argo Workflows CI with a focus on making it easy to run so that the user can disect it to work out what's going on.

Therefore quite a lot of security is blown out of the water!

https://github.com/sendible-labs/argo-workflows-ci-example

1 reply

larrycai Jun 10, 2022
Author

this is a good example as well to use argo-workflow as ci system

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[doc] can we have a complete guideline "Getting started argo CI workflow with gitlab/github"? #8813

{{title}}

Replies: 7 comments 8 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

[doc] can we have a complete guideline "Getting started argo CI workflow with gitlab/github"? #8813

larrycai May 20, 2022

Getting started with github golang project for CI

Installation

Getting started with golang project for CI

Replies: 7 comments · 8 replies

alexec May 21, 2022 Collaborator

larrycai May 21, 2022 Author

larrycai May 21, 2022 Author

larrycai May 26, 2022 Author

Introduction

How CI works

trigger

token to access git repo

workspace

archive the result

Example for github python project

Jenkins + argo workflow (without events)

Suggestion to argoproj

Reference

larrycai May 26, 2022 Author

alexec May 26, 2022 Collaborator

larrycai May 28, 2022 Author

larrycai May 29, 2022 Author

alexec May 26, 2022 Collaborator

larrycai May 28, 2022 Author

Getting started with github golang project for CI

Installation

Getting started with golang project for CI

Use argo-ci namespace

git checkout

workspace

Further check

Reference

from community

larrycai May 28, 2022 Author

larrycai May 29, 2022 Author

argo workflow CI for python project@github

larrycai May 29, 2022 Author

tim-sendible Jun 10, 2022

larrycai Jun 10, 2022 Author

larrycai
May 20, 2022

Replies: 7 comments 8 replies

alexec
May 21, 2022
Collaborator

larrycai May 21, 2022
Author

larrycai May 21, 2022
Author

larrycai
May 26, 2022
Author

larrycai May 26, 2022
Author

alexec
May 26, 2022
Collaborator

larrycai May 28, 2022
Author

larrycai May 29, 2022
Author

alexec
May 26, 2022
Collaborator

larrycai
May 28, 2022
Author

larrycai May 28, 2022
Author

larrycai
May 29, 2022
Author

larrycai May 29, 2022
Author

tim-sendible
Jun 10, 2022

larrycai Jun 10, 2022
Author