An internal CI/CD system compatible with Github actions workflow yaml syntax, action yaml syntax and most action plugins

[lunny]

Internal CI/CD system

I propose to implement an internal CI/CD system that is compatible with Github workflows yaml syntax, action yaml syntax, and most action plugins. And this is in fact not only a CI/CD system, but you can also choose to not check out the code for some events for example, when an issue is posted, you want to reply to them, then you don't need to check out the code. You can just reply to that issue. So the sub-system name is botsactions, if you have a better name than that, please let me know.

Pros

• Migrated repositories from Github could easily reuse the actions workflow files
• Plugins could be copied from https://github.com/actions or https://gitea.com/actions to your self-hosted Gitea instances or any git service repository address
• Transparent for the CI/CD process. You can check out all workflow files and plugin codes you are using.
• Bring version management for your CI/CD scripts and plugins
• Customize your image. Github's image for ubuntu-latest is very big but Gitea now will use node:16-buster as the default image which is about 150MB and of course you could change it yourself.

Cons

• Images are maybe very big for those small devices and have to be changed when you have new dependencies.
• Many Gitea users are familiar with Drone/Woodpecker
• More feedbacks are welcome

Reference documentation

At the moment, we lack documents, but if you want to know more details about what Github Actions looks like, please visit https://docs.github.com/en/actions.

Names chosen

Different from Github's related sub-system's name, we call the plugins action or actions. One action in fact is a repository with a file named action.yml in the root, the repository is hosted in any git service server which could be gitea.com, your self-hosted Gitea instance, and even github.com. There is a default configuration but we should also support absolute git URL as a plugin repository.

Task summary

There are many things need to do

• Secrets management Secrets storage with SecretKey encrypted #22142
• An event listener and task scheduler Implement actions #21937
• A stable communication protocol between Gitea and runners https://gitea.com/gitea/actions-proto-def
• A runner based on docker https://gitea.com/gitea/act_runner
• A UI that will show tasks and logs Implement actions #21937
• Some core actions are now in https://gitea.com/actions which currently are mirrors but may be changed if there are incompatible between Github API v3 and Gitea API v1
• Support absolute plugin address like uses: https://github.com/peaceiris/actions-hugo@v2 https://gitea.com/gitea/act/pulls/14
• Artifact protocol compatible with Github's, some reports could be displayed in UI directly i.e. Golang standard coverage file Implement actions artifacts #22738
• Service support https://gitea.com/gitea/act/pulls/5
• Cron support chore(actions): support cron schedule task #26655
• Cache action https://gitea.com/gitea/act_runner/pulls/25
• The default plugin programming language is Javascript/Typescript, but we need to support more programming languages
  • Python https://gitea.com/gitea/act_runner/issues/44
  • Golang
  • Rust
  • PHP
  • More
• Check tab in the pull request to list results on the pull request page so that we don't need to go to actions page
• Composite workflow, a workflow file could reference another workflow as a step.

Although we will have an internal CI/CD system Gitea will still be open to integrating external systems and will provide more features to make the integration smooth. The below tasks do not belong to the internal CI/CD system but are very important for integrating the external CI/CD system

• A standard protocol to get tasks and get logs back https://gitea.com/gitea/actions-proto-def

[a1012112796]

tips about api for ci tools to repot detail message:

1. lint messages, can show them like code comment (but can marked as wrong by manager)
   the stype of data that provide by api

{
    commit: xxxxxxx
    path: "xxxxx/xxxxx",
    line: -1,
    type:  bug/ security/ codesmell/ info,
    detail: xxxxxxx,
   provider: xxxxx,
}

example in gitee

2. code coverage data like codecov

{
    commit: xxxxxxx
    path: "xxxxx/xxxxx",
    group: golang,    // only data in same group will be calculate together
   provider: xxxxx,
   start_line: xx,
   start_cols: xx,
   end_line: xx,
  end_cols: xx,
  matched: 0 -- not, .>1 -- yes
}

do some calculate and show some usefull data on ui.

3. detal log in ci steps, should show them in an individual page like gh actions.
   then user can design a sample ci tools which not need it's own ui :)
   style:

{
    commit: xxxxxxx
    step_name: aaa,
    parent_step_name: bbb,
   previous_step_name: ccc,
    log: xxxxx, or link to log data
   status: pending or runing or success ....
}

example on gh

[Be-ing]

Support github actions files and allow external agent to execute the action via Gitea's API and returned result as above results.

Can you clarify what this will entail? Would this mean GitHub Actions could run on a github.com mirror of a repo and the results would appear on a Gitea server?

[lunny]

That means Github actions could be running on Gitea(with this CI feature) servers(actually on user's agent, Gitea will only a coordinator and result saver).
Then mirrors/migrations of repositories from github/github enterprise could run the ci directly and no(or small) modification needed.

[a1012112796]

see

• https://docs.github.com/en/rest/reference/checks
• https://docs.github.com/en/rest/reference/code-scanning

[moqmar]

Would this be implemented by explicitly checking for a .github directory and sending those repos to the agent automatically on push using something like a global webhook? Because I think it would be great to not limit this to just GitHub Actions, but rather have a "global webhook" for pushes to an agent that can then decide what to do with that push - either using the GitHub Actions definitions, or for example running other CI implementations.

[lunny]

@moqmar Global webhooks have been implemented. The issue is to be compatible with actions for those migrations from github.

[ChristopherHX]

I have been working on an external webhook endpoint for github actions self-hosted runners to use with my gitea instance or locally on any pc. This is basically a fork of the runner to be able to reuse their code for a missing aspnetcore server. During my tests I was able to use gitea almost like a ghes instance, some actions like actions/checkout@v2 have had minor incompatibilities with authentication.

I would be glad to see endpoints for CI logs and have an integrated UI. Currently I have a basic react UI to display my logs, added as an external tab to gitea.

Experimental Usage
Download and extract https://github.com/ChristopherHX/runner.server/releases/latest for your system

Setup Gitea as an ghes endpoint, file bin/appsettings.json, change the url from http://localhost:5000 to a different http url / port

  "Runner.Server": {
    "GitServerUrl": "https://gitea:3042",
    "GitApiServerUrl": "https://gitea:3042/api/v1",
    "GitGraphQlServerUrl": null,
    "GITHUB_TOKEN": "Add your personal access token here",
    "ActionDownloadUrls": [
      {
        "TarballUrl": "https://gitea:3042/{0}/archive/{1}.tar.gz",
        "ZipballUrl": "https://gitea:3042/{0}/archive/{1}.zip"
      }
    ]
  }

Add <url of Runner.Server>/runner/server/_apis/v1/Message (http://localhost:5000/runner/server/_apis/v1/Message) as a webhook (select gitea) for your repo, to enable actions.

Setup https and use openid connect with gitea (Optional for Security, only working if the github actions server is using https)

• Create an oauth2 app (User settings)
• You will need to add the clientid and secret into bin/appsettings.json
• You will need a pem certificate pair (cert.pem (only a single certificate will work, no cert chain) , key.pem)

"Kestrel": {
    "Endpoints": {
      "HttpsFromPem": {
        "Url": "https://*:5001",
        "Certificate": {
          "Path": "./cert.pem",
          "KeyPath": "./key.pem"
        }
      }
    }
  },
  "ClientId": "ClientId of your Oauth app",
  "ClientSecret": "Client secret of your Oauth app",
  "Authority": "https://gitea:3042",

Add <url of Runner.Server>/signin-oidc (https://localhost:5001/signin-oidc) as redirect url for the OAuth app in gitea.

Start bin/Runner.Server(.exe)
Open <url of Runner.Server> (http://localhost:5000, https://localhost:5001) to see if a job was received

Configure self-hosted runners, https://github.com/ChristopherHX/runner.server#advanced-usage

EDIT 2021/10/26
I added a ( temporary ) public test instance of runner.server connected to try.gitea.io https://try.gitea.io/ChristopherHX/actions-on-gitea/commit/7ac850a3a96a7c60417c1306f50d49ea08d2e7d7

Press rerun workflow to see in progress job runs
EDIT 2021/02/02
Fix openid sample config to actually have working https with pem

[techknowlogick]

@ChristopherHX whoa! That's amazing!!

[Bobby99as]

This seems nice