Alternate triggers (TagBot v2)

[christopher-dG]

Closes #62

@tkf had this awesome, delightfully simple idea (#62 (comment)) to use the registry CI to create issue comments on package repos, so that package repos can run TagBot on issue comments only. This is a huge improvement because TagBot will a couple hundred times per day at most instead of something like 50,000 times.

Here's the plan in more detail:

Registry

The General registry will have a new CI step added on merged PRs.
In this step, we parse out the repo, version, and commit from the newly merged PR so we know where to open our issue. Then, we check if TagBot is installed on that repo and if so, we open an issue and create a comment containing the new version and commit SHA (all the info that TagBot needs). If there's an existing issue on the repo that we created for a previous version, we can use that one to avoid creating extra noise in the repos.
This work will go in RegistryCI.jl so that maintainers of all registries can use it. I already have a prototype written.
Note: This requires a user token rather than the one provided by GitHub. Using one that belongs to @JuliaTagBot makes the most sense I think.

TagBot

TagBot's recommended workflow configuration will change to something like this:

name: TagBot
on:
  issue_comment:
    types: created
jobs:
  TagBot:
    runs-on: ubuntu-latest
    steps:
      - uses: JuliaRegistries/TagBot@v2
        if: github.actor == 'JuliaTagBot'

Instead of fetching information from the registry, TagBot will get its info from the issue comment body which contains the version to release and the commit SHA.
For custom registries, users can change the if condition and a new commenter input to specify the user that will be making comments.

---

Am I covering all the bases here? This is still kind of a kludge but it's way better than the current kludge. I don't think that official support for cross-repo events is coming any time soon.

[SaschaMann]

Instead of fetching information from the registry, TagBot will get its info from the issue comment body which contains the version to release and the commit SHA.

Could you explain the reason behind this?

I'm thinking about what would happen if the JuliaTagBot account was compromised, and it seems like the risk would be smaller if it's solely used as a trigger while the information is pulled from the registry. I might be wrong about this, though.

[DilumAluthge]

What are the downsides of the current cron-based approach?

If resource use is the concern, we can recommend that people run the TagBot cron job once daily. Also, keep in mind that the Julia project is not getting charged for downloads of Julia on GitHub Actions.

I guess I don't really see the problem with the cron approach. Whereas this issue-trigger approach now adds a central point of potential security concern, namely a GitHub user account that needs to be trusted.

[DilumAluthge]

Also: It's not uncommon for GitHub action runs to fail sporadically. What happens if a post-merge run fails? The tag never gets made?

[SaschaMann]

Whereas this issue-trigger approach now adds a central point of potential security concern, namely a GitHub user account that needs to be trusted.

AFAIU the account doesn't need any access to the repo (unless it's private but in that case you can't register it to a public registry anyway) because it creates the issue like any normal user would.

[DilumAluthge]

Whereas this issue-trigger approach now adds a central point of potential security concern, namely a GitHub user account that needs to be trusted.

AFAIU the account doesn't need any access to the repo (unless it's private but in that case you can't register it to a public registry anyway) because it creates the issue like any normal user would.

Sure but you are trusting the contents of the issue, which means you are trusting the user.

[tkf]

I guess TagBot can always check the comment against the General repository? In fact, I suppose JuliaTagBot can just comment ping :) and TagBot action can do what currently it does. Of course, if someone get the control of JuliaTagBot, they can DDoS entire Julia ecosystem which still is bad.

[SaschaMann]

Sure but you are trusting the contents of the issue, which means you are trusting the user.

Ok, then we agree. That's what I meant in my previous comment as well : )

Of course, if someone get the control of JuliaTagBot, they can DDoS entire Julia ecosystem which still is bad.

GitHub has rate limits, and everyone could do that by mass-opening PRs and triggering CI with their own account, too.

[christopher-dG]

Sorry to have ignored these responses -- I've been on vacation.

Let's talk about the security concern first: If we use the issue comment as information to TagBot, then the commenter needs to be trusted. Totally fair point. I personally don't see it as a giant risk to add @JuliaTagBot to the JuliaRegistries org (thereby requiring 2FA), but I can understand if you were to still consider it a threat. At the very least, TagBot should verify any information that it gets by looking in the registry, and using it solely as a ping is a good approach too.

Now let's consider what there is to gain. I'd say the two biggest things are:

• Immediate releases like we used to have with the GitHub App. TagBot becomes "properly event-based". Less clogged /actions panels on repos, etc.
• Less computers in the world running TagBot.Yes, we could just tell people to run TagBot less, but even at once a day we're still running an order of magnitude more than we need to. And we get even further from the first benefit.

Dilum's point about actions sometimes failing is a good one, that's really not ideal. I'm not sure how frequently stuff like that happens but it's not to be taken lightly. At least on the TagBot side, TagBot errors could be made more obvious to repo owners so that the odd time that they do appear, the repo owner can just hit retry (although I'd be a bit scared to do this because I'm well-known to ship broken code...). But failures in the registry would need to be handled by us. Adding one more cron action in the registry to list recent runs and retry failed ones wouldn't be too difficult, but it's another moving part to worry about, no matter how small.

I think maybe a good route forward would be to support both cron and issue comment triggers rather than replacing cron. That way:

• everyone using cron continues to get updates as TagBot remains at v1.x
• if people move to issue comments, they can keep an infrequent cron trigger to handle any potential failures

I'll let this sit for some time before digging too deep as I'm sure there's more discussion to be had.

[DilumAluthge]

I think I might have a slightly simpler idea (similar to yours, but I think a little easier to maintain). I’ll write it up in a comment later today.

[DilumAluthge]

Less computers in the world running TagBot.

I think this is the main benefit. By reducing the number of TagBot runs, we can reduce the energy footprint, and therefore reduce our environmental impact.

The General registry will have a new CI step added on merged PRs.

Unfortunately, I think that this is a non-starter. There is a hard cap on how many simultaneous GH Actions jobs we can have running on the General registry at the same time. If we have a CI step running every time a PR is merged, then we will hit that cap more frequently, and then automerge will get backed up. I don't want to back up the automerge queue.

---

Here is my idea. It involves no modifications to the TagBot code, a small modification to the .github/workflows/TagBot.yml workflow file in everyone's repositories, and the addition of a new GitHub Action to the General registry.

We add a new GitHub Action that runs as a cron job on the General registry. It runs e.g. every 20 minutes. (I don't want to run it too frequently, because we risk hitting our cap on parallel jobs, and thus delaying automerge.) This new cron job will do the following:

1. Compile a list of all new releases made in the last 2 weeks. (This is simple: first, make a list of all versions of all packages based on the latest master. Then, check out a commit from approximately 2 weeks ago, and make a list of all versions of all packages based on that commit. Then take the set difference.)
2. For each package that has at least one new release made in the last 2 weeks, check if it is a GitHub repository. If it is, clone it and get a list of all of the tags. If the list of new releases is a (not necessarily proper) subset of the list of all tags, then do nothing. However, if there is a new release that does not have a tag, then we need to ping the repository. We use the GitHub API to get a list of all issues on the repository. If there is an open issue by the JuliaTagBot user, we close it. If there is a closed issue by the JuliaTagBot user, we reopen it and then we close it. If there are no open or closed issues by the JuliaTagBot user, we create a new issue, and then we close it. This serves as a trigger to run the TagBot job.

The recommended .github/workflows/TagBot.yml becomes:

name: TagBot
on:
  schedule:
    - cron: 0 0 * * 0 # Notice that I have made the TagBot cron job run once a week. This is a fallback, in case the issue-trigged run fails for any reason.
  issues: # I have added this line
    types: [opened, closed, reopened] # I have added this line

jobs:
  TagBot:
    runs-on: ubuntu-latest
    steps:
      - uses: JuliaRegistries/TagBot@v1
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
         if: github.actor == 'JuliaTagBot' || github.event_name == 'schedule' # I have added this line

---

I intentionally chose issues and not issue comments. The opening, reopening, or closing of an issue is a relatively uncommon event. So GitHub doesn't need to do this logic very frequently. In contrast, issue comments are quite frequent. So GitHub would have to do some work every time an issue comment was created. We don't want that.

[DilumAluthge]

I'm almost finished with the code that implements this. I'll share it soon.

[DilumAluthge]

Also, this cron job doesn't need to be run on the General registry. It can be run everywhere.

So it probably will make the most sense to run the cron job on this repo (the TagBot repo). Then you can give it a PAT for JuliaTagBot. And you can run it every 5 minutes, which is the fastest that GitHub will allow. So then package authors will see their tags getting created approximately 5 minutes after the PR is merged into General, which is very close to instant.

[christopher-dG]

Cool idea. What about using edited triggers to avoid user notifications?

[christopher-dG]

Also where's the number of allowed parallel jobs? I found this but it only lists the non-open-source rates.
Even at the free rate, I don't really see us hitting that.

edit: I ran a test on a public repo and it is running 40 workflows at once. This makes sense since I'm on the "Pro" plan (they haven't checked my student status in quite some time...), so it would make sense for JuliaRegistries to be on the "Free" plan and get 20 concurrent runs.

[christopher-dG]

Oh one more thing I thought of: If this registry-wide polling for new versions happens outside of the registry, support for registries that aren't General breaks somewhat. Or at least, the steps to hosting custom registries start to diverge slightly from General.

Also it seems like there's a lot of work being duplicated, i.e. what was described to look up new versions of packages is already exactly what TagBot does. I think I'm still in the "use PR events" camp; I don't see us hitting our concurrency limit except in exceedingly rare circumstances, the work to be done to support the merged-PR strategy is less complex than polling (in my opinion, at least).