release-bot as a GitHub Application

[marusinm]

Implementation of necessary stuff like celery tasks, token generation, Dockerfile, run over multiple projects, etc. Deployment for OpenShift is available here.

WIP:

• track installation_ids for installed repositories and save them into Redis - this is needed for the token generation to make access for actions over HTTP e.g. git push
• remove packit-service leftovers from /files
• delete old s2i scripts and Dockerfiles, edit the makefile
• Update Readme

The reference repo with an installed release-bot app is available here. I added you as collaborators @rpitonak @TomasTomecek @jpopelka you can play with it.

PR is related to #119 #112

[rpitonak]

Nice progress! Great that you make it works on the example repository 🎉

There are few changes requested from my side (some of them you have already mentioned in the TODO list of the PR). Take them more like suggestions as it is still WIP.

[marusinm]

There is one task left to finish this PR. We need to have installation_id for each installed repository to make access for actions over HTTP (e.g. git push). We have two options:

1. After every webhook, release-bot will request for installation_id of the repository which makes bot slower
2. Release-bot can track installation_id into Redis, but when we lost this data all installed repositories will need a reinstall of the Github app.

Can we discuss or vote for one of these options @jpopelka @rpitonak

[rpitonak]

2. Release-bot can track installation_id into Redis, but when we lost this data all installed repositories will need a reinstall of the Github app.

We can make regular back-ups to prevent this.

[rpitonak]

isn't installation ID unique for every org/user? Wouldn't be enough to save {org/username : ID}

[marusinm]

I can't find documentation so I am not sure that this will be enough. Therefore, I saved the installation id for every repo. But I can change it can you send me some link to confirm your assumption? @rpitonak

[jpopelka]

Haven't tested yet, I'll give it a try once it's not WIP

[jpopelka]

Last several notes, sorry to not raising these earlier.

[jpopelka]

See also user-cont/release-bot-deployment#11

[jpopelka]

I've put the suggested changes into marusinm#1

I've deployed it locally and was able to make release-bot work as Github App, but it still doesn't work as I'd expect, because when I, for example, create an issue all I see in the pod/release-bot is:

  |  webhooks.py       INFO   New github webhook call detected
  | 127.0.0.1 - - [09/Oct/2019 13:39:30] "POST /webhook-handler/ HTTP/1.1" 200 -
  | INFO/MainProcess] Received task: task.celery_task.parse_web_hook_payload[ba70ff06-c4cf-4c83-8c1b-e1bc87ba6031]
  | DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f2966af2c20> (args:('task.celery_task.parse_web_hook_payload', 'ba70ff06-c4cf-4c83-8c1b-e1bc87ba6031', {'lang': 'py', 'task': 'task.celery_task.parse_web_hook_payload', 'id': 'ba70ff06-c4cf-4c83-8c1b-e1bc87ba6031', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'ba70ff06-c4cf-4c83-8c1b-e1bc87ba6031', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': "{'webhook_payload': {'action': 'opened', 'issue': {'url': 'https://api.github.com/repos/jpopelka/release-bot-test/issues/37' , 'repository_url': 'https://api.github.com/repos/jpopelka/release-bot-test' , 'labels_url': 'https://api.github.com/repos/jpopelka/release-bot-test/issues/37/labels{/name}' , 'comments_url': 'https://api.github.com/repos/jpopelka/release-bot-test/issues/37/comments' , 'events_url': 'https://api.github.com/repos/jpopelka/release-bot-test/issues/37/events' , 'html_url': 'https://github.com/jpopelka/release-bot-test/issues/37' , 'id': 504658787, 'node_id': 'MDU6SXNzdWU1MDQ2NTg3ODc=', 'number': 37,... kwargs:{})
  | DEBUG/MainProcess] Task accepted: task.celery_task.parse_web_hook_payload[ba70ff06-c4cf-4c83-8c1b-e1bc87ba6031] pid:16
  | INFO   Resolving opened issue
  | INFO/ForkPoolWorker-2] Resolving opened issue
  | DEBUG/ForkPoolWorker-2] Starting new HTTPS connection (1): api.github.com:443

and that's all I get, there's nothing else in the log - which makes me wonder where are the other debug logs I'd expect to see.
The Github App has all the permission it needs.

[marusinm]

Thanks for your changes in marusnm#1 I just merged them.

When I check my old deployment logs with INFO debugging mode I can see this and many more.

10.129.24.137 - - [10/Oct/2019 08:05:31] "POST /webhook-handler/ HTTP/1.1" 200 -
[2019-10-10 08:05:31,075: INFO/MainProcess] Received task: task.celery_task.parse_web_hook_payload[a70867ac-1253-4ca3-993c-3add3b283bd6]  
/usr/lib/python3.7/site-packages/celery/platforms.py:796: RuntimeWarning: You're running the worker with superuser privileges: this is absolutely not recommended!

Please specify a different user using the --uid option.

User information: uid=1019380000 euid=1019380000 gid=0 egid=0

  uid=uid, euid=euid, gid=gid, egid=egid,
08:05:31.554 celery_task.py    INFO   Resolving opened issue
[2019-10-10 08:05:31,554: INFO/ForkPoolWorker-1] Resolving opened issue
08:05:32.432 github.py         ERROR  Failed to fetch setup.cfg
[2019-10-10 08:05:32,432: ERROR/ForkPoolWorker-1] Failed to fetch setup.cfg
08:05:32.433 configuration.py  WARNING pypi_project is not set, falling back to repository_name
[2019-10-10 08:05:32,433: WARNING/ForkPoolWorker-1] pypi_project is not set, falling back to repository_name
08:05:44.663 releasebot.py     INFO   Found new release issue with version: 0.2.66
[2019-10-10 08:05:44,663: INFO/ForkPoolWorker-1] Found new release issue with version: 0.2.66
08:05:55.673 releasebot.py     INFO   Making a new PR for release of version 0.2.66 based on the issue.
[2019-10-10 08:05:55,673: INFO/ForkPoolWorker-1] Making a new PR for release of version 0.2.66 based on the issue.
......

The annoying is that all logs are printed twice because of the celery worker in the same pod. However, I can see them all. When I look at your log, I guess r-bot detected your new issue via webhook. Then it stopped executing. This is caused when the Github App installation token isn't created correctly.

When I find this error for the first time I just had bad workflow which should be the following:

1. Create a Github App with correct webhook URL
2. Deploy image into OpenShift
3. Install Github App onto repository which will send the initial webhook and save installation id into Redis. It means that Github tokens will be generated correctly

At the moment I can't see another issue.

[jpopelka]

The annoying is that all logs are printed twice because of the celery worker in the same pod.

#241

workflow which should be the following 1. Create a Github App with correct webhook URL 2. Deploy image into OpenShift

Ok, that might be the problem because since I run the cluster locally I need to do some magic (oc port-forward & ngrok) after I deploy it, in order for it to be available publicly. And because ngrok changes the public name (i.e. webhook URL) randomly I don't know the webhook URL prior to deploying.

So I have to believe you it works.

[jpopelka]

Ok, I think it's time to merge this.

Unfortunately, I've created a conflict by merging #247

Please rebase so that I can merge.

[jpopelka]

Totally forgot about this. Merging. Thanks for the outstanding work! 🎆