Finished build check not set - workers race condition

[dhodovsk]

We've recorded some events when the build end has not been set in PR checks.

The reason for it now is likely the way workers process jobs. When one worker processes copr_build_finished sooner than the other worker processes copr_build_started, then copr_build_started will be the final result.

Example:

• Add support for automatic patch/source numbering rebase-helper/rebase-helper#768
• check status history:
  https://api.github.com/repos/rebase-helper/rebase-helper/statuses/bc3cc6688b3aada55b589e61b5543ddb2d8df753
  • RPM build has started..." - "created_at": "2020-01-28T13:35:20Z",
  • RPMs were built successfully. - "created_at": "2020-01-28T13:33:53Z",

And taken from the logs:

• worker 2 - start

[2020-01-28 13:35:21,062: INFO/ForkPoolWorker-1] Task task.steve_jobs.process_message[8fa39cb2-e6d0-4e1f-9b74-9321307a7ec8] succeeded in 9.840091235935688s: {'jobs': {'copr_build_started': {'success': True, 'details': {...}}}, 'event': {'trigger': 'commit', 'created_at': 1580218511, 'project_url': 'https://github.com/rebase-helper/rebase-helper', 'topic': 'org.fedoraproject.prod.copr.build.start', 'build_id': 1205102, 'build': {'commit_sha': 'bc3cc6688b3aada55b589e61b5543ddb2d8df753', 'pr_id': 768, 'repo_name': 'rebase-helper', 'repo_namespace': 'rebase-helper', 'ref': 'bc3cc6688b3aada55b589e61b5543ddb2d8df753', 'https_url': 'https://github.com/rebase-helper/rebase-helper'}, 'chroot': 'fedora-31-x86_64', 'status': 3, 'owner': 'packit', 'project_name': 'rebase-helper-rebase-helper-768', 'pkg': 'rebase-helper', 'base_repo_name': 'rebase-helper', 'base_repo_namespace': 'rebase-helper', 'pr_id': 768, 'ref': 'bc3cc6688b3aada55b589e61b5543ddb2d8df753', 'commit_sha': 'bc3cc6688b3aada55b589e61b5543ddb2d8df753'}}

• worker 1 - finish

[2020-01-28 13:33:54,117: INFO/ForkPoolWorker-1] Task task.steve_jobs.process_message[ab59c00f-c8c5-4084-87d7-d6cd3bfc3462] succeeded in 10.50708981975913s: {'jobs': {'copr_build_finished': {'success': True, 'details': {...}}}, 'event': {'trigger': 'commit', 'created_at': 1580218423, 'project_url': 'https://github.com/rebase-helper/rebase-helper', 'topic': 'org.fedoraproject.prod.copr.build.end', 'build_id': 1205102, 'build': {'commit_sha': 'bc3cc6688b3aada55b589e61b5543ddb2d8df753', 'pr_id': 768, 'repo_name': 'rebase-helper', 'repo_namespace': 'rebase-helper', 'ref': 'bc3cc6688b3aada55b589e61b5543ddb2d8df753', 'https_url': 'https://github.com/rebase-helper/rebase-helper'}, 'chroot': 'fedora-31-x86_64', 'status': 1, 'owner': 'packit', 'project_name': 'rebase-helper-rebase-helper-768', 'pkg': 'rebase-helper', 'base_repo_name': 'rebase-helper', 'base_repo_namespace': 'rebase-helper', 'pr_id': 768, 'ref': 'bc3cc6688b3aada55b589e61b5543ddb2d8df753', 'commit_sha': 'bc3cc6688b3aada55b589e61b5543ddb2d8df753'}}

[dhodovsk]

Possible solution: verify the build status by API call to copr before setting RPM build start.