Consumer: failing jobs immediately re-scheduled forever

[bzz]

As a developer running Borges consumer, I would like to prevent infinite loop of rescheduling for failing jobs.

Right now, if input queue only has 2 jobs, both of which are not processable - consumer will immediately reschedule them forever, keeping CPU buys.

It is also not possible to just remove those from the queue - it seems that they got put back to the queue right after a failing attempt to process them, forever.

How to reproduce:

echo 'git://github.com:damoeb/kalipo.git
git://github.com/CyanogenMod/android_hardware_qcom_fm.git' > 2.txt

borges producer --file 2.txt
borges consumer --workers=2

Full log below

DBUG[06-29|22:45:57] job started                              module=borges WorkerID=1 RepositoryID=015cf599-df58-597a-3513-5c496ef77f94 caller=consumer.go:44
DBUG[06-29|22:45:57] job started                              module=borges WorkerID=0 RepositoryID=015cf599-df42-721c-ecdc-bb521f41eb96 caller=consumer.go:44
DBUG[06-29|22:45:57] repository model obtained                job=015cf599-df42-721c-ecdc-bb521f41eb96 status=pending last-fetch=nil references=0 caller=archiver.go:93
DBUG[06-29|22:45:57] endpoint selected                        job=015cf599-df42-721c-ecdc-bb521f41eb96 endpoint=git://github.com:damoeb/kalipo.git caller=archiver.go:100
DBUG[06-29|22:45:57] local temporary directory created        job=015cf599-df42-721c-ecdc-bb521f41eb96 temp-path=local_repos/015cf599-df42-721c-ecdc-bb521f41eb96/1498769157287452998 caller=archiver.go:109
DBUG[06-29|22:45:57] local repository created                 job=015cf599-df42-721c-ecdc-bb521f41eb96 caller=archiver.go:116
DBUG[06-29|22:45:57] repository model obtained                job=015cf599-df58-597a-3513-5c496ef77f94 status=pending last-fetch=nil references=0 caller=archiver.go:93
DBUG[06-29|22:45:57] endpoint selected                        job=015cf599-df58-597a-3513-5c496ef77f94 endpoint=git://github.com/CyanogenMod/android_hardware_qcom_fm.git caller=archiver.go:100
DBUG[06-29|22:45:57] local temporary directory created        job=015cf599-df58-597a-3513-5c496ef77f94 temp-path=local_repos/015cf599-df58-597a-3513-5c496ef77f94/1498769157312643440 caller=archiver.go:109
DBUG[06-29|22:45:57] local repository created                 job=015cf599-df58-597a-3513-5c496ef77f94 caller=archiver.go:116
EROR[06-29|22:45:57] error fetching repository                job=015cf599-df42-721c-ecdc-bb521f41eb96 error="repository not found" caller=archiver.go:134
EROR[06-29|22:45:57] job errored                              module=borges WorkerID=0 RepositoryID=015cf599-df42-721c-ecdc-bb521f41eb96 error="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=consumer.go:49
EROR[06-29|22:45:57] error on job                             module=worker id=0 err="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=worker.go:46
DBUG[06-29|22:45:57] job started                              module=borges WorkerID=0 RepositoryID=015cf599-df42-721c-ecdc-bb521f41eb96 caller=consumer.go:44
DBUG[06-29|22:45:57] repository model obtained                job=015cf599-df42-721c-ecdc-bb521f41eb96 status=pending last-fetch=nil references=0 caller=archiver.go:93
DBUG[06-29|22:45:57] endpoint selected                        job=015cf599-df42-721c-ecdc-bb521f41eb96 endpoint=git://github.com:damoeb/kalipo.git caller=archiver.go:100
DBUG[06-29|22:45:57] local temporary directory created        job=015cf599-df42-721c-ecdc-bb521f41eb96 temp-path=local_repos/015cf599-df42-721c-ecdc-bb521f41eb96/1498769157592851853 caller=archiver.go:109
DBUG[06-29|22:45:57] local repository created                 job=015cf599-df42-721c-ecdc-bb521f41eb96 caller=archiver.go:116
EROR[06-29|22:45:57] error fetching repository                job=015cf599-df42-721c-ecdc-bb521f41eb96 error="repository not found" caller=archiver.go:134
EROR[06-29|22:45:57] job errored                              module=borges WorkerID=0 RepositoryID=015cf599-df42-721c-ecdc-bb521f41eb96 error="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=consumer.go:49
EROR[06-29|22:45:57] error on job                             module=worker id=0 err="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=worker.go:46
DBUG[06-29|22:45:57] job started                              module=borges WorkerID=0 RepositoryID=015cf599-df42-721c-ecdc-bb521f41eb96 caller=consumer.go:44
DBUG[06-29|22:45:57] repository model obtained                job=015cf599-df42-721c-ecdc-bb521f41eb96 status=pending last-fetch=nil references=0 caller=archiver.go:93
DBUG[06-29|22:45:57] endpoint selected                        job=015cf599-df42-721c-ecdc-bb521f41eb96 endpoint=git://github.com:damoeb/kalipo.git caller=archiver.go:100
DBUG[06-29|22:45:57] local temporary directory created        job=015cf599-df42-721c-ecdc-bb521f41eb96 temp-path=local_repos/015cf599-df42-721c-ecdc-bb521f41eb96/1498769157853411811 caller=archiver.go:109
DBUG[06-29|22:45:57] local repository created                 job=015cf599-df42-721c-ecdc-bb521f41eb96 caller=archiver.go:116
EROR[06-29|22:45:58] error fetching repository                job=015cf599-df42-721c-ecdc-bb521f41eb96 error="repository not found" caller=archiver.go:134
EROR[06-29|22:45:58] job errored                              module=borges WorkerID=0 RepositoryID=015cf599-df42-721c-ecdc-bb521f41eb96 error="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=consumer.go:49
EROR[06-29|22:45:58] error on job                             module=worker id=0 err="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=worker.go:46
DBUG[06-29|22:45:58] job started                              module=borges WorkerID=0 RepositoryID=015cf599-df42-721c-ecdc-bb521f41eb96 caller=consumer.go:44
DBUG[06-29|22:45:58] repository model obtained                job=015cf599-df42-721c-ecdc-bb521f41eb96 status=pending last-fetch=nil references=0 caller=archiver.go:93
DBUG[06-29|22:45:58] endpoint selected                        job=015cf599-df42-721c-ecdc-bb521f41eb96 endpoint=git://github.com:damoeb/kalipo.git caller=archiver.go:100
DBUG[06-29|22:45:58] local temporary directory created        job=015cf599-df42-721c-ecdc-bb521f41eb96 temp-path=local_repos/015cf599-df42-721c-ecdc-bb521f41eb96/1498769158118079322 caller=archiver.go:109
DBUG[06-29|22:45:58] local repository created                 job=015cf599-df42-721c-ecdc-bb521f41eb96 caller=archiver.go:116
EROR[06-29|22:45:58] error fetching repository                job=015cf599-df42-721c-ecdc-bb521f41eb96 error="repository not found" caller=archiver.go:134
EROR[06-29|22:45:58] job errored                              module=borges WorkerID=0 RepositoryID=015cf599-df42-721c-ecdc-bb521f41eb96 error="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=consumer.go:49
EROR[06-29|22:45:58] error on job                             module=worker id=0 err="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=worker.go:46
DBUG[06-29|22:45:58] job started                              module=borges WorkerID=0 RepositoryID=015cf599-df42-721c-ecdc-bb521f41eb96 caller=consumer.go:44
DBUG[06-29|22:45:58] repository model obtained                job=015cf599-df42-721c-ecdc-bb521f41eb96 status=pending last-fetch=nil references=0 caller=archiver.go:93
DBUG[06-29|22:45:58] endpoint selected                        job=015cf599-df42-721c-ecdc-bb521f41eb96 endpoint=git://github.com:damoeb/kalipo.git caller=archiver.go:100
DBUG[06-29|22:45:58] local temporary directory created        job=015cf599-df42-721c-ecdc-bb521f41eb96 temp-path=local_repos/015cf599-df42-721c-ecdc-bb521f41eb96/1498769158405943242 caller=archiver.go:109
DBUG[06-29|22:45:58] local repository created                 job=015cf599-df42-721c-ecdc-bb521f41eb96 caller=archiver.go:116
EROR[06-29|22:45:58] error fetching repository                job=015cf599-df42-721c-ecdc-bb521f41eb96 error="repository not found" caller=archiver.go:134
EROR[06-29|22:45:58] job errored                              module=borges WorkerID=0 RepositoryID=015cf599-df42-721c-ecdc-bb521f41eb96 error="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=consumer.go:49
EROR[06-29|22:45:58] error on job                             module=worker id=0 err="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=worker.go:46
DBUG[06-29|22:45:58] job started                              module=borges WorkerID=0 RepositoryID=015cf599-df42-721c-ecdc-bb521f41eb96 caller=consumer.go:44
DBUG[06-29|22:45:58] repository model obtained                job=015cf599-df42-721c-ecdc-bb521f41eb96 status=pending last-fetch=nil references=0 caller=archiver.go:93
DBUG[06-29|22:45:58] endpoint selected                        job=015cf599-df42-721c-ecdc-bb521f41eb96 endpoint=git://github.com:damoeb/kalipo.git caller=archiver.go:100
DBUG[06-29|22:45:58] local temporary directory created        job=015cf599-df42-721c-ecdc-bb521f41eb96 temp-path=local_repos/015cf599-df42-721c-ecdc-bb521f41eb96/1498769158653526154 caller=archiver.go:109
DBUG[06-29|22:45:58] local repository created                 job=015cf599-df42-721c-ecdc-bb521f41eb96 caller=archiver.go:116
EROR[06-29|22:45:58] error fetching repository                job=015cf599-df42-721c-ecdc-bb521f41eb96 error="repository not found" caller=archiver.go:134
EROR[06-29|22:45:58] job errored                              module=borges WorkerID=0 RepositoryID=015cf599-df42-721c-ecdc-bb521f41eb96 error="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=consumer.go:49
EROR[06-29|22:45:58] error on job                             module=worker id=0 err="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=worker.go:46
DBUG[06-29|22:45:58] job started                              module=borges WorkerID=0 RepositoryID=015cf599-df42-721c-ecdc-bb521f41eb96 caller=consumer.go:44
DBUG[06-29|22:45:58] repository model obtained                job=015cf599-df42-721c-ecdc-bb521f41eb96 status=pending last-fetch=nil references=0 caller=archiver.go:93
DBUG[06-29|22:45:58] endpoint selected                        job=015cf599-df42-721c-ecdc-bb521f41eb96 endpoint=git://github.com:damoeb/kalipo.git caller=archiver.go:100
DBUG[06-29|22:45:58] local temporary directory created        job=015cf599-df42-721c-ecdc-bb521f41eb96 temp-path=local_repos/015cf599-df42-721c-ecdc-bb521f41eb96/1498769158971635041 caller=archiver.go:109
DBUG[06-29|22:45:58] local repository created                 job=015cf599-df42-721c-ecdc-bb521f41eb96 caller=archiver.go:116
EROR[06-29|22:45:59] error fetching repository                job=015cf599-df42-721c-ecdc-bb521f41eb96 error="repository not found" caller=archiver.go:134
EROR[06-29|22:45:59] job errored                              module=borges WorkerID=0 RepositoryID=015cf599-df42-721c-ecdc-bb521f41eb96 error="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=consumer.go:49
EROR[06-29|22:45:59] error on job                             module=worker id=0 err="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=worker.go:46
DBUG[06-29|22:45:59] job started                              module=borges WorkerID=0 RepositoryID=015cf599-df42-721c-ecdc-bb521f41eb96 caller=consumer.go:44
DBUG[06-29|22:45:59] repository model obtained                job=015cf599-df42-721c-ecdc-bb521f41eb96 status=pending last-fetch=nil references=0 caller=archiver.go:93
DBUG[06-29|22:45:59] endpoint selected                        job=015cf599-df42-721c-ecdc-bb521f41eb96 endpoint=git://github.com:damoeb/kalipo.git caller=archiver.go:100
DBUG[06-29|22:45:59] local temporary directory created        job=015cf599-df42-721c-ecdc-bb521f41eb96 temp-path=local_repos/015cf599-df42-721c-ecdc-bb521f41eb96/1498769159215042159 caller=archiver.go:109
DBUG[06-29|22:45:59] local repository created                 job=015cf599-df42-721c-ecdc-bb521f41eb96 caller=archiver.go:116
EROR[06-29|22:45:59] error fetching repository                job=015cf599-df42-721c-ecdc-bb521f41eb96 error="repository not found" caller=archiver.go:134
EROR[06-29|22:45:59] job errored                              module=borges WorkerID=0 RepositoryID=015cf599-df42-721c-ecdc-bb521f41eb96 error="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=consumer.go:49
EROR[06-29|22:45:59] error on job                             module=worker id=0 err="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=worker.go:46
DBUG[06-29|22:45:59] job started                              module=borges WorkerID=0 RepositoryID=015cf599-df42-721c-ecdc-bb521f41eb96 caller=consumer.go:44
DBUG[06-29|22:45:59] repository model obtained                job=015cf599-df42-721c-ecdc-bb521f41eb96 status=pending last-fetch=nil references=0 caller=archiver.go:93
DBUG[06-29|22:45:59] endpoint selected                        job=015cf599-df42-721c-ecdc-bb521f41eb96 endpoint=git://github.com:damoeb/kalipo.git caller=archiver.go:100
DBUG[06-29|22:45:59] local temporary directory created        job=015cf599-df42-721c-ecdc-bb521f41eb96 temp-path=local_repos/015cf599-df42-721c-ecdc-bb521f41eb96/1498769159463250685 caller=archiver.go:109
DBUG[06-29|22:45:59] local repository created                 job=015cf599-df42-721c-ecdc-bb521f41eb96 caller=archiver.go:116
EROR[06-29|22:45:59] error fetching repository                job=015cf599-df42-721c-ecdc-bb521f41eb96 error="repository not found" caller=archiver.go:134
EROR[06-29|22:45:59] job errored                              module=borges WorkerID=0 RepositoryID=015cf599-df42-721c-ecdc-bb521f41eb96 error="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=consumer.go:49
EROR[06-29|22:45:59] error on job                             module=worker id=0 err="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=worker.go:46
DBUG[06-29|22:45:59] job started                              module=borges WorkerID=0 RepositoryID=015cf599-df42-721c-ecdc-bb521f41eb96 caller=consumer.go:44
DBUG[06-29|22:45:59] repository model obtained                job=015cf599-df42-721c-ecdc-bb521f41eb96 status=pending last-fetch=nil references=0 caller=archiver.go:93
DBUG[06-29|22:45:59] endpoint selected                        job=015cf599-df42-721c-ecdc-bb521f41eb96 endpoint=git://github.com:damoeb/kalipo.git caller=archiver.go:100
DBUG[06-29|22:45:59] local temporary directory created        job=015cf599-df42-721c-ecdc-bb521f41eb96 temp-path=local_repos/015cf599-df42-721c-ecdc-bb521f41eb96/1498769159708196999 caller=archiver.go:109
DBUG[06-29|22:45:59] local repository created                 job=015cf599-df42-721c-ecdc-bb521f41eb96 caller=archiver.go:116
EROR[06-29|22:45:59] error fetching repository                job=015cf599-df42-721c-ecdc-bb521f41eb96 error="repository not found" caller=archiver.go:134
EROR[06-29|22:45:59] job errored                              module=borges WorkerID=0 RepositoryID=015cf599-df42-721c-ecdc-bb521f41eb96 error="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=consumer.go:49
EROR[06-29|22:45:59] error on job                             module=worker id=0 err="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=worker.go:46
DBUG[06-29|22:45:59] job started                              module=borges WorkerID=0 RepositoryID=015cf599-df42-721c-ecdc-bb521f41eb96 caller=consumer.go:44
DBUG[06-29|22:45:59] repository model obtained                job=015cf599-df42-721c-ecdc-bb521f41eb96 status=pending last-fetch=nil references=0 caller=archiver.go:93
DBUG[06-29|22:45:59] endpoint selected                        job=015cf599-df42-721c-ecdc-bb521f41eb96 endpoint=git://github.com:damoeb/kalipo.git caller=archiver.go:100
DBUG[06-29|22:45:59] local temporary directory created        job=015cf599-df42-721c-ecdc-bb521f41eb96 temp-path=local_repos/015cf599-df42-721c-ecdc-bb521f41eb96/1498769159968231309 caller=archiver.go:109
DBUG[06-29|22:45:59] local repository created                 job=015cf599-df42-721c-ecdc-bb521f41eb96 caller=archiver.go:116
EROR[06-29|22:46:00] error fetching repository                job=015cf599-df42-721c-ecdc-bb521f41eb96 error="repository not found" caller=archiver.go:134
EROR[06-29|22:46:00] job errored                              module=borges WorkerID=0 RepositoryID=015cf599-df42-721c-ecdc-bb521f41eb96 error="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=consumer.go:49
EROR[06-29|22:46:00] error on job                             module=worker id=0 err="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=worker.go:46
DBUG[06-29|22:46:00] job started                              module=borges WorkerID=0 RepositoryID=015cf599-df42-721c-ecdc-bb521f41eb96 caller=consumer.go:44
DBUG[06-29|22:46:00] repository model obtained                job=015cf599-df42-721c-ecdc-bb521f41eb96 status=pending last-fetch=nil references=0 caller=archiver.go:93
DBUG[06-29|22:46:00] endpoint selected                        job=015cf599-df42-721c-ecdc-bb521f41eb96 endpoint=git://github.com:damoeb/kalipo.git caller=archiver.go:100
DBUG[06-29|22:46:00] local temporary directory created        job=015cf599-df42-721c-ecdc-bb521f41eb96 temp-path=local_repos/015cf599-df42-721c-ecdc-bb521f41eb96/1498769160228889781 caller=archiver.go:109
DBUG[06-29|22:46:00] local repository created                 job=015cf599-df42-721c-ecdc-bb521f41eb96 caller=archiver.go:116
EROR[06-29|22:46:00] error fetching repository                job=015cf599-df42-721c-ecdc-bb521f41eb96 error="repository not found" caller=archiver.go:134
EROR[06-29|22:46:00] job errored                              module=borges WorkerID=0 RepositoryID=015cf599-df42-721c-ecdc-bb521f41eb96 error="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=consumer.go:49
EROR[06-29|22:46:00] error on job                             module=worker id=0 err="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=worker.go:46
DBUG[06-29|22:46:00] job started                              module=borges WorkerID=0 RepositoryID=015cf599-df42-721c-ecdc-bb521f41eb96 caller=consumer.go:44
DBUG[06-29|22:46:00] repository model obtained                job=015cf599-df42-721c-ecdc-bb521f41eb96 status=pending last-fetch=nil references=0 caller=archiver.go:93
DBUG[06-29|22:46:00] endpoint selected                        job=015cf599-df42-721c-ecdc-bb521f41eb96 endpoint=git://github.com:damoeb/kalipo.git caller=archiver.go:100
DBUG[06-29|22:46:00] local temporary directory created        job=015cf599-df42-721c-ecdc-bb521f41eb96 temp-path=local_repos/015cf599-df42-721c-ecdc-bb521f41eb96/1498769160483977602 caller=archiver.go:109
DBUG[06-29|22:46:00] local repository created                 job=015cf599-df42-721c-ecdc-bb521f41eb96 caller=archiver.go:116
EROR[06-29|22:46:00] error fetching repository                job=015cf599-df42-721c-ecdc-bb521f41eb96 error="repository not found" caller=archiver.go:134
EROR[06-29|22:46:00] job errored                              module=borges WorkerID=0 RepositoryID=015cf599-df42-721c-ecdc-bb521f41eb96 error="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=consumer.go:49
EROR[06-29|22:46:00] error on job                             module=worker id=0 err="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=worker.go:46
DBUG[06-29|22:46:00] job started                              module=borges WorkerID=0 RepositoryID=015cf599-df42-721c-ecdc-bb521f41eb96 caller=consumer.go:44
DBUG[06-29|22:46:00] repository model obtained                job=015cf599-df42-721c-ecdc-bb521f41eb96 status=pending last-fetch=nil references=0 caller=archiver.go:93
DBUG[06-29|22:46:00] endpoint selected                        job=015cf599-df42-721c-ecdc-bb521f41eb96 endpoint=git://github.com:damoeb/kalipo.git caller=archiver.go:100
DBUG[06-29|22:46:00] local temporary directory created        job=015cf599-df42-721c-ecdc-bb521f41eb96 temp-path=local_repos/015cf599-df42-721c-ecdc-bb521f41eb96/1498769160823397140 caller=archiver.go:109
DBUG[06-29|22:46:00] local repository created                 job=015cf599-df42-721c-ecdc-bb521f41eb96 caller=archiver.go:116
EROR[06-29|22:46:01] error fetching repository                job=015cf599-df42-721c-ecdc-bb521f41eb96 error="repository not found" caller=archiver.go:134
EROR[06-29|22:46:01] job errored                              module=borges WorkerID=0 RepositoryID=015cf599-df42-721c-ecdc-bb521f41eb96 error="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=consumer.go:49
EROR[06-29|22:46:01] error on job                             module=worker id=0 err="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=worker.go:46
DBUG[06-29|22:46:01] job started                              module=borges WorkerID=0 RepositoryID=015cf599-df42-721c-ecdc-bb521f41eb96 caller=consumer.go:44
DBUG[06-29|22:46:01] repository model obtained                job=015cf599-df42-721c-ecdc-bb521f41eb96 status=pending last-fetch=nil references=0 caller=archiver.go:93
DBUG[06-29|22:46:01] endpoint selected                        job=015cf599-df42-721c-ecdc-bb521f41eb96 endpoint=git://github.com:damoeb/kalipo.git caller=archiver.go:100
DBUG[06-29|22:46:01] local temporary directory created        job=015cf599-df42-721c-ecdc-bb521f41eb96 temp-path=local_repos/015cf599-df42-721c-ecdc-bb521f41eb96/1498769161145569590 caller=archiver.go:109
DBUG[06-29|22:46:01] local repository created                 job=015cf599-df42-721c-ecdc-bb521f41eb96 caller=archiver.go:116
DBUG[06-29|22:46:01] changes obtained                         job=015cf599-df58-597a-3513-5c496ef77f94 roots=1 caller=archiver.go:143
EROR[06-29|22:46:01] job errored                              module=borges WorkerID=1 RepositoryID=015cf599-df58-597a-3513-5c496ef77f94 error="reference not found" caller=consumer.go:49
EROR[06-29|22:46:01] error on job                             module=worker id=1 err="reference not found" caller=worker.go:46

Actual result:

• 2 jobs fail and get re-scheduled forever in infinite loop, Workers using CPU
• can not remove them from Rabbit queue
• can not get empty queue

Expected result:

• 2 jobs fail (1st due to malformed URL, 2nd due to Consumer: can not process the repository error="reference not found" #72 ) and workers become idle (0%CPU)
• go to http://localhost:8081 and take 2 items from the queue (\wo requeue them)
• get empty queue

[ajnavarro]

@bzz Did you update all the dependencies using glide up ? I'm not able to reproduce this bug:

DBUG[06-30|10:16:17] starting                                 module=worker id=0 caller=worker.go:33
DBUG[06-30|10:16:17] job started                              module=borges WorkerID=0 RepositoryID=015cf812-4cf9-bd7b-4572-2eb657ca9093 caller=consumer.go:46
DBUG[06-30|10:16:17] repository model obtained                job=015cf812-4cf9-bd7b-4572-2eb657ca9093 status=pending last-fetch=nil references=0 caller=archiver.go:93
DBUG[06-30|10:16:17] endpoint selected                        job=015cf812-4cf9-bd7b-4572-2eb657ca9093 endpoint=git://github.com:damoeb/kalipo.git caller=archiver.go:100
DBUG[06-30|10:16:17] local temporary directory created        job=015cf812-4cf9-bd7b-4572-2eb657ca9093 temp-path=local_repos/015cf812-4cf9-bd7b-4572-2eb657ca9093/1498810577500195369 caller=archiver.go:109
DBUG[06-30|10:16:17] local repository created                 job=015cf812-4cf9-bd7b-4572-2eb657ca9093 caller=archiver.go:116
EROR[06-30|10:16:17] error fetching repository                job=015cf812-4cf9-bd7b-4572-2eb657ca9093 error="repository not found" caller=archiver.go:134
EROR[06-30|10:16:17] job errored                              module=borges WorkerID=0 RepositoryID=015cf812-4cf9-bd7b-4572-2eb657ca9093 error="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=consumer.go:51
EROR[06-30|10:16:17] error on job                             module=worker id=0 err="fetching git://github.com:damoeb/kalipo.git failed: repository not found" caller=worker.go:46
DBUG[06-30|10:16:17] job started                              module=borges WorkerID=0 RepositoryID=015cf812-4cfc-bb6f-41c8-183bf36d6ea1 caller=consumer.go:46
DBUG[06-30|10:16:17] repository model obtained                job=015cf812-4cfc-bb6f-41c8-183bf36d6ea1 status=pending last-fetch=nil references=0 caller=archiver.go:93
DBUG[06-30|10:16:17] endpoint selected                        job=015cf812-4cfc-bb6f-41c8-183bf36d6ea1 endpoint=git://github.com/CyanogenMod/android_hardware_qcom_fm.git caller=archiver.go:100
DBUG[06-30|10:16:17] local temporary directory created        job=015cf812-4cfc-bb6f-41c8-183bf36d6ea1 temp-path=local_repos/015cf812-4cfc-bb6f-41c8-183bf36d6ea1/1498810577790912974 caller=archiver.go:109
DBUG[06-30|10:16:17] local repository created                 job=015cf812-4cfc-bb6f-41c8-183bf36d6ea1 caller=archiver.go:116
DBUG[06-30|10:16:22] changes obtained                         job=015cf812-4cfc-bb6f-41c8-183bf36d6ea1 roots=1 caller=archiver.go:143
EROR[06-30|10:16:22] job errored                              module=borges WorkerID=0 RepositoryID=015cf812-4cfc-bb6f-41c8-183bf36d6ea1 error="reference not found" caller=consumer.go:51
EROR[06-30|10:16:22] error on job                             module=worker id=0 err="reference not found" caller=worker.go:46

[NO MORE LOGS HERE, WAITING FOR KEEP MORE JOBS FROM THE QUEUE]

Both jobs are now into the dead-letter queue.

Maybe you added both repos a lot of times into the queue?

[bzz]

Thank you for prompt responce!

I'm checking again now, on a clean environment \wo any Go installed.

glide install
make packages
./bin/<arch>/borges

[ajnavarro]

@bzz could you reproduce the error again?

[ajnavarro]

@bzz ping

[bzz]

Sorry for confusion, indeed this was due to my local env build \wo glide and all the latest fixes :/

This does not happen any more.