Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Autopush seems to be broken #950

Closed
bowlofeggs opened this issue Sep 21, 2016 · 5 comments · Fixed by #976

Comments

@bowlofeggs
Copy link
Member

commented Sep 21, 2016

Usually when an update reaches the stable karma threshold, there will be a comment from the bodhi user stating that the update has reached this status and that it will be pushed to stable. For example, this update from bodhi-2.0-212 exibited that behavior:

https://bodhi.fedoraproject.org/updates/FEDORA-2016-ffc86f2b30

Since deploying 2.2.0 to production, it has been noted that updates no longer seem to get that comment from Bodhi when they reach stable karma. For example, see this update in stg:

https://bodhi.stg.fedoraproject.org/updates/FEDORA-2016-4d0198119a

My comment should have caused bodhi to submit it for stable automatically, but it did not. Here is one more example of a non-critical path update that should be autopushed:

https://bodhi.stg.fedoraproject.org/updates/FEDORA-2016-93c58fa1b3

@bowlofeggs bowlofeggs added the Critical label Sep 21, 2016

@trishnaguha

This comment has been minimized.

Copy link
Contributor

commented Sep 21, 2016

@bowlofeggs Not sure why. We have test for this.

@trishnaguha

This comment has been minimized.

Copy link
Contributor

commented Sep 21, 2016

What about this one? https://bodhi.fedoraproject.org/updates/quantum-espresso-5.4.0-10.fc25
This seems to be autopushed.

@bowlofeggs

This comment has been minimized.

Copy link
Member Author

commented Sep 22, 2016

@trishnaguha I think that update may have happened before we finished updating production to 2.2.0. The outage started at 19:00:00 UTC that day, and that update comment came it at ~19:25, but it took us longer than that to get everything done. I don't know the exact time we started/finished, but I could probably find out in the logs if we want to know for sure. I could be wrong.

Furthermore, there is some suspicion that the way we resumed a failed push might have put some strange data into the database that might be causing this issue for some updates. It's possible also that on some updates, the client submitted a form with the old version of bodhi running in the browser that was in some way incompatible with the new server. We really don't know what is causing this, but there are definitely updates in production that have some issue here. I was even able to reproduce the problem in stg by adding a comment that should have triggered the stability.

@bowlofeggs

This comment has been minimized.

Copy link
Member Author

commented Sep 23, 2016

This may be related to #949 in some way, but that's just a guess.

@bowlofeggs bowlofeggs self-assigned this Sep 24, 2016

@bowlofeggs

This comment has been minimized.

Copy link
Member Author

commented Sep 24, 2016

@puiterwijk has come to the rescue again! It turns out that there was a commit a while ago that added an autokarma column to the Update model. A later commit added a schema migration, but the schema migration did not include a data migration to fill out this column for existing commits. When the code that detected autokarma was changed to use this column, all existing Updates were skipped in the autokarma logic since they had their autokarma columns set to None.

I will fix this by writing a new migration that sets all Updates with autokarma set to NULL to True.

bowlofeggs added a commit to bowlofeggs/bodhi that referenced this issue Sep 24, 2016
bowlofeggs added a commit that referenced this issue Sep 24, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.