Fix disappearing badges

• Fix #160: badges were being removed only when abandoning a task, which triggered the incorrect removal of all badges