tasks sort order changing, causing unnecessary restarts

[rasputnik]

We've been running bamboo 0.2.12 against marathon 0.7.5 happily for months, but today
our haproxies have been reloading every minute or so - although the state of marathon is
not changing.

Did some investigation (diffing haproxy.cfg, marathon and bamboo state URLs) it seems the order of tasks returned by http://our-marathon:8080/v2/tasks is changing - for reasons I can't understand.

This causes the generated haproxy.cfg to differ enough for bamboo to reload configuration.

our haproxy template is here:
https://gist.github.com/rasputnik/3c4a846b5aedcef6ea02

Is there any way for me to sort the tasks (in the ' {{ range $page, $task := .Tasks }}' block) using bamboos templating?

Or am I going to have to patch our bamboo to ensure these tasks remain in order?

[rasputnik]

Odd, it looks like we already sort the taskList we get from marathon here:

https://github.com/QubitProducts/bamboo/blob/v0.2.12/services/marathon/marathon.go#L140

but for some reason it gets jumbled up again by the time it's merged into the main bamboo state.

Can anyone give me a clue what's going on here?

[rasputnik]

I've filed this ticket against marathon for some clarification of what should be expected from the rest api:
mesosphere/marathon#2567

[rasputnik]

Marathon line is that the tasks are returned in random order and it's consumers responsibility to sort them. I'm still digging around to find how the tasks become jumbled, since they're definitely sorted on line 140.

[rasputnik]

Liberal use of glog.Infof confirms the Tasks within the apps are not sorted when returned by createApps. The marathonTasks are sorted when passed to the function, I'm not seeing where they get jumbled.

Also appears that https://github.com/QubitProducts/bamboo/blob/v0.2.12/services/marathon/marathon.go#L235 is not sorting the Task list, just the parent Apps.

[rasputnik]

found 2 little bugs

• sorting marathonTasks by .Id instead of .StagedAt resolves this consistently - unsure if there is an issue due to .StagedAt being a String so sorting inconsistently (our clocks recently changed so timezones may come into play), but this reliably fixes things for us
• big chunks of marathon.go use []MarathonTask instead of MarathonTaskList, which may be causing sorts to happen oddly

PR incoming (just verifying the modified code handles normal deployments correctly).