Old execution history hidden after running pipelines in >= 1.8.x

[mbasc]

Issue Summary:

Old (<= 1.7.x) execution history hidden after running pipelines in >= 1.8.x

Environment:

Distributed Spinnaker (V1) deployment into GKE
Upgrading from 1.7.4 to 1.8.5

Feature Area:

Orca / Execution history

Description:

After upgrading from 1.7.4 to 1.8.5, execution history is present. Immediately after starting a pipeline post upgrade, old execution history for that pipeline no longer shows up. Old executions are still present - can access directly via the API if you know the ID, or if you filter on statuses which don't have a 1.8.x execution yet. Seen in UI, verification in API per additional details below.

Not a huge deal for us, but didn't see anything in the changelog or issues, curious if this is expected, if others are seeing the same. Verified same behavior after upgrading to 1.9.0 (though via 1.8.5 - have not tried 1.7.x -> 1.9.0 directly).

Additional Details:

# after 1.8.5 upgrade, prior to running an execution on this pipeline in 1.8.5, two executions found
$ curl -Ss 'https://spinnaker.mydomain.com:8084/executions?pipelineConfigIds=9485dcd7-3a65-4c8e-a1a2-ab1076b04651&limit=5' -H "Cookie: SESSION=$SESSION_ID" -o - | jq -c '.[] | [.id, .type, (.startTime | ./1000 | todate), .status ]'
["72a5d8eb-59d4-4171-9690-0a45edd9c41a","PIPELINE","2018-07-31T20:49:49Z","TERMINAL"]
["0357f69e-b561-4973-b552-3c29c9e31448","PIPELINE","2018-07-31T21:01:54Z","SUCCEEDED"]

# as soon as a new execution (ULIDs now) is running, old executions no longer show up.
$ curl -Ss 'https://spinnaker.mydomain.com:8084/executions?pipelineConfigIds=9485dcd7-3a65-4c8e-a1a2-ab1076b04651&limit=5' -H "Cookie: SESSION=$SESSION_ID" -o - | jq -c '.[] | [.id, .type, (.startTime | ./1000 | todate), .status ]'
["01CNPYQA7VFW5T1N38B5G7PMZD","PIPELINE","2018-08-24T21:51:25Z","RUNNING"]
$ curl -Ss 'https://spinnaker.mydomain.com:8084/executions?pipelineConfigIds=9485dcd7-3a65-4c8e-a1a2-ab1076b04651&limit=5' -H "Cookie: SESSION=$SESSION_ID" -o - | jq -c '.[] | [.id, .type, (.startTime | ./1000 | todate), .status ]'
["01CNPYQA7VFW5T1N38B5G7PMZD","PIPELINE","2018-08-24T21:51:25Z","SUCCEEDED"]

# can still see old executions if filter by status (which doesn't have a new execution i assume). no idea why id a6ae8db6... wasn't in initial api call
$ curl -Ss 'https://spinnaker.mydomain.com:8084/executions?pipelineConfigIds=9485dcd7-3a65-4c8e-a1a2-ab1076b04651&limit=5&statuses=TERMINAL' -H "Cookie: SESSION=$SESSION_ID" -o - | jq -c '.[] | [.id, .type, (.startTime | ./1000 | todate), .status ]'
["a6ae8db6-f5b6-4ca2-9443-714937fe8a7c","PIPELINE","2018-07-31T20:43:23Z","TERMINAL"]
["72a5d8eb-59d4-4171-9690-0a45edd9c41a","PIPELINE","2018-07-31T20:49:49Z","TERMINAL"]

# can still see old executions via pipeline api
$ curl -Ss 'https://spinnaker.mydomain.com:8084/pipelines/0357f69e-b561-4973-b552-3c29c9e31448' -H "Cookie: SESSION=$SESSION_ID" -o - | jq -c '[.id, .type, (.startTime | ./1000 | todate), .status ]'
["0357f69e-b561-4973-b552-3c29c9e31448","PIPELINE","2018-07-31T21:01:54Z","SUCCEEDED"]

[ezimanyi]

I tried this quickly myself---I started a 1.7.4 Spinnaker, ran some pipelines, upgraded to 1.8.5, and was still able to see the prior executions (even after doing some executions in 1.8.5). So there's obviously some set of conditions under which this bug happens, but I didn't hit that case in my testing.

[lwander]

@mbasc can you check if there are any errors in your chrome console when loading the executions?

[lwander]

@christopherthielen or @robzienert does this ring any bells as far either UI changes or pipeline migrations go? 1.7 was released 2018-04-25 and 1.8 was released 2018-06-22

[mbasc]

no errors in the chrome console when loading an affected #/applications/<app>/executions page.

after @ezimanyi's comment (thanks for testing!), i also set up a clean spinnaker instance and as he found, did keep execution history after the upgrade. i'll try to compare settings, old and new execution records, etc between the affected and working spinnaker instances this morning.

fwiw, i'm seeing this on two dev spinnaker instances that should have the same configuration a prod instance, but have less activity.

[robzienert]

Hmm, I don't recall this happening for our deployments. Perhaps clear your browser cache and see if that helps at all? ¯\_(ツ)_/¯

[christopherthielen]

this doesn't ring a bell for me either

[mbasc]

bah, artifact of the execution history i was using on these dev instances. '/executions' doesn't show execution history older than 2 weeks if a more recent run exists. true in 1.7.4 as well.

resolving invalid, apologies for the noise.

[rchench]

I know this issue has been closed. but may I ask how to retrieve the execution history more than 14 days in this situation?

I used the API and got the same result from the UI that the history more than 14 days older than the latest one cannot be retrieved. but if I deleted my latest execution, the old pipelines (over 14 days) were back again from both UI and API.

[duftler]

Pretty sure you're running into this: https://github.com/spinnaker/orca/blob/master/orca-web/src/main/groovy/com/netflix/spinnaker/orca/controllers/TaskController.groovy#L554

Looks tweekable via config: https://github.com/spinnaker/orca/blob/master/orca-web/src/main/groovy/com/netflix/spinnaker/orca/controllers/TaskController.groovy#L83-L84

[Jurawa]

Updated locations for where this cutoff is coming from:

https://github.com/spinnaker/orca/blob/master/orca-web/src/main/groovy/com/netflix/spinnaker/orca/controllers/TaskController.groovy#L666

And the config to tweak:

https://github.com/spinnaker/orca/blob/master/orca-web/src/main/groovy/com/netflix/spinnaker/orca/controllers/TaskController.groovy#L93-L94

Based on a look at where the tasks.daysOfExecutionHistory config value is used, the docs here are misleading, it does not affect the cleanup poller.

Instead, it controls how many days of pipeline execution history to include when retrieving execution history from the Orca API (via Gate)... the /applications/{application}/pipelines endpoint (and possibly others). It therefore also controls how old displayed pipelines will be in the Deck UI as it uses that endpoint to retrieve executions.