New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Old execution history hidden after running pipelines in >= 1.8.x #3256

Closed
mbasc opened this Issue Aug 27, 2018 · 9 comments

Comments

Projects
None yet
7 participants
@mbasc
Copy link

mbasc commented Aug 27, 2018

Issue Summary:

Old (<= 1.7.x) execution history hidden after running pipelines in >= 1.8.x

Environment:

Distributed Spinnaker (V1) deployment into GKE
Upgrading from 1.7.4 to 1.8.5

Feature Area:

Orca / Execution history

Description:

After upgrading from 1.7.4 to 1.8.5, execution history is present. Immediately after starting a pipeline post upgrade, old execution history for that pipeline no longer shows up. Old executions are still present - can access directly via the API if you know the ID, or if you filter on statuses which don't have a 1.8.x execution yet. Seen in UI, verification in API per additional details below.

Not a huge deal for us, but didn't see anything in the changelog or issues, curious if this is expected, if others are seeing the same. Verified same behavior after upgrading to 1.9.0 (though via 1.8.5 - have not tried 1.7.x -> 1.9.0 directly).

Additional Details:

# after 1.8.5 upgrade, prior to running an execution on this pipeline in 1.8.5, two executions found
$ curl -Ss 'https://spinnaker.mydomain.com:8084/executions?pipelineConfigIds=9485dcd7-3a65-4c8e-a1a2-ab1076b04651&limit=5' -H "Cookie: SESSION=$SESSION_ID" -o - | jq -c '.[] | [.id, .type, (.startTime | ./1000 | todate), .status ]'
["72a5d8eb-59d4-4171-9690-0a45edd9c41a","PIPELINE","2018-07-31T20:49:49Z","TERMINAL"]
["0357f69e-b561-4973-b552-3c29c9e31448","PIPELINE","2018-07-31T21:01:54Z","SUCCEEDED"]

# as soon as a new execution (ULIDs now) is running, old executions no longer show up.
$ curl -Ss 'https://spinnaker.mydomain.com:8084/executions?pipelineConfigIds=9485dcd7-3a65-4c8e-a1a2-ab1076b04651&limit=5' -H "Cookie: SESSION=$SESSION_ID" -o - | jq -c '.[] | [.id, .type, (.startTime | ./1000 | todate), .status ]'
["01CNPYQA7VFW5T1N38B5G7PMZD","PIPELINE","2018-08-24T21:51:25Z","RUNNING"]
$ curl -Ss 'https://spinnaker.mydomain.com:8084/executions?pipelineConfigIds=9485dcd7-3a65-4c8e-a1a2-ab1076b04651&limit=5' -H "Cookie: SESSION=$SESSION_ID" -o - | jq -c '.[] | [.id, .type, (.startTime | ./1000 | todate), .status ]'
["01CNPYQA7VFW5T1N38B5G7PMZD","PIPELINE","2018-08-24T21:51:25Z","SUCCEEDED"]

# can still see old executions if filter by status (which doesn't have a new execution i assume). no idea why id a6ae8db6... wasn't in initial api call
$ curl -Ss 'https://spinnaker.mydomain.com:8084/executions?pipelineConfigIds=9485dcd7-3a65-4c8e-a1a2-ab1076b04651&limit=5&statuses=TERMINAL' -H "Cookie: SESSION=$SESSION_ID" -o - | jq -c '.[] | [.id, .type, (.startTime | ./1000 | todate), .status ]'
["a6ae8db6-f5b6-4ca2-9443-714937fe8a7c","PIPELINE","2018-07-31T20:43:23Z","TERMINAL"]
["72a5d8eb-59d4-4171-9690-0a45edd9c41a","PIPELINE","2018-07-31T20:49:49Z","TERMINAL"]

# can still see old executions via pipeline api
$ curl -Ss 'https://spinnaker.mydomain.com:8084/pipelines/0357f69e-b561-4973-b552-3c29c9e31448' -H "Cookie: SESSION=$SESSION_ID" -o - | jq -c '[.id, .type, (.startTime | ./1000 | todate), .status ]'
["0357f69e-b561-4973-b552-3c29c9e31448","PIPELINE","2018-07-31T21:01:54Z","SUCCEEDED"]

@mbasc mbasc changed the title <= 1.7.x execution history hidden after running pipelines in >= 1.8.x Old execution history hidden after running pipelines in >= 1.8.x Aug 27, 2018

@ezimanyi

This comment has been minimized.

Copy link
Member

ezimanyi commented Aug 28, 2018

I tried this quickly myself---I started a 1.7.4 Spinnaker, ran some pipelines, upgraded to 1.8.5, and was still able to see the prior executions (even after doing some executions in 1.8.5). So there's obviously some set of conditions under which this bug happens, but I didn't hit that case in my testing.

@lwander

This comment has been minimized.

Copy link
Member

lwander commented Aug 28, 2018

@mbasc can you check if there are any errors in your chrome console when loading the executions?

@lwander

This comment has been minimized.

Copy link
Member

lwander commented Aug 28, 2018

@christopherthielen or @robzienert does this ring any bells as far either UI changes or pipeline migrations go? 1.7 was released 2018-04-25 and 1.8 was released 2018-06-22

@mbasc

This comment has been minimized.

Copy link

mbasc commented Aug 28, 2018

no errors in the chrome console when loading an affected #/applications/<app>/executions page.

after @ezimanyi's comment (thanks for testing!), i also set up a clean spinnaker instance and as he found, did keep execution history after the upgrade. i'll try to compare settings, old and new execution records, etc between the affected and working spinnaker instances this morning.

fwiw, i'm seeing this on two dev spinnaker instances that should have the same configuration a prod instance, but have less activity.

@robzienert

This comment has been minimized.

Copy link
Member

robzienert commented Aug 28, 2018

Hmm, I don't recall this happening for our deployments. Perhaps clear your browser cache and see if that helps at all? ¯\_(ツ)_/¯

@christopherthielen

This comment has been minimized.

Copy link

christopherthielen commented Aug 28, 2018

this doesn't ring a bell for me either

@mbasc

This comment has been minimized.

Copy link

mbasc commented Aug 28, 2018

bah, artifact of the execution history i was using on these dev instances. '/executions' doesn't show execution history older than 2 weeks if a more recent run exists. true in 1.7.4 as well.

resolving invalid, apologies for the noise.

@mbasc mbasc closed this Aug 28, 2018

@rchench

This comment has been minimized.

Copy link

rchench commented Oct 12, 2018

I know this issue has been closed. but may I ask how to retrieve the execution history more than 14 days in this situation?

I used the API and got the same result from the UI that the history more than 14 days older than the latest one cannot be retrieved. but if I deleted my latest execution, the old pipelines (over 14 days) were back again from both UI and API.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment