Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

singlePullRequest convergence/gated pipelines unreliable due to versions getting overwritten #2587

Open
ryandawsonuk opened this Issue Mar 1, 2019 · 7 comments

Comments

Projects
None yet
2 participants
@ryandawsonuk
Copy link
Member

ryandawsonuk commented Mar 1, 2019

Seems to be related to use of rebase, which causes the convergence checks to fail. I think it's actually the record of the versions for the chart-level convergence checks which is the problem. The content of the chart requirements file is fine at the point when the issue occurs. See comments from jenkins-x/updatebot#43 (comment) onwards and the linked examples.

@ryandawsonuk

This comment has been minimized.

Copy link
Member Author

ryandawsonuk commented Mar 4, 2019

Tried a couple of experiments above but hit problems with the shell syntax. Getting a closer to an approach.

ryandawsonuk added a commit to Activiti/example-cloud-connector that referenced this issue Mar 4, 2019

ryandawsonuk added a commit to Activiti/activiti-cloud-notifications-graphql that referenced this issue Mar 4, 2019

ryandawsonuk added a commit to Activiti/activiti-cloud-query that referenced this issue Mar 4, 2019

ryandawsonuk added a commit to Activiti/activiti-cloud-query that referenced this issue Mar 4, 2019

ryandawsonuk added a commit to Activiti/example-cloud-connector that referenced this issue Mar 4, 2019

@ryandawsonuk

This comment has been minimized.

Copy link
Member Author

ryandawsonuk commented Mar 4, 2019

The approach in these PRs is to do a backoff and retry.

This is now running in the example-runtime-bundle, activiti-cloud-query, example-cloud-connector and activiti-cloud-notifications-graphql pipelines.

It simply retries the push-version, it doesn't remove the local .updatebot-repos/. There's a risk that could be needed if updatebot doesn't do a fetch. We should see whether it still happens and that will help us find out.

It would be preferable if we could resolve in updatebot itself but our PRs aren't getting reviewed there (I've had one open for months) and it feels like a fairly big change because the updatebot whose change was overwritten thinks it got the change through. We'd have to get updatebot to wait and re-refetch and run again, which is what we're doing with this change anyway.

ryandawsonuk added a commit to Activiti/activiti-cloud-notifications-graphql that referenced this issue Mar 4, 2019

@ryandawsonuk

This comment has been minimized.

Copy link
Member Author

ryandawsonuk commented Mar 4, 2019

Just seen a case that initially looked like an instance of this Activiti/activiti-cloud-application-chart#139 (comment) but actually turned out to be due to a github failure - so there can be different causes

@ryandawsonuk

This comment has been minimized.

Copy link
Member Author

ryandawsonuk commented Mar 5, 2019

OK it still happens. Again http://jenkins.jx.35.228.195.195.nip.io/job/Activiti/job/activiti-cloud-query/job/master/81/console pushed v7.1.20 but the chart PR doesn't have the commit https://github.com/Activiti/activiti-cloud-application-chart/pull/142/commits

Probably this is because it doesn't remove the local .updatebot-repos, as noted above #2587 (comment)

So either we remove those between pushes or we try to get it to merge without the rebase and force-push

ryandawsonuk added a commit to Activiti/activiti-cloud-query that referenced this issue Mar 5, 2019

ryandawsonuk added a commit to Activiti/activiti-cloud-notifications-graphql that referenced this issue Mar 5, 2019

ryandawsonuk added a commit to Activiti/activiti-cloud-query that referenced this issue Mar 5, 2019

ryandawsonuk added a commit to Activiti/activiti-cloud-notifications-graphql that referenced this issue Mar 5, 2019

@ryandawsonuk

This comment has been minimized.

Copy link
Member Author

ryandawsonuk commented Mar 5, 2019

So this is/was most pronounced with charts but turns out it also affects singlePR convergence for activiti-cloud-dependencies. The conversation comments in Activiti/activiti-cloud-dependencies#327 reveal that activiti-cloud-acceptance-tests did push a version to it (as its pipeline also says http://jenkins.jx.35.242.205.159.nip.io/job/Activiti/job/activiti-cloud-acceptance-tests/job/develop/88/console) but then service-common pushed its version with a force-push and overwrote it.

ryandawsonuk added a commit to Activiti/activiti-cloud-acceptance-tests that referenced this issue Mar 5, 2019

@ryandawsonuk ryandawsonuk changed the title chart pipelines unreliable due to versions getting overwitten singlePullRequest convergence/gated pipelines unreliable due to versions getting overwitten Mar 6, 2019

@ryandawsonuk ryandawsonuk changed the title singlePullRequest convergence/gated pipelines unreliable due to versions getting overwitten singlePullRequest convergence/gated pipelines unreliable due to versions getting overwritten Mar 6, 2019

ryandawsonuk added a commit to Activiti/activiti-cloud-acceptance-tests that referenced this issue Mar 6, 2019

@ryandawsonuk

This comment has been minimized.

Copy link
Member Author

ryandawsonuk commented Mar 6, 2019

Build 103 of service-common produced v7.1.16 and that has now propagated through cloud-dependencies , producing v7.1.22 there and then this was propagated through the examples and into application chart and example-chart - all without any manual intervention.

As a further test build 199 of cloud-dependencies produced v7.1.23 and that has gone through smoothly too

@ryandawsonuk

This comment has been minimized.

Copy link
Member Author

ryandawsonuk commented Mar 7, 2019

To be sure I wasn't missing something I triggered another run and v7.1.25 also propagated through without any manual intervention

@salaboy salaboy modified the milestones: 7.1.0.M1, 7.1.0.M2 Mar 8, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.