Backport script #39235

Felixoid · 2022-07-14T16:24:21Z

Changelog category (leave one):

Not for changelog (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Rewrite cherry-pick/backport task to PyGitHub API

It runs for 30 seconds now instead of 1h40, see the comment below.

Felixoid · 2022-07-14T19:08:18Z

> time python cherry_pick.py --token $GH_TOKEN --debug-helpers
INFO:root:Getting release PRs
DEBUG:github_helper:Search issues, args=(), kwards={'query': 'type:pr repo:ClickHouse/ClickHouse is:open', 'sort': 'created', 'order': 'asc', 'type': 'pr', 'label': 'release'}
DEBUG:github_helper:Getting PR #26176 from API
DEBUG:github_helper:Getting PR #35373 from API
DEBUG:github_helper:Getting PR #36501 from API
DEBUG:github_helper:Getting PR #37345 from API
DEBUG:github_helper:Getting PR #38146 from API
INFO:root:Active releases: 21.8, 22.3, 22.4, 22.5, 22.6
DEBUG:git_helper:Running command: git merge-base 21.8 master
DEBUG:git_helper:Running command: git log -1 --format=format:%cs f48c5af90c2ad51955d1ee3b6b05d006b03e4238
INFO:root:Receive PRs suppose to be backported
DEBUG:github_helper:Search issues, args=(), kwards={'query': 'type:pr repo:ClickHouse/ClickHouse -label:pr-backported', 'label': 'v21.8-must-backport,v22.3-must-backport,v22.4-must-backport,v22.5-must-backport,v22.6-must-backport,pr-must-backport', 'merged': '2021-07-09..2022-07-15'}
DEBUG:github_helper:Getting PR #39125 from API
DEBUG:github_helper:Getting PR #38838 from API
DEBUG:github_helper:Getting PR #38731 from API
DEBUG:github_helper:Getting PR #38722 from API
DEBUG:github_helper:Getting PR #38029 from API
DEBUG:github_helper:Getting PR #37273 from API
DEBUG:github_helper:Getting PR #37155 from API
INFO:root:PRs to be backported:
 https://github.com/ClickHouse/ClickHouse/pull/39125
 https://github.com/ClickHouse/ClickHouse/pull/38838
 https://github.com/ClickHouse/ClickHouse/pull/38731
 https://github.com/ClickHouse/ClickHouse/pull/38722
 https://github.com/ClickHouse/ClickHouse/pull/38029
 https://github.com/ClickHouse/ClickHouse/pull/37273
 https://github.com/ClickHouse/ClickHouse/pull/37155
INFO:root:  PR #39125 is suppose to be backported to 21.8, 22.3, 22.4, 22.5, 22.6
DEBUG:github_helper:Search issues, args=(), kwards={'query': 'type:pr repo:ClickHouse/ClickHouse head:backport/21.8/39125 head:cherrypick/21.8/ec24f730b12074446b334eafbf94e7d505cdec6c head:backport/22.3/39125 head:cherrypick/22.3/ec24f730b12074446b334eafbf94e7d505cdec6c head:backport/22.4/39125 head:cherrypick/22.4/ec24f730b12074446b334eafbf94e7d505cdec6c head:backport/22.5/39125 head:cherrypick/22.5/ec24f730b12074446b334eafbf94e7d505cdec6c head:backport/22.6/39125 head:cherrypick/22.6/ec24f730b12074446b334eafbf94e7d505cdec6c'}
DEBUG:github_helper:Getting PR #39152 from API
DEBUG:github_helper:Getting PR #39151 from API
DEBUG:github_helper:Getting PR #39150 from API
DEBUG:github_helper:Getting PR #39149 from API
DEBUG:github_helper:Getting PR #39146 from API
DEBUG:github_helper:Getting PR #39145 from API
DEBUG:github_helper:Getting PR #39144 from API
DEBUG:github_helper:Getting PR #39143 from API
DEBUG:github_helper:Getting PR #39142 from API
INFO:root:Cherrypick PR #39145 for PR #39125 have conflicts and unable to be merged
INFO:root:  PR #38838 is suppose to be backported to 21.8, 22.3, 22.4, 22.5, 22.6
DEBUG:github_helper:Search issues, args=(), kwards={'query': 'type:pr repo:ClickHouse/ClickHouse head:backport/21.8/38838 head:cherrypick/21.8/4a320ef45d0a3427c8aa6efc42c7fae6935640db head:backport/22.3/38838 head:cherrypick/22.3/4a320ef45d0a3427c8aa6efc42c7fae6935640db head:backport/22.4/38838 head:cherrypick/22.4/4a320ef45d0a3427c8aa6efc42c7fae6935640db head:backport/22.5/38838 head:cherrypick/22.5/4a320ef45d0a3427c8aa6efc42c7fae6935640db head:backport/22.6/38838 head:cherrypick/22.6/4a320ef45d0a3427c8aa6efc42c7fae6935640db'}
DEBUG:github_helper:Getting PR #38853 from API
DEBUG:github_helper:Getting PR #38847 from API
DEBUG:github_helper:Getting PR #38846 from API
DEBUG:github_helper:Getting PR #38845 from API
DEBUG:github_helper:Getting PR #38844 from API
DEBUG:github_helper:Getting PR #38843 from API
INFO:root:Cherrypick PR #38844 for PR #38838 have conflicts and unable to be merged
INFO:root:Cherrypick PR #38843 for PR #38838 have conflicts and unable to be merged
INFO:root:Cherrypick PR #38846 for PR #38838 have conflicts and unable to be merged
INFO:root:Cherrypick PR #38845 for PR #38838 have conflicts and unable to be merged
INFO:root:  PR #38731 is suppose to be backported to 21.8, 22.3, 22.4, 22.5, 22.6
DEBUG:github_helper:Search issues, args=(), kwards={'query': 'type:pr repo:ClickHouse/ClickHouse head:backport/21.8/38731 head:cherrypick/21.8/c7110123991f617069ee76ae509aef923907519a head:backport/22.3/38731 head:cherrypick/22.3/c7110123991f617069ee76ae509aef923907519a head:backport/22.4/38731 head:cherrypick/22.4/c7110123991f617069ee76ae509aef923907519a head:backport/22.5/38731 head:cherrypick/22.5/c7110123991f617069ee76ae509aef923907519a head:backport/22.6/38731 head:cherrypick/22.6/c7110123991f617069ee76ae509aef923907519a'}
DEBUG:github_helper:Getting PR #38932 from API
DEBUG:github_helper:Getting PR #38925 from API
DEBUG:github_helper:Getting PR #38924 from API
DEBUG:github_helper:Getting PR #38923 from API
DEBUG:github_helper:Getting PR #38922 from API
DEBUG:github_helper:Getting PR #38921 from API
INFO:root:Cherrypick PR #38922 for PR #38731 have conflicts and unable to be merged
INFO:root:Cherrypick PR #38921 for PR #38731 have conflicts and unable to be merged
INFO:root:Cherrypick PR #38923 for PR #38731 have conflicts and unable to be merged
INFO:root:Cherrypick PR #38925 for PR #38731 have conflicts and unable to be merged
INFO:root:  PR #38722 is suppose to be backported to 21.8, 22.3, 22.4, 22.5, 22.6
DEBUG:github_helper:Search issues, args=(), kwards={'query': 'type:pr repo:ClickHouse/ClickHouse head:backport/21.8/38722 head:cherrypick/21.8/bfc9ed61725ae1e123a0ba7ef5b4ccce8491f153 head:backport/22.3/38722 head:cherrypick/22.3/bfc9ed61725ae1e123a0ba7ef5b4ccce8491f153 head:backport/22.4/38722 head:cherrypick/22.4/bfc9ed61725ae1e123a0ba7ef5b4ccce8491f153 head:backport/22.5/38722 head:cherrypick/22.5/bfc9ed61725ae1e123a0ba7ef5b4ccce8491f153 head:backport/22.6/38722 head:cherrypick/22.6/bfc9ed61725ae1e123a0ba7ef5b4ccce8491f153'}
DEBUG:github_helper:Getting PR #38779 from API
DEBUG:github_helper:Getting PR #38778 from API
DEBUG:github_helper:Getting PR #38777 from API
DEBUG:github_helper:Getting PR #38776 from API
DEBUG:github_helper:Getting PR #38762 from API
DEBUG:github_helper:Getting PR #38761 from API
DEBUG:github_helper:Getting PR #38760 from API
DEBUG:github_helper:Getting PR #38759 from API
DEBUG:github_helper:Getting PR #38758 from API
INFO:root:Cherrypick PR #38761 for PR #38722 have conflicts and unable to be merged
INFO:root:  PR #38029 is suppose to be backported to 21.8, 22.3, 22.4, 22.5, 22.6
DEBUG:github_helper:Search issues, args=(), kwards={'query': 'type:pr repo:ClickHouse/ClickHouse head:backport/21.8/38029 head:cherrypick/21.8/fab62513aac636ec6e46abc420048f4d6cfd679a head:backport/22.3/38029 head:cherrypick/22.3/fab62513aac636ec6e46abc420048f4d6cfd679a head:backport/22.4/38029 head:cherrypick/22.4/fab62513aac636ec6e46abc420048f4d6cfd679a head:backport/22.5/38029 head:cherrypick/22.5/fab62513aac636ec6e46abc420048f4d6cfd679a head:backport/22.6/38029 head:cherrypick/22.6/fab62513aac636ec6e46abc420048f4d6cfd679a'}
DEBUG:github_helper:Getting PR #38244 from API
DEBUG:github_helper:Getting PR #38243 from API
DEBUG:github_helper:Getting PR #38242 from API
DEBUG:github_helper:Getting PR #38241 from API
DEBUG:github_helper:Getting PR #38233 from API
DEBUG:github_helper:Getting PR #38232 from API
DEBUG:github_helper:Getting PR #38231 from API
DEBUG:github_helper:Getting PR #38230 from API
DEBUG:github_helper:Getting PR #38229 from API
INFO:root:Cherrypick PR #38231 for PR #38029 have conflicts and unable to be merged
INFO:root:  PR #37273 is suppose to be backported to 21.8, 22.3, 22.4, 22.5, 22.6
DEBUG:github_helper:Search issues, args=(), kwards={'query': 'type:pr repo:ClickHouse/ClickHouse head:backport/21.8/37273 head:cherrypick/21.8/cd2ae72aacd03deb11cc0419d3da0bad5fe61e5c head:backport/22.3/37273 head:cherrypick/22.3/cd2ae72aacd03deb11cc0419d3da0bad5fe61e5c head:backport/22.4/37273 head:cherrypick/22.4/cd2ae72aacd03deb11cc0419d3da0bad5fe61e5c head:backport/22.5/37273 head:cherrypick/22.5/cd2ae72aacd03deb11cc0419d3da0bad5fe61e5c head:backport/22.6/37273 head:cherrypick/22.6/cd2ae72aacd03deb11cc0419d3da0bad5fe61e5c'}
DEBUG:github_helper:Getting PR #37748 from API
DEBUG:github_helper:Getting PR #37747 from API
DEBUG:github_helper:Getting PR #37734 from API
DEBUG:github_helper:Getting PR #37733 from API
DEBUG:github_helper:Getting PR #37732 from API
DEBUG:github_helper:Getting PR #37731 from API
INFO:root:Cherrypick PR #37732 for PR #37273 have conflicts and unable to be merged
INFO:root:Cherrypick PR #37734 for PR #37273 have conflicts and unable to be merged
DEBUG:git_helper:Running command: git -c user.email=robot-clickhouse@clickhouse.com -c user.name=robot-clickhouse -c commit.gpgsign=false checkout -f 22.6
warning: unable to rmdir 'contrib/base-x': Directory not empty
warning: unable to rmdir 'contrib/nats-io': Directory not empty
warning: unable to rmdir 'contrib/vectorscan': Directory not empty
Switched to branch '22.6'
DEBUG:git_helper:Running command: git -c user.email=robot-clickhouse@clickhouse.com -c user.name=robot-clickhouse -c commit.gpgsign=false checkout -B backport/22.6/37273
Switched to and reset branch 'backport/22.6/37273'
DEBUG:git_helper:Running command: git rev-parse cd2ae72aacd03deb11cc0419d3da0bad5fe61e5c^1
DEBUG:git_helper:Running command: git -c user.email=robot-clickhouse@clickhouse.com -c user.name=robot-clickhouse -c commit.gpgsign=false merge -s ours --no-edit dea1706d4e65572a23636483bb8800f3c733a460
DEBUG:git_helper:Running command: git -c user.email=robot-clickhouse@clickhouse.com -c user.name=robot-clickhouse -c commit.gpgsign=false branch -f cherrypick/22.6/cd2ae72aacd03deb11cc0419d3da0bad5fe61e5c cd2ae72aacd03deb11cc0419d3da0bad5fe61e5c
DEBUG:git_helper:Running command: git -c user.email=robot-clickhouse@clickhouse.com -c user.name=robot-clickhouse -c commit.gpgsign=false merge --no-commit --no-ff cherrypick/22.6/cd2ae72aacd03deb11cc0419d3da0bad5fe61e5c
INFO:root:Release branch 22.6 already contain changes from 37273
INFO:root:  PR #37155 is suppose to be backported to 21.8, 22.3, 22.4, 22.5, 22.6
DEBUG:github_helper:Search issues, args=(), kwards={'query': 'type:pr repo:ClickHouse/ClickHouse head:backport/21.8/37155 head:cherrypick/21.8/c66e2c8095e627588d1d422d7ba9396129ce3b4f head:backport/22.3/37155 head:cherrypick/22.3/c66e2c8095e627588d1d422d7ba9396129ce3b4f head:backport/22.4/37155 head:cherrypick/22.4/c66e2c8095e627588d1d422d7ba9396129ce3b4f head:backport/22.5/37155 head:cherrypick/22.5/c66e2c8095e627588d1d422d7ba9396129ce3b4f head:backport/22.6/37155 head:cherrypick/22.6/c66e2c8095e627588d1d422d7ba9396129ce3b4f'}
DEBUG:github_helper:Getting PR #38821 from API
DEBUG:github_helper:Getting PR #38820 from API
DEBUG:github_helper:Getting PR #38819 from API
DEBUG:github_helper:Getting PR #38818 from API
DEBUG:github_helper:Getting PR #38817 from API
INFO:root:Cherrypick PR #38817 for PR #37155 have conflicts and unable to be merged
INFO:root:Cherrypick PR #38819 for PR #37155 have conflicts and unable to be merged
INFO:root:Cherrypick PR #38820 for PR #37155 have conflicts and unable to be merged
INFO:root:Cherrypick PR #38821 for PR #37155 have conflicts and unable to be merged
INFO:root:Cherrypick PR #38818 for PR #37155 have conflicts and unable to be merged
DEBUG:git_helper:Running command: git checkout -f backport-script
warning: unable to rmdir 'contrib/hyperscan': Directory not empty
Switched to branch 'backport-script'
python cherry_pick.py --token $GH_TOKEN --debug-helpers  1,71s user 1,06s system 8% cpu 33,277 total

Felixoid · 2022-07-15T10:46:02Z

Annnnnnnd it's broken

https://github.com/ClickHouse/ClickHouse/runs/7356488540?check_suite_focus=true

Felixoid · 2022-07-15T12:18:03Z

Aaaaaand it works! https://github.com/ClickHouse/ClickHouse/runs/7357708663?check_suite_focus=true#step:5:1

Felixoid · 2022-07-15T16:02:03Z

Wow, it works! #39268

And the last check after clean-up is done https://github.com/ClickHouse/ClickHouse/runs/7361182928?check_suite_focus=true

tests/ci/github_helper.py

Felixoid · 2022-07-18T22:28:24Z

Thank you for review, I'll merge it tomorrow morning at ~8-10 UTC

Felixoid added the do not test disable testing on pull request label Jul 14, 2022

robot-ch-test-poll1 added the pr-not-for-changelog This PR should not be mentioned in the changelog label Jul 14, 2022

Felixoid force-pushed the backport-script branch 2 times, most recently from 4934222 to e649bd0 Compare July 14, 2022 16:27

This was referenced Jul 14, 2022

Backport #35512 to 21.8: Fix crash with enabled optimize_functions_to_subcolumns #39239

Closed

Cherry pick #33050 to 21.8: Native Out of Bounds Column Index #39241

Closed

Felixoid force-pushed the backport-script branch from e649bd0 to 438f979 Compare July 14, 2022 18:57

Felixoid marked this pull request as ready for review July 14, 2022 19:01

Felixoid force-pushed the backport-script branch from c5cd2f8 to a2a9db7 Compare July 14, 2022 19:05

Felixoid force-pushed the backport-script branch from a2a9db7 to 438f979 Compare July 15, 2022 10:39

Felixoid force-pushed the backport-script branch 4 times, most recently from d0c6d07 to 279df27 Compare July 15, 2022 12:15

Felixoid force-pushed the backport-script branch from 3eabd82 to e90533f Compare July 15, 2022 14:40

Felixoid added 6 commits July 15, 2022 17:21

Rewrite cherry_pick.py to PyGithub API

909e871

Rename workflow yml to reflect the name, increase frequency

0c766a6

Add early check if PR is already in a release branch

30b0995

Process exceptions per PR to not fail the script

4879766

Replace sys.exit by Exception to rollback stash and clear_repo

dc6db5b

Clean-up some staff left after finishing

6e48b08

Felixoid force-pushed the backport-script branch from e90533f to 6e48b08 Compare July 15, 2022 15:59

Felixoid force-pushed the backport-script branch from 8068f76 to 6e48b08 Compare July 15, 2022 16:02

Felixoid commented Jul 15, 2022

View reviewed changes

tests/ci/github_helper.py Outdated Show resolved Hide resolved

Felixoid commented Jul 15, 2022

View reviewed changes

tests/ci/github_helper.py Outdated Show resolved Hide resolved

Felixoid commented Jul 15, 2022

View reviewed changes

tests/ci/github_helper.py Outdated Show resolved Hide resolved

Felixoid commented Jul 15, 2022

View reviewed changes

tests/ci/github_helper.py Outdated Show resolved Hide resolved

Felixoid added 2 commits July 17, 2022 14:05

Fix found points during the review

9cc06ea

Improve debug logging of github_helper

caef672

nickitat approved these changes Jul 18, 2022

View reviewed changes

yakov-olkhovskiy self-assigned this Jul 18, 2022

yakov-olkhovskiy approved these changes Jul 19, 2022

View reviewed changes

Felixoid merged commit 617f774 into master Jul 19, 2022

Felixoid deleted the backport-script branch July 19, 2022 08:36

Felixoid added v22.3-must-backport v22.5-must-backport labels Aug 10, 2022

This was referenced Aug 10, 2022

Cherry pick #39235 to 22.3: Backport script #40083

Merged

Cherry pick #39235 to 22.5: Backport script #40084

Merged

Cherry pick #39235 to 22.6: Backport script #40085

Merged

robot-clickhouse pushed a commit that referenced this pull request Aug 10, 2022

Backport #39235 to 22.3: Backport script

d53e81d

robot-ch-test-poll mentioned this pull request Aug 10, 2022

Backport #39235 to 22.3: Backport script #40087

Closed

robot-ch-test-poll mentioned this pull request Aug 10, 2022

Backport #39235 to 22.5: Backport script #40088

Closed

robot-clickhouse pushed a commit that referenced this pull request Aug 10, 2022

Backport #39235 to 22.5: Backport script

757b9a5

robot-ch-test-poll mentioned this pull request Aug 10, 2022

Backport #39235 to 22.6: Backport script #40089

Closed

robot-clickhouse pushed a commit that referenced this pull request Aug 10, 2022

Backport #39235 to 22.6: Backport script

081793a

robot-ch-test-poll added the pr-backports-created Backport PRs are successfully created, it won't be processed by CI script anymore label Aug 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Backport script #39235

Backport script #39235

Felixoid commented Jul 14, 2022 •

edited

Felixoid commented Jul 14, 2022

Felixoid commented Jul 15, 2022

Felixoid commented Jul 15, 2022

Felixoid commented Jul 15, 2022 •

edited

Felixoid commented Jul 18, 2022

Backport script #39235

Backport script #39235

Conversation

Felixoid commented Jul 14, 2022 • edited

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Felixoid commented Jul 14, 2022

Felixoid commented Jul 15, 2022

Felixoid commented Jul 15, 2022

Felixoid commented Jul 15, 2022 • edited

Felixoid commented Jul 18, 2022

Felixoid commented Jul 14, 2022 •

edited

Felixoid commented Jul 15, 2022 •

edited