Added option to generate changelog from a list of PRs

CHANGELOG.md

@@ -1,3 +1,29 @@
+## ClickHouse release 19.11.4.24, 2019-08-01
+
+### Bug Fix
+* Fix bug with writing secondary indices marks with adaptive granularity. [#6126](https://github.com/yandex/ClickHouse/pull/6126) ([alesapin](https://github.com/alesapin))
+* Fix `WITH ROLLUP` and `WITH CUBE` modifiers of `GROUP BY` with two-level aggregation. [#6225](https://github.com/yandex/ClickHouse/pull/6225) ([Anton Popov](https://github.com/CurtizJ))
+* Fixed hang in `JSONExtractRaw` function. Fixed [#6195](https://github.com/yandex/ClickHouse/issues/6195) [#6198](https://github.com/yandex/ClickHouse/pull/6198) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Fix segfault in ExternalLoader::reloadOutdated(). [#6082](https://github.com/yandex/ClickHouse/pull/6082) ([Vitaly Baranov](https://github.com/vitlibar))
+* Fixed the case when server may close listening sockets but not shutdown and continue serving remaining queries. You may end up with two running clickhouse-server processes. Sometimes, the server may return an error `bad_function_call` for remaining queries. [#6231](https://github.com/yandex/ClickHouse/pull/6231) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Fixed useless and incorrect condition on update field for initial loading of external dictionaries via ODBC, MySQL, ClickHouse and HTTP. This fixes [#6069](https://github.com/yandex/ClickHouse/issues/6069) [#6083](https://github.com/yandex/ClickHouse/pull/6083) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Fixed irrelevant exception in cast of `LowCardinality(Nullable)` to not-Nullable column in case if it doesn't contain Nulls (e.g. in query like `SELECT CAST(CAST('Hello' AS LowCardinality(Nullable(String))) AS String)`. [#6094](https://github.com/yandex/ClickHouse/issues/6094) [#6119](https://github.com/yandex/ClickHouse/pull/6119) ([Nikolai Kochetov](https://github.com/KochetovNicolai))
+* Fix non-deterministic result of "uniq" aggregate function in extreme rare cases. The bug was present in all ClickHouse versions. [#6058](https://github.com/yandex/ClickHouse/pull/6058) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Segfault when we set a little bit too high CIDR on the function `IPv6CIDRToRange`. [#6068](https://github.com/yandex/ClickHouse/pull/6068) ([Guillaume Tassery](https://github.com/YiuRULE))
+* Fixed small memory leak when server throw many exceptions from many different contexts. [#6144](https://github.com/yandex/ClickHouse/pull/6144) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Fix the situation when consumer got paused before subscription and not resumed afterwards. [#6075](https://github.com/yandex/ClickHouse/pull/6075) ([Ivan](https://github.com/abyss7)) Note that Kafka is broken in this version.
+* Clearing the Kafka data buffer from the previous read operation that was completed with an error [#6026](https://github.com/yandex/ClickHouse/pull/6026) ([Nikolay](https://github.com/bopohaa)) Note that Kafka is broken in this version.
+* Since `StorageMergeTree::background_task_handle` is initialized in `startup()` the `MergeTreeBlockOutputStream::write()` may try to use it before initialization. Just check if it is initialized. [#6080](https://github.com/yandex/ClickHouse/pull/6080) ([Ivan](https://github.com/abyss7))
+
+### Build/Testing/Packaging Improvement
+* Added official `rpm` packages. [#5740](https://github.com/yandex/ClickHouse/pull/5740) ([proller](https://github.com/proller)) ([alesapin](https://github.com/alesapin))
+* Add an ability to build `.rpm` and `.tgz` packages with `packager` script. [#5769](https://github.com/yandex/ClickHouse/pull/5769) ([alesapin](https://github.com/alesapin))
+* Fixes for "Arcadia" build system. [#6223](https://github.com/yandex/ClickHouse/pull/6223) ([proller](https://github.com/proller))
+
+### Backward Incompatible Change
+* `Kafka` is broken in this version.
+
+
 ## ClickHouse release 19.11.3.11, 2019-07-18
 
 ### New Feature
@@ -35,6 +61,7 @@
 * clickhouse-copier: Fix use-after free on shutdown [#5752](https://github.com/yandex/ClickHouse/pull/5752) ([proller](https://github.com/proller))
 * Updated `simdjson`. Fixed the issue that some invalid JSONs with zero bytes successfully parse. [#5938](https://github.com/yandex/ClickHouse/pull/5938) ([alexey-milovidov](https://github.com/alexey-milovidov))
 * Fix shutdown of SystemLogs [#5802](https://github.com/yandex/ClickHouse/pull/5802) ([Anton Popov](https://github.com/CurtizJ))
+* Fix hanging when condition in invalidate_query depends on a dictionary. [#6011](https://github.com/yandex/ClickHouse/pull/6011) ([Vitaly Baranov](https://github.com/vitlibar))
 
 ### Improvement
 * Allow unresolvable addresses in cluster configuration. They will be considered unavailable and tried to resolve at every connection attempt. This is especially useful for Kubernetes. This fixes [#5714](https://github.com/yandex/ClickHouse/issues/5714) [#5924](https://github.com/yandex/ClickHouse/pull/5924) ([alexey-milovidov](https://github.com/alexey-milovidov))
@@ -55,13 +82,12 @@
 * Inverting ngramSearch to be more intuitive [#5807](https://github.com/yandex/ClickHouse/pull/5807) ([Danila Kutenin](https://github.com/danlark1))
 * Add user parsing in HDFS engine builder [#5946](https://github.com/yandex/ClickHouse/pull/5946) ([akonyaev90](https://github.com/akonyaev90))
 * Update default value of `max_ast_elements parameter` [#5933](https://github.com/yandex/ClickHouse/pull/5933) ([Artem Konovalov](https://github.com/izebit))
+* Added a notion of obsolete settings. The obsolete setting `allow_experimental_low_cardinality_type` can be used with no effect. [0f15c01c6802f7ce1a1494c12c846be8c98944cd](https://github.com/yandex/ClickHouse/commit/0f15c01c6802f7ce1a1494c12c846be8c98944cd) [Alexey Milovidov](https://github.com/alexey-milovidov)
 
 ### Performance Improvement
 * Increase number of streams to SELECT from Merge table for more uniform distribution of threads. Added setting `max_streams_multiplier_for_merge_tables`. This fixes [#5797](https://github.com/yandex/ClickHouse/issues/5797) [#5915](https://github.com/yandex/ClickHouse/pull/5915) ([alexey-milovidov](https://github.com/alexey-milovidov))
 
 ### Build/Testing/Packaging Improvement
-* Added official `rpm` packages. [#5740](https://github.com/yandex/ClickHouse/pull/5740) ([proller](https://github.com/proller)) ([alesapin](https://github.com/alesapin))
-* Add an ability to build `.rpm` and `.tgz` packages with `packager` script. [#5769](https://github.com/yandex/ClickHouse/pull/5769) ([alesapin](https://github.com/alesapin))
 * Add a backward compatibility test for client-server interaction with different versions of clickhouse. [#5868](https://github.com/yandex/ClickHouse/pull/5868) ([alesapin](https://github.com/alesapin))
 * Test coverage information in every commit and pull request. [#5896](https://github.com/yandex/ClickHouse/pull/5896) ([alesapin](https://github.com/alesapin))
 * Cooperate with address sanitizer to support our custom allocators (`Arena` and `ArenaWithFreeLists`) for better debugging of "use-after-free" errors. [#5728](https://github.com/yandex/ClickHouse/pull/5728) ([akuzm](https://github.com/akuzm))

utils/make_changelog.py

@@ -131,6 +131,24 @@ def find_pull_requests(commits, token, max_retries, retry_timeout):
     return not_found_commits, pull_requests
 
 
+# Find pull requests by list of numbers
+def find_pull_requests_by_num(pull_requests_nums, token, max_retries, retry_timeout):
+    pull_requests = {}
+
+    for pr in pull_requests_nums:
+        item = github_api_get_json('repos/{}/pulls/{}'.format(repo, pr), token, max_retries, retry_timeout)
+
+        number = item['number']
+        if number not in pull_requests:
+            pull_requests[number] = {
+                'title': item['title'],
+                'description': item['body'],
+                'user': item['user']['login'],
+            }
+
+    return pull_requests
+
+
 # Get users for all unknown commits and pull requests.
 def get_users_info(pull_requests, commits_info, token, max_retries, retry_timeout):
 
@@ -305,10 +323,12 @@ def save_state(state_file, state):
         f.write(json.dumps(state, indent=4, separators=(',', ': ')))
 
 
-def make_changelog(new_tag, prev_tag, repo, repo_folder, state_file, token, max_retries, retry_timeout):
+def make_changelog(new_tag, prev_tag, pull_requests_nums, repo, repo_folder, state_file, token, max_retries, retry_timeout):
 
-    base_sha = get_merge_base(new_tag, prev_tag, repo_folder)
-    logging.info('Base sha: %s', base_sha)
+    base_sha = None
+    if new_tag and prev_tag:
+        base_sha = get_merge_base(new_tag, prev_tag, repo_folder)
+        logging.info('Base sha: %s', base_sha)
 
     # Step 1. Get commits from merge_base to new_tag HEAD.
     # Result is a list of commits + map with commits info (author, message)
@@ -349,24 +369,27 @@ def make_changelog(new_tag, prev_tag, repo, repo_folder, state_file, token, max_
             users = state['users']
             is_users_loaded = True
 
-    state['base_sha'] = base_sha
-    state['new_tag'] = new_tag
-    state['prev_tag'] = prev_tag
+    if base_sha:
+        state['base_sha'] = base_sha
+        state['new_tag'] = new_tag
+        state['prev_tag'] = prev_tag
 
-    if not is_commits_loaded:
-        logging.info('Getting commits using github api.')
-        commits = get_commits_from_branch(repo, new_tag, base_sha, commits_info, 100, token, max_retries, retry_timeout)
-        state['commits'] = commits
-        state['commits_info'] = commits_info
+        if not is_commits_loaded:
+            logging.info('Getting commits using github api.')
+            commits = get_commits_from_branch(repo, new_tag, base_sha, commits_info, 100, token, max_retries, retry_timeout)
+            state['commits'] = commits
+            state['commits_info'] = commits_info
 
-    logging.info('Found %d commits from %s to %s.\n', len(commits), new_tag, base_sha)
-    save_state(state_file, state)
+        logging.info('Found %d commits from %s to %s.\n', len(commits), new_tag, base_sha)
+        save_state(state_file, state)
 
-    if not is_pull_requests_loaded:
-        logging.info('Searching for pull requests using github api.')
-        unknown_commits, pull_requests = find_pull_requests(commits, token, max_retries, retry_timeout)
-        state['unknown_commits'] = unknown_commits
-        state['pull_requests'] = pull_requests
+        if not is_pull_requests_loaded:
+            logging.info('Searching for pull requests using github api.')
+            unknown_commits, pull_requests = find_pull_requests(commits, token, max_retries, retry_timeout)
+            state['unknown_commits'] = unknown_commits
+            state['pull_requests'] = pull_requests
+    else:
+        pull_requests = find_pull_requests_by_num(pull_requests_nums.split(','), token, max_retries, retry_timeout)
 
     logging.info('Found %d pull requests and %d unknown commits.\n', len(pull_requests), len(unknown_commits))
     save_state(state_file, state)
@@ -393,8 +416,9 @@ def make_changelog(new_tag, prev_tag, repo, repo_folder, state_file, token, max_
 if __name__ == '__main__':
 
     parser = argparse.ArgumentParser(description='Make changelog.')
-    parser.add_argument('prev_release_tag', help='Git tag from previous release.')
-    parser.add_argument('new_release_tag', help='Git tag for new release.')
+    parser.add_argument('prev_release_tag', nargs='?', help='Git tag from previous release.')
+    parser.add_argument('new_release_tag', nargs='?', help='Git tag for new release.')
+    parser.add_argument('--pull-requests', help='Process the specified list of pull-request numbers (comma separated) instead of commits between tags.')
     parser.add_argument('--token', help='Github token. Use it to increase github api query limit.')
     parser.add_argument('--directory', help='ClickHouse repo directory. Script dir by default.')
     parser.add_argument('--state', help='File to dump inner states result.', default='changelog_state.json')
@@ -412,9 +436,16 @@ def make_changelog(new_tag, prev_tag, repo, repo_folder, state_file, token, max_
     repo = args.repo
     max_retry = args.max_retry
     retry_timeout = args.retry_timeout
+    pull_requests = args.pull_requests
+
+    if (not prev_release_tag or not new_release_tag) and not pull_requests:
+        raise Exception('Either release tags or --pull-requests must be specified')
+
+    if prev_release_tag and new_release_tag and pull_requests:
+        raise Exception('Either release tags or --pull-requests must be specified')
 
     logging.basicConfig(level=logging.INFO, format='%(asctime)s %(message)s')
 
     repo_folder = os.path.expanduser(repo_folder)
 
-    make_changelog(new_release_tag, prev_release_tag, repo, repo_folder, state_file, token, max_retry, retry_timeout)
+    make_changelog(new_release_tag, prev_release_tag, pull_requests, repo, repo_folder, state_file, token, max_retry, retry_timeout)