Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert to python3 #15007

Merged
merged 115 commits into from
Oct 2, 2020
Merged

Convert to python3 #15007

merged 115 commits into from
Oct 2, 2020

Conversation

azat
Copy link
Collaborator

@azat azat commented Sep 18, 2020

Please squash before merge

Changelog category (leave one):

  • Build/Testing/Packaging Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Convert to python3. This closes #14886

Detailed description / Documentation draft:
python2 is already EOL, since January 1, 2020

Checklist:

  • scripts
  • stateless tests
  • stateful tests
  • integration tests
  • tests/external_models/catboost/data/build_catboost.sh left as-is, since it uses some ya command (cc @KochetovNicolai)

Tech checklist:

  • tests (via 2to3)
  • scripts (via 2to3)
  • scripts (via `sed, package list)
  • some documentation
  • Dockerfile (via sed, package list)
  • pip (via s/pip/python3 -m pip/)
  • shebang's (via sed)
  • division in py3 is not c-like, use //

P.S I used 2to3 before for converting, and it was OK, I guess for massive converting it is to use it.
P.P.S. some commits have commands, that has been executed to update files

This does not do any significant cleanup (but there are lots of icky code), only convert py2 to py3

Refs: #15006
Refs: #10562 (comment)

$ git ls-files '*.py' | xargs 2to3 -wn
$ gg 'env python$' | cut -d: -f1 | sort -u | xargs 2to3 -nw
$ gg 'env python$' | cut -d: -f1 | sort -u | xargs sed -i 's/env python$/env python3/g'
$ git ls-files -- */Dockerfile | xargs sed -i 's/\<python\>/python3/g'
$ git co docker/test/integration/resolver/Dockerfile
$ gg '^python ' tests/ | cut -d: -f1 | xargs sed -i 's/^python /python3 /g'
$ gg 'pip ' docker | cut -d: -f1 | sort -u | xargs sed -i 's/RUN pip /RUN python3 -m pip /'
@robot-clickhouse robot-clickhouse added the pr-build Pull request with build/testing/packaging improvement label Sep 18, 2020
@azat azat marked this pull request as draft September 18, 2020 22:51
@azat
Copy link
Collaborator Author

azat commented Sep 20, 2020

Conflicts with #15032 (since it has few bits for converting to python3 too)

@azat
Copy link
Collaborator Author

azat commented Sep 29, 2020

If upstream/master will not go ahead then this maybe enough

@azat azat marked this pull request as ready for review September 30, 2020 05:13
docs/tools/test.py Outdated Show resolved Hide resolved
New tests:

$ git di ...upstream/master --stat tests/integration/
 tests/integration/helpers/cluster.py                                        |  40 +++++++++++++++++++++++++++++++++++++---
 tests/integration/runner                                                    |   4 +++-
 tests/integration/test_compression_nested_columns/__init__.py               |   0
 tests/integration/test_compression_nested_columns/test.py                   |  68 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/integration/test_mysql_database_engine/test.py                        |  33 +++++++++++++++++++++++++++++++++
 tests/integration/test_storage_kerberized_kafka/__init__.py                 |   0
 tests/integration/test_storage_kerberized_kafka/clickhouse_path/EMPTY_DIR   |   0
 tests/integration/test_storage_kerberized_kafka/configs/kafka.xml           |  26 ++++++++++++++++++++++++++
 tests/integration/test_storage_kerberized_kafka/configs/log_conf.xml        |  11 +++++++++++
 tests/integration/test_storage_kerberized_kafka/kerberos_image_config.sh    | 132 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/integration/test_storage_kerberized_kafka/secrets/broker_jaas.conf    |  14 ++++++++++++++
 tests/integration/test_storage_kerberized_kafka/secrets/krb.conf            |  22 ++++++++++++++++++++++
 tests/integration/test_storage_kerberized_kafka/secrets/zookeeper_jaas.conf |  14 ++++++++++++++
 tests/integration/test_storage_kerberized_kafka/test.py                     | 146 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 14 files changed, 506 insertions(+), 4 deletions(-)
$ git di ...upstream/master --stat tests/integration/
 tests/integration/test_merge_tree_s3/test.py  |   3 ---
 tests/integration/test_rename_column/test.py  |   1 -
 tests/integration/test_storage_s3/test.py     | 121 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/integration/test_ttl_replicated/test.py |   1 -
 4 files changed, 121 insertions(+), 5 deletions(-)
* upstream/master:
  Throw an error when a single parameter is passed to ReplicatedMergeTree instead of ignoring it
@azat
Copy link
Collaborator Author

azat commented Oct 2, 2020

Integration tests (release) — fail: 1, passed: 901, error: 0

test_distributed_over_live_view/test.py::TestLiveViewOverDistributedSuite::test_distributed_over_live_view_order_by_node[lv_over_distributed_table-node0] is flacky (fixed by increasing buffer size for now)
Apart from this, other tests are ok (except for Integration tests flaky check, but they are not related to this conversion)

@abyss7 abyss7 merged commit 9cb3c74 into ClickHouse:master Oct 2, 2020
@azat azat deleted the python3 branch October 2, 2020 19:24
traceon added a commit to traceon/ClickHouse that referenced this pull request Oct 2, 2020
* master: (279 commits)
  Convert to python3 (ClickHouse#15007)
  update en and ru docs: add 128/256 types (ClickHouse#15515)
  Drop unused code for numeric_limits<int128> in MergeTreeDataSelectExecutor (ClickHouse#15519)
  Fix comment
  More defaults on defaults
  Recursive defaults
  Fix bash
  Update ActionsVisitor.cpp
  Don't add memcpy for sanitizers builds
  Fix gRPC build scripts.
  Neutrialize thinlto's memcpy libcall gen.
  Revert "Add fno-builtin-memcpy"
  Throw an error when a single parameter is passed to ReplicatedMergeTree instead of ignoring it
  Update SECURITY.md
  Update tsan_suppressions.txt
  Update ActionsVisitor.cpp
  remove unrelated changes
  Update SECURITY.md
  fixes
  Update ActionsVisitor.cpp
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-build Pull request with build/testing/packaging improvement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Rewrite integration tests and clickhouse-test script to python 3.
8 participants