Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new functions to format datetime in joda datetime style #43818

Merged
merged 9 commits into from
Jan 2, 2023

Conversation

taiyang-li
Copy link
Contributor

Changelog category (leave one):

  • New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

format datetime in joda datetime style. Refer to https://joda-time.sourceforge.net/apidocs/org/joda/time/format/DateTimeFormat.html

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

@robot-ch-test-poll robot-ch-test-poll added the pr-feature Pull request with new product feature label Nov 30, 2022
@taiyang-li taiyang-li marked this pull request as ready for review November 30, 2022 10:50
@taiyang-li taiyang-li changed the title format datetime in joda datetime style Add new functions to format datetime in joda datetime style Nov 30, 2022
@taiyang-li
Copy link
Contributor Author

 :) SELECT formatDateTimeWithJodaPattern(toDateTime('2010-01-04 12:34:56'), 'yyyy-MM-dd HH:mm:ss')

SELECT formatDateTimeWithJodaPattern(toDateTime('2010-01-04 12:34:56'), 'yyyy-MM-dd HH:mm:ss')

Query id: 7bacef4a-b2ed-46fa-97c2-eeff2aa75d78

┌─formatDateTimeWithJodaPattern(toDateTime('2010-01-04 12:34:56'), 'yyyy-MM-dd HH:mm:ss')─┐
│ 2010-01-04 12:34:56                                                                     │
└─────────────────────────────────────────────────────────────────────────────────────────┘

1 rows in set. Elapsed: 0.002 sec. 

 :) SELECT fromUnixTimestampWithJodaPattern(1669804872, 'yyyy-MM-dd HH:mm:ss', 'UTC');

SELECT fromUnixTimestampWithJodaPattern(1669804872, 'yyyy-MM-dd HH:mm:ss', 'UTC')

Query id: bd710c2b-94f2-4107-b256-a94c563b0322

┌─fromUnixTimestampWithJodaPattern(1669804872, 'yyyy-MM-dd HH:mm:ss', 'UTC')─┐
│ 2022-11-30 10:41:12                                                        │
└────────────────────────────────────────────────────────────────────────────┘

1 rows in set. Elapsed: 0.002 sec. 

src/Functions/formatDateTime.cpp Show resolved Hide resolved
src/Functions/formatDateTime.cpp Outdated Show resolved Hide resolved
src/Functions/formatDateTime.cpp Outdated Show resolved Hide resolved
@taiyang-li
Copy link
Contributor Author

The new adding function fromUnixTimestampWithJodaPattern is 2.8x faster than spark function FROM_UNIXTIME

SELECT count(1)
FROM
(

    SELECT fromUnixTimestampWithJodaPattern(1483200000 + number ,   'yyyy-MM-dd HH:mm:ss')
    FROM numbers(10000000)
)

Costs 1.6s 


SELECT count(1)
FROM
(

    SELECT FROM_UNIXTIME(1483200000 + id,   'yyyy-MM-dd HH:mm:ss')
    FROM range(10000000)
)

Costs 4.482 s 

@rschu1ze rschu1ze self-assigned this Dec 2, 2022
docs/en/sql-reference/functions/date-time-functions.md Outdated Show resolved Hide resolved
docs/en/sql-reference/functions/date-time-functions.md Outdated Show resolved Hide resolved
docs/en/sql-reference/functions/date-time-functions.md Outdated Show resolved Hide resolved
docs/en/sql-reference/functions/date-time-functions.md Outdated Show resolved Hide resolved
CMakeLists.txt Outdated Show resolved Hide resolved
src/Functions/formatDateTime.cpp Outdated Show resolved Hide resolved
src/Functions/formatDateTime.cpp Show resolved Hide resolved
src/Functions/formatDateTime.cpp Outdated Show resolved Hide resolved
src/Functions/formatDateTime.cpp Outdated Show resolved Hide resolved
src/Functions/formatDateTime.cpp Outdated Show resolved Hide resolved
@taiyang-li
Copy link
Contributor Author

@rschu1ze Thanks very much for your review. Let's wait for the CI/CD pipelines.

Copy link
Member

@rschu1ze rschu1ze left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@taiyang-li Thanks for your fixes. The PR looks overall good and is (almost) ready to merge. When I did the first round of review over the weekend, I skipped the tests (sorry). I now had a look at them, so please find some further comments included. It is mostly minor stuff, so we should be able to merge this PR afterwards.

src/Functions/formatDateTime.cpp Outdated Show resolved Hide resolved
src/Functions/formatDateTime.cpp Outdated Show resolved Hide resolved
src/Functions/formatDateTime.cpp Outdated Show resolved Hide resolved
src/Functions/formatDateTime.cpp Outdated Show resolved Hide resolved
src/Functions/formatDateTime.cpp Outdated Show resolved Hide resolved
src/Functions/formatDateTime.cpp Outdated Show resolved Hide resolved
src/Functions/formatDateTime.cpp Outdated Show resolved Hide resolved
src/Functions/formatDateTime.cpp Outdated Show resolved Hide resolved
@alexey-milovidov alexey-milovidov added the can be tested Allows running workflows for external contributors label Dec 12, 2022
@rschu1ze
Copy link
Member

LGTM.

Style check (rightfully) complains that "formatDateTime.cpp" prints to stdout. Once you removed that (I don't have permissions to fix it by myself) and test are green, I will merge. Thanks!

@robot-ch-test-poll robot-ch-test-poll added the submodule changed At least one submodule changed in this PR. label Dec 12, 2022
@robot-ch-test-poll1 robot-ch-test-poll1 removed the submodule changed At least one submodule changed in this PR. label Dec 13, 2022
@rschu1ze
Copy link
Member

@taiyang-li About the failure of fasttest, please see my earlier comment #43818 (comment).

I also wonder why commit eb9d496 is needed? Submodules updated in other PRs are automatically obtained by merges from master. To get rid of this commit, you could squash your (and only your) commits, rebase them on top of the current master branch state and force-push to branch "improve_from_unixtime".

@taiyang-li
Copy link
Contributor Author

image

@rschu1ze We'd better add label 'pr-performance'. Because this pr may affect function formatDateTime

@rschu1ze rschu1ze added the pr-performance Pull request with some performance improvements label Dec 15, 2022
@rschu1ze rschu1ze added the pr-performance-tests Forces full performance tests in CI label Dec 15, 2022
@robot-ch-test-poll robot-ch-test-poll removed the pr-performance Pull request with some performance improvements label Dec 16, 2022
@rschu1ze
Copy link
Member

Functional tests look good.
Couldn't find a regression in performance, in fact one query involving formatDateTime actually became faster.

Let me merge once all builds/tests are done (please don't merge from master in the meantime).

@taiyang-li
Copy link
Contributor Author

The failed tests seem not related to this PR, see similar failed tests in #43239

@rschu1ze
Copy link
Member

rschu1ze commented Dec 20, 2022

Clang-tidy complains ... because of that the merge button is greyed out for me, sorry.
@taiyang-li Would you like to address the finding? Thanks!

Dec 20 04:01:49 FAILED: src/Functions/CMakeFiles/clickhouse_functions_obj.dir/formatDateTime.cpp.o 
Dec 20 04:01:49 /usr/bin/cmake -E __run_co_compile --launcher="prlimit;--as=10000000000;--data=5000000000;--cpu=1000;/usr/bin/ccache" --tidy="/usr/bin/clang-tidy-cache;/usr/bin/clang-tidy-15;--extra-arg-before=--driver-mode=g++" --source=/build/src/Functions/formatDateTime.cpp -- /usr/bin/clang++-15 --target=x86_64-linux-gnu --sysroot=/build/cmake/linux/../../contrib/sysroot/linux-x86_64/x86_64-linux-gnu/libc -DANNOYLIB_MULTITHREADED_BUILD -DAWS_SDK_VERSION_MAJOR=1 -DAWS_SDK_VERSION_MINOR=7 -DAWS_SDK_VERSION_PATCH=231 -DBOOST_ASIO_HAS_STD_INVOKE_RESULT=1 -DBOOST_ASIO_STANDALONE=1 -DCARES_STATICLIB -DENABLE_ANNOY -DENABLE_MULTITARGET_CODE=1 -DENABLE_OPENSSL_ENCRYPTION -DHAS_RESERVED_IDENTIFIER -DHAS_SUGGEST_DESTRUCTOR_OVERRIDE -DHAS_SUGGEST_OVERRIDE -DPOCO_ENABLE_CPP11 -DPOCO_HAVE_FD_EPOLL -DPOCO_OS_FAMILY_UNIX -DSTD_EXCEPTION_HAS_STACK_TRACE=1 -DUNALIGNED_OK -DWITH_COVERAGE=0 -DWITH_GZFILEOP -DX86_64 -DXXH_INLINE_ALL -DZLIB_COMPAT -D_LIBCPP_ENABLE_THREAD_SAFETY_ANNOTATIONS -I/build/build_docker/includes/configs -I/build/base/glibc-compatibility/memcpy -I/build/contrib/libfarmhash -I/build/src -I/build/build_docker/src -I/build/build_docker/src/Core/include -I/build/base/base/.. -I/build/build_docker/base/base/.. -I/build/contrib/cctz/include -I/build/base/pcg-random/. -I/build/contrib/miniselect/include -I/build/contrib/zstd/lib -I/build/src/Common/mysqlxx/. -I/build/contrib/libmetrohash/src -I/build/contrib/murmurhash/include -I/build/rust/BLAKE3/include -isystem /build/contrib/llvm-project/libcxx/include -isystem /build/contrib/llvm-project/libcxxabi/include -isystem /build/contrib/libunwind/include -isystem /build/contrib/wyhash -isystem /build/contrib/cityhash102/include -isystem /build/contrib/abseil-cpp -isystem /build/contrib/boost -isystem /build/contrib/poco/Net/include -isystem /build/contrib/poco/Foundation/include -isystem /build/contrib/poco/NetSSL_OpenSSL/include -isystem /build/contrib/poco/Crypto/include -isystem /build/contrib/boringssl/include -isystem /build/contrib/poco/Util/include -isystem /build/contrib/poco/JSON/include -isystem /build/contrib/poco/XML/include -isystem /build/contrib/replxx/include -isystem /build/contrib/fmtlib-cmake/../fmtlib/include -isystem /build/contrib/magic_enum/include -isystem /build/contrib/double-conversion -isystem /build/contrib/dragonbox/include -isystem /build/contrib/re2 -isystem /build/build_docker/contrib/re2-cmake -isystem /build/contrib/zlib-ng -isystem /build/build_docker/contrib/zlib-ng-cmake -isystem /build/contrib/pdqsort -isystem /build/contrib/xz/src/liblzma/api -isystem /build/contrib/aws-c-common/include -isystem /build/contrib/aws-c-event-stream/include -isystem /build/contrib/aws/aws-cpp-sdk-s3/include -isystem /build/contrib/aws/aws-cpp-sdk-core/include -isystem /build/build_docker/contrib/aws-s3-cmake/include -isystem /build/contrib/snappy -isystem /build/build_docker/contrib/snappy-cmake -isystem /build/contrib/msgpack-c/include -isystem /build/contrib/fast_float/include -isystem /build/contrib/consistent-hashing -isystem /build/build_docker/contrib/orc/c++/include -isystem /build/contrib/llvm-project/llvm/include -isystem /build/build_docker/contrib/llvm-project/llvm/include -isystem /build/contrib/croaring/cpp -isystem /build/contrib/croaring/include -isystem /build/contrib/NuRaft/include -isystem /build/contrib/poco/MongoDB/include -isystem /build/build_docker/contrib/mariadb-connector-c-cmake/include-public -isystem /build/contrib/mariadb-connector-c/include -isystem /build/contrib/mariadb-connector-c/libmariadb -isystem /build/build_docker/src/Server/grpc_protos -isystem /build/contrib/grpc/include -isystem /build/contrib/c-ares/src/lib -isystem /build/contrib/c-ares/include -isystem /build/contrib/c-ares-cmake/linux -isystem /build/contrib/protobuf/src -isystem /build/contrib/s2geometry/src -isystem /build/contrib/AMQP-CPP/include -isystem /build/contrib/AMQP-CPP -isystem /build/contrib/libuv/include -isystem /build/contrib/sqlite-amalgamation -isystem /build/contrib/rocksdb/include -isystem /build/contrib/libpqxx/include -isystem /build/contrib/libpq -isystem /build/contrib/libpq/include -isystem /build/contrib/libstemmer_c/include -isystem /build/contrib/wordnet-blast -isystem /build/contrib/lemmagen-c/include -isystem /build/contrib/annoy/src -isystem /build/contrib/hashidsxx -isystem /build/contrib/morton-nd/include -isystem /build/contrib/libdivide/. -isystem /build/contrib/xxHash -isystem /build/contrib/icu/icu4c/source/i18n -isystem /build/contrib/icu/icu4c/source/common -isystem /build/contrib/fastops -isystem /build/contrib/base64 -isystem /build/contrib/cld2/public -isystem /build/contrib/h3/src/h3lib/include -isystem /build/build_docker/contrib/h3/src/h3lib/include -isystem /build/contrib/vectorscan/src -isystem /build/contrib/simdjson/include -isystem /build/contrib/rapidjson/include --gcc-toolchain=/build/cmake/linux/../../contrib/sysroot/linux-x86_64 -std=c++20 -fdiagnostics-color=always -Xclang -fuse-ctor-homing -fsized-deallocation  -UNDEBUG -gdwarf-aranges -pipe -mssse3 -msse4.1 -msse4.2 -mpclmul -mpopcnt -fasynchronous-unwind-tables -falign-functions=32 -mbranches-within-32B-boundaries -fdiagnostics-absolute-paths -fstrict-vtable-pointers -fexperimental-new-pass-manager -Wall -Wextra -Weverything -Wpedantic -Wno-zero-length-array -Wno-c++98-compat-pedantic -Wno-c++98-compat -Wno-c++20-compat -Wno-sign-conversion -Wno-implicit-int-conversion -Wno-implicit-int-float-conversion -Wno-ctad-maybe-unsupported -Wno-disabled-macro-expansion -Wno-documentation-unknown-command -Wno-double-promotion -Wno-exit-time-destructors -Wno-float-equal -Wno-global-constructors -Wno-missing-prototypes -Wno-missing-variable-declarations -Wno-padded -Wno-switch-enum -Wno-undefined-func-template -Wno-unused-template -Wno-vla -Wno-weak-template-vtables -Wno-weak-vtables -Wno-thread-safety-negative -g -O0 -g -gdwarf-4 -fno-inline  -D_LIBCPP_DEBUG=0   -D OS_LINUX -I/build/base -I/build/contrib/magic_enum/include -include /build/src/Core/iostream_debug_helpers.h -Werror -nostdinc++ -std=gnu++20 -MD -MT src/Functions/CMakeFiles/clickhouse_functions_obj.dir/formatDateTime.cpp.o -MF src/Functions/CMakeFiles/clickhouse_functions_obj.dir/formatDateTime.cpp.o.d -o src/Functions/CMakeFiles/clickhouse_functions_obj.dir/formatDateTime.cpp.o -c /build/src/Functions/formatDateTime.cpp
Dec 20 04:01:49 /build/src/Functions/formatDateTime.cpp:282:44: error: Division by zero [clang-analyzer-core.DivideZero,-warnings-as-errors]
Dec 20 04:01:49                 writeNumber2(dest + pos, n / w);
Dec 20 04:01:49                                            ^
Dec 20 04:01:49 /build/src/Functions/formatDateTime.cpp:585:20: note: Calling 'Action::writeNumberWithPadding'
Dec 20 04:01:49             return writeNumberWithPadding(dest, second_of_minute, min_represent_digits);
Dec 20 04:01:49                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dec 20 04:01:49 /build/src/Functions/formatDateTime.cpp:252:13: note: Loop condition is true.  Entering loop body
Dec 20 04:01:49             while (n)
Dec 20 04:01:49             ^
Dec 20 04:01:49 /build/src/Functions/formatDateTime.cpp:252:13: note: Loop condition is false. Execution continues on line 260
Dec 20 04:01:49 /build/src/Functions/formatDateTime.cpp:262:27: note: 'is_signed_v' is false
Dec 20 04:01:49             if constexpr (is_signed_v<T>)
Dec 20 04:01:49                           ^~~~~~~~~~~~~~
Dec 20 04:01:49 /build/src/Functions/formatDateTime.cpp:262:13: note: Taking false branch
Dec 20 04:01:49             if constexpr (is_signed_v<T>)
Dec 20 04:01:49             ^
Dec 20 04:01:49 /build/src/Functions/formatDateTime.cpp:271:17: note: Assuming 'min_digits' is <= 'digits'
Dec 20 04:01:49             if (min_digits > digits)
Dec 20 04:01:49                 ^~~~~~~~~~~~~~~~~~~
Dec 20 04:01:49 /build/src/Functions/formatDateTime.cpp:271:13: note: Taking false branch
Dec 20 04:01:49             if (min_digits > digits)
Dec 20 04:01:49             ^
Dec 20 04:01:49 /build/src/Functions/formatDateTime.cpp:278:20: note: 'n' is >= 10
Dec 20 04:01:49             while (n >= 10)
Dec 20 04:01:49                    ^
Dec 20 04:01:49 /build/src/Functions/formatDateTime.cpp:278:13: note: Loop condition is true.  Entering loop body
Dec 20 04:01:49             while (n >= 10)
Dec 20 04:01:49             ^
Dec 20 04:01:49 /build/src/Functions/formatDateTime.cpp:280:17: note: The value 0 is assigned to 'w'
Dec 20 04:01:49                 w /= 100;
Dec 20 04:01:49                 ^~~~~~~~
Dec 20 04:01:49 /build/src/Functions/formatDateTime.cpp:282:44: note: Division by zero
Dec 20 04:01:49                 writeNumber2(dest + pos, n / w);
Dec 20 04:01:49                                          ~~^~~
Dec 20 04:01:49 188136 warnings generated.
Dec 20 04:01:49 Suppressed 188958 warnings (188135 in non-user code, 823 NOLINT).
Dec 20 04:01:49 Use -header-filter=.* to display errors from all non-system headers. Use -system-headers to display errors from system headers as well.
Dec 20 04:01:49 1 warning treated as error

@taiyang-li
Copy link
Contributor Author

The failed tests seem not related to this PR.

@rschu1ze
Copy link
Member

Yes, the test failures are unrelated. However, #44060 was a little bit quicker to merge and created merge conflicts, sorry. Would you like to resolve these first? Thanks!

@taiyang-li
Copy link
Contributor Author

Sorry, I had a fever in recent days. I will continue this work when I'm fine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
can be tested Allows running workflows for external contributors pr-feature Pull request with new product feature pr-performance-tests Forces full performance tests in CI
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants