[function](date function)add new date function 'last_day' by lsy3993 · Pull Request #13400 · apache/doris

lsy3993 · 2022-10-16T15:55:51Z

Proposed changes

Issue Number: close #13139

Problem summary

Describe your changes.

Checklist(Required)

Does it affect the original behavior:
- Yes
- No
- I don't know
Has unit tests been added:
- Yes
- No
- No Need
Has document been added or modified:
- Yes
- No
- No Need
Does it need to update dependencies:
- Yes
- No
Are there any changes that cannot be rolled back:
- Yes (If Yes, please explain WHY)
- No

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

stalary · 2022-10-17T00:16:38Z

be/src/exprs/timestamp_functions.cpp

+
+    DateTimeValue ts_value = DateTimeValue::from_datetime_val(ts_val);
+
+    bool is_leap_year = (ts_value.year() % 400 == 0) ||


You can use doris::is_leap(year)

Same as get_last_month_day?

stalary · 2022-10-17T00:23:30Z

be/src/vec/functions/function_timestamp.cpp

+    }
+
+    static int get_last_month_day(int year, int month) {
+        bool is_leap_year = (year % 400 == 0) || (year % 4 == 0 && year % 100 != 0);


Use doris::is_leap

Gabriel39 · 2022-10-17T10:53:05Z

regression-test/suites/query_p0/sql_functions/datetime_functions/test_date_function.groovy

    qt_sql """ select seconds_sub(test_time2,1) result from ${tableName}; """
+
+    // test last_day
+    sql """ SET enable_vectorized_engine = TRUE; """


also add a case for row-based engine

Gabriel39 · 2022-10-17T10:54:12Z

be/src/vec/functions/function_timestamp.cpp

+
+    String get_name() const override { return name; }
+
+    bool use_default_implementation_for_nulls() const override { return true; }


use_default_implementation_for_constants also need to override and set to true

Gabriel39 · 2022-10-17T10:55:48Z

be/src/vec/functions/function_timestamp.cpp

+    // input DateTime and Date, return Date
+    // input DateTimeV2 and DateV2, return DateV2
+    DataTypePtr get_return_type_impl(const ColumnsWithTypeAndName& arguments) const override {
+        if constexpr (std::is_same_v<DateType, DataTypeDateTime>) {


Use or to collapse these two conditions

Gabriel39 · 2022-10-17T10:56:45Z

be/src/vec/functions/function_timestamp.cpp

+    }
+
+    DataTypes get_variadic_argument_types_impl() const override {
+        if constexpr (std::is_same_v<DateType, DataTypeDateTime>) {


DataTypeDate is also possible as input argument

zhangstar333 · 2022-10-17T10:59:21Z

be/src/vec/functions/function_timestamp.cpp

+        } else {
+            // neither DateTime nor DateTimeV2/DateV2, return null
+            if constexpr (std::is_same_v<DateType, DataTypeDateTime>) {
+                res_column = ColumnInt64::create(input_rows_count);


this check seems to have been done at the above

you are right, I delete the 'else'

…mn (#13365) We should prevent insert while value overflow. 1. create table: `CREATE TABLE test_array_load_test_array_int_insert_db.test_array_load_test_array_int_insert_tb ( k1 int NULL, k2 array<int> NULL ) DUPLICATE KEY(k1) DISTRIBUTED BY HASH(k1) BUCKETS 5` 2. try insert data less than INT_MIN. `insert into test_array_load_test_array_int_insert_tb values (1005, [-2147483649])` Before this pr, the insert will success, but the value it not correct. Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>

Doris do not support explicitly cast NULL_TYPE to ANY type . ``` mysql> select cast(NULL as int); ERROR 1105 (HY000): errCode = 2, detailMessage = Invalid type cast of NULL from NULL_TYPE to INT ``` So we should also forbid user from casting NULL_TYPE to ARRAY type. This commit will produce the following effect: ``` mysql> select cast(NULL as array<int>); ERROR 1105 (HY000): errCode = 2, detailMessage = Invalid type cast of NULL from NULL_TYPE to ARRAY<INT(11)> ```

… table name (#13395)

…olumn (#13345) 1. this pr is used to update the json load docs for import data to array column. when we use json to import data to array column, the Rapidjson will cause precision problems. so we update the json-load docs to specify how to avoid these problems. Issue Number: #7570 Co-authored-by: hucheng01 <hucheng01@baidu.com>

…f en docs. (#13391) Co-authored-by: smallhibiscus <8449081280@qq.com>

…table for most cases (#13370) It is frustrated that a long running job fails due to small timeout. Actually, users do not expect a timeout for a log running job.

Previously, bthread_getspecific was called every time bthread local was used. In the test at #10823, it was found that frequent calls to bthread_getspecific had performance problems. So a cache is implemented on pthread local based on the btls key, but the btls key cannot correctly sense bthread switching. So, based on bthread_self to get the bthread id to implement the cache.

support `group_bitmap_xor` agg function

…13307) Issue Number: #12689

…ework (#13354) 1. Refactor the file reader creation in FileFactory, for simplicity. Previously, FileFactory had too many `create_file_reader` interfaces. Now unified into two categories: the interface used by the previous BrokerScanNode, and the interface used by the new FileScanNode. And separate the creation methods of readers that read `StreamLoadPipe` and other readers that read files. 2. Modify the StreamLoadPlanner on FE side to support using ExternalFileScanNode 3. Now for generic reader, the file reader will be created inside the reader, not passed from the outside. 4. Add some test cases for csv stream load, the behavior is same as the old broker scanner.

…ate/datetime type (#13407) This pr is used to expand the supported data type for array_min/array_max function. Before the change , the array_min/array_max function can't support the date/datetime type. After the change, array_min/array_max function can support the date/datetime type. Co-authored-by: hucheng01 <hucheng01@baidu.com>

…e, CHAR and VARCHAR type (#13311) 1. enable varchar/char type set min/max value. take first 8 chars as long, and convert to double. 2. fix bug when set min/max value for date and datav2

…13195) # Proposed changes This PR fixed lots of issues when building from source on macOS with Apple M1 chip. ## ATTENTION The job for supporting macOS with Apple M1 chip is too big and there are lots of unresolved issues during runtime: 1. Some errors with memory tracker occur when BE (RELEASE) starts. 2. Some UT cases fail. ... Temporarily, the following changes are made on macOS to start BE successfully. 1. Disable memory tracker. 2. Use tcmalloc instead of jemalloc. This PR kicks off the job. Guys who are interested in this job can continue to fix these runtime issues. ## Use case ```shell ./build.sh -j 8 --be --clean cd output/be/bin ulimit -n 60000 ./start_be.sh --daemon ``` ## Something else It takes around _**10+**_ minutes to build BE (with prebuilt third-parties) on macOS with M1 chip. We will improve the development experience on macOS greatly when we finish the adaptation job.

…13404) Use apache orc to read orc file, and convert ColumnVectorBatch to doris block.

Co-authored-by: yixiutt <yixiu@selectdb.com>

…13399) 1. Collect statistics by sampling sql-tasks. 2. Consolidate statistics SQL statements and remove redundant statements.

add nee date function 'last_day'

4a34729

github-actions bot added area/sql/function Issues or PRs related to the SQL functions area/vectorization kind/docs Categorizes issue or PR as related to documentation. kind/test labels Oct 16, 2022

stalary reviewed Oct 17, 2022

View reviewed changes

Gabriel39 reviewed Oct 17, 2022

View reviewed changes

zhangstar333 reviewed Oct 17, 2022

View reviewed changes

zxealous and others added 22 commits October 18, 2022 17:45

[test](cache)Add remote cache ut (#13377)

5bfdae7

[fix](memory) Fix USE_JEMALLOC=true UBSAN compilation error #13398

75381e9

[fix](regression-test) fix that multiple cases conflict with the same…

e2b5d60

… table name (#13395)

[thirdparty](zstd)update dist info and thirdparty change log (#13392)

170bc96

[Opt](vec) opt runtime filter for TPCH Q22 (#13339)

2a9ac94

[Feature](Retention) support retention function (#13056)

e2274dc

[typo](docs) Fixed thrift_client_timeout_ms's incorrect description o…

306a9d2

…f en docs. (#13391) Co-authored-by: smallhibiscus <8449081280@qq.com>

[improvement](config) let default value of alter and load timeout sui…

5bb770a

…table for most cases (#13370) It is frustrated that a long running job fails due to small timeout. Actually, users do not expect a timeout for a log running job.

[feature](agg) support group_bitmap_xor agg function. (#13287)

c1447c9

support `group_bitmap_xor` agg function

[Enhancement](array-type) Tuple is null predicate support array type (#…

3174aee

…13307) Issue Number: #12689

[Bug](funciton) fix repeat coredump when step is to long (#13408)

fa3d5d0

[feature](nereids) let minValue and maxValue in stats support for Dat…

d57a237

…e, CHAR and VARCHAR type (#13311) 1. enable varchar/char type set min/max value. take first 8 chars as long, and convert to double. 2. fix bug when set min/max value for date and datav2

[Improvement](join) optimize join probing phase (#13357)

db38f37

[Regression](javaudf) add regression test for javaudf (#13266)

d59da22

[feature-wip](multi-catalog) use apache orc reader to read orc file (#…

c08db67

…13404) Use apache orc to read orc file, and convert ColumnVectorBatch to doris block.

yixiutt and others added 7 commits October 18, 2022 17:45

[improvement](compaction) delete num based compaction policy (#13409)

37ebc7e

Co-authored-by: yixiutt <yixiu@selectdb.com>

[feature-wip](statistics) collect statistics by sampling sql-tasks (#…

8d8a8de

…13399) 1. Collect statistics by sampling sql-tasks. 2. Consolidate statistics SQL statements and remove redundant statements.

Merge remote-tracking branch 'upstream/master'

930b3b4

add new date function 'lsy_day'

be6ea8c

code format

458016f

bug fix

aacb88d

code optimize

f237a2d

lsy3993 closed this by deleting the head repository Oct 19, 2022


		DateTimeValue ts_value = DateTimeValue::from_datetime_val(ts_val);

		bool is_leap_year = (ts_value.year() % 400 == 0) \|\|


		String get_name() const override { return name; }

		bool use_default_implementation_for_nulls() const override { return true; }

Conversation

lsy3993 commented Oct 16, 2022

Proposed changes

Problem summary

Checklist(Required)

Further comments

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants