ARROW-13561: [C++] Implement week kernel that accepts WeekOptions #11026

rok · 2021-08-28T22:13:11Z

This is to resolve ARROW-13561.

github-actions · 2021-08-28T22:13:29Z

https://issues.apache.org/jira/browse/ARROW-13561

rok · 2021-08-28T22:15:10Z

Shall we also change DayOfWeekOptions to WeekOptions? It would be a better fit overall if this PR is merged.

lidavidm

So essentially, ISOWeek starts from 1, Week starts from 0, and neither use DayOfWeekOptions.week_start? In that case I would just make them two optionless kernels.

rok · 2021-08-30T16:40:14Z

So essentially, ISOWeek starts from 1, Week starts from 0, and neither use DayOfWeekOptions.week_start? In that case I would just make them two optionless kernels.

To get parity with MySQL DayOfWeekOptions.week_start should be settable either 1 or 7.

lidavidm · 2021-08-30T17:02:02Z

So essentially, ISOWeek starts from 1, Week starts from 0, and neither use DayOfWeekOptions.week_start? In that case I would just make them two optionless kernels.

To get parity with MySQL DayOfWeekOptions.week_start should be settable either 1 or 7.

Ah, ok. But unless I'm mistaken, week_start is not actually used by the week kernel.

rok · 2021-08-30T17:03:10Z

Ah, ok. But unless I'm mistaken, week_start is not actually used by the week kernel.

I'm just trying to understand if it does :)

rok · 2021-08-30T17:14:06Z

This evaluates to true in MySQL on sqlfiddle:

SELECT WEEK('2008-1-6', 1) != WEEK('2008-1-6', 4)

Mode	First day of week	Range	Week 1 is the first week
0	Sunday	0-53	with a Sunday in this year
1	Monday	0-53	with 4 or more days this year
2	Sunday	1-53	with a Sunday in this year
3	Monday	1-53	with 4 or more days this year
4	Sunday	0-53	with 4 or more days this year
5	Monday	0-53	with a Monday in this year
6	Sunday	1-53	with 4 or more days this year
7	Monday	1-53	with a Monday in this year

cpp/src/arrow/compute/kernels/scalar_temporal.cc

lidavidm · 2021-08-31T13:21:00Z

cpp/src/arrow/compute/kernels/scalar_temporal.cc

@@ -337,6 +339,41 @@ struct ISOWeek {
  Localizer localizer_;
 };


Given ISOWeek is a special case of Week now, maybe we should just have one kernel (and perhaps some conveniences, e.g. DayOfWeekOptions::IsoWeek())?

Oh yeah, that's a nice idea.
I'm still trying to exactly match WEEK from MySQL and once I do I'll look into this. Sorry for the WIP.

Oh, no worries. I can hold off looking at this until you're ready.

Early feedback is great though! :)

Ping on this? Since it seems the two kernels are very similar.

rok · 2021-09-01T15:47:54Z

@ianmcook This now covers MySQL WEEK modes 1, 3, 4 and 6 (week 1 is the first week with 4 or more days this year).
If we want to cover modes 0, 2, 5, and 7 (week 1 is the first week with a Monday/Sunday in this year) we could add another parameter e.g.: DayOfWeekOptions.week_start_in_year. Do you think we should?

Another thing to note here is that DayOfWeekOptions.one_based_numbering doesn't exactly mean we count from 0 or 1 but rather that a date from iso week 52 of previous year that is in this year will be counted as 0 instead of 52. It's a MySQL convention.

rok · 2021-09-01T16:31:41Z

we could to add another parameter e.g.: DayOfWeekOptions.week_start_in_year.

Perhaps DayOfWeekOptions.count_full_weeks is more idiomatic.

lidavidm · 2021-09-01T19:29:24Z

If the option doesn't apply to DayOfWeek, maybe it should be a separate WeekOptions in that case.

rok · 2021-09-01T19:42:53Z

So then we now have:
DayOfWeekOptions with week_start and one_based_numbering

We would need to add three binary parameters to match the 8 modes of MySQL:

WeekOptions with week_start (first day of week),count_from_zero (range), count_full_weeks (week 1 is ..)

Mode	First day of week	Range	Week 1 is the first week
0	Sunday	0-53	with a Sunday in this year
1	Monday	0-53	with 4 or more days this year
2	Sunday	1-53	with a Sunday in this year
3	Monday	1-53	with 4 or more days this year
4	Sunday	0-53	with 4 or more days this year
5	Monday	0-53	with a Monday in this year
6	Sunday	1-53	with 4 or more days this year
7	Monday	1-53	with a Monday in this year

rok · 2021-09-01T22:56:23Z

I tried adding WeekOptions with the following parameters:

  /// What day does the week start with (Monday=true, Sunday=false)
  bool week_starts_monday;
  /// Days in current year that fall into last years ISO week return week 0 if true
  bool count_from_zero;
  /// Is the first week fully in the the year or only its 4 or more days
  bool first_week_in_year;

I've also added tests with all the MySQL modes. Docs and the logic need some more work, but first: do we go this way or stick to iso week only?

rok · 2021-09-23T09:01:44Z

@lidavidm @pitrou I've removed the ISOWeek kernel and added iso_week and us_week functions that use week with specific options.

rok · 2021-09-23T12:26:33Z

R CI is reporting:

── Error (test-dplyr-lubridate.R:80:3): extract isoweek from timestamp ─────────
Error: Invalid: Attempted to initialize KernelState from null FunctionOptions
Backtrace:
     █
  1. ├─arrow:::expect_dplyr_equal(...) test-dplyr-lubridate.R:80:2
  2. │ ├─testthat::expect_warning(...) helper-expectation.R:101:4
  3. │ │ └─testthat:::quasi_capture(enquo(object), label, capture_warnings)
  4. │ │   ├─testthat:::.capture(...)
  5. │ │   │ └─base::withCallingHandlers(...)
  6. │ │   └─rlang::eval_bare(quo_get_expr(.quo), quo_get_env(.quo))
  7. │ └─rlang::eval_tidy(expr, rlang::new_data_mask(rlang::env(input = record_batch(tbl))))
  8. ├─input %>% mutate(x = isoweek(datetime)) %>% collect()
  9. ├─dplyr::collect(.)
 10. └─arrow:::collect.arrow_dplyr_query(.)
 11.   └─arrow:::do_exec_plan(x)
 12.     └─plan$Build(.data)
 13.       └─node$Project(projection)
 14.         ├─self$preserve_sort(ExecNode_Project(self, cols, names(cols)))
 15.         └─arrow:::ExecNode_Project(self, cols, names(cols))
── Error (test-dplyr-lubridate.R:205:3): extract isoweek from date ─────────────
Error: Invalid: Attempted to initialize KernelState from null FunctionOptions

I suppose we then need ISOWeek and USWeek declared with options in the API.

lidavidm · 2021-09-23T12:42:19Z

Yeah, we'd need a case here:

arrow/r/src/compute.cpp

Line 129 in 075c6c6

std::shared_ptr<arrow::compute::FunctionOptions> make_compute_options(

rok · 2021-09-23T13:31:57Z

Yeah, we'd need a case here:

arrow/r/src/compute.cpp

Line 129 in 075c6c6

std::shared_ptr<arrow::compute::FunctionOptions> make_compute_options(

@lidavidm oh, didn't realize it had to be there.
Maybe would be good to add R tests as well then?

lidavidm

Some nits on style/wording.

If you're up to add R tests, I think that would be appreciated, but otherwise we can put that in a follow-up JIRA (since there might be other work to bind the new kernels to dplyr features or whatnot).

r/src/compute.cpp

docs/source/cpp/compute.rst

lidavidm · 2021-09-23T13:27:32Z

cpp/src/arrow/compute/kernels/scalar_temporal.cc

@@ -1135,10 +1210,24 @@ void RegisterScalarTemporal(FunctionRegistry* registry) {
      "iso_year", {WithDates, WithTimestamps}, int64(), &iso_year_doc);
  DCHECK_OK(registry->AddFunction(std::move(iso_year)));

-  auto iso_week = MakeTemporal<ISOWeek, TemporalComponentExtract, Int64Type>(
-      "iso_week", {WithDates, WithTimestamps}, int64(), &iso_week_doc);
+  static const auto default_iso_week_options = WeekOptions(true, false, false);


Similarly here, can we put comments for the parameter literals? Or actually, it might be nice to define WeekOptions::IsoDefaults and WeekOptions::UsDefaults which could then be used here and in the R bindings.

Added WeekOptions::ISODefaults and WeekOptions::USDefaults.

lidavidm · 2021-09-23T13:29:28Z

cpp/src/arrow/compute/api_scalar.h

 /// \note API not yet finalized
-ARROW_EXPORT Result<Datum> ISOWeek(const Datum& values, ExecContext* ctx = NULLPTR);
+ARROW_EXPORT Result<Datum> Week(const Datum& values, WeekOptions options = WeekOptions(),
+                                ExecContext* ctx = NULLPTR);


Also add a helper for USWeek?

It appears to be missing its counterpart in api_scalar.cc.

cpp/src/arrow/compute/kernels/scalar_temporal.cc

lidavidm

Thanks, LGTM.

rok · 2021-09-23T14:57:22Z

Thanks for the review @lidavidm ! :)

Co-authored-by: Ian Cook <ianmcook@gmail.com>

Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

Co-authored-by: David Li <li.davidm96@gmail.com>

pitrou

Rebased, will merge

rok · 2021-09-27T13:51:46Z

Thanks @pitrou @ianmcook @jorisvandenbossche @lidavidm !

This is to resolve [ARROW-13561](https://issues.apache.org/jira/browse/ARROW-13561`). Closes apache#11026 from rok/ARROW-13561 Lead-authored-by: Rok <rok@mihevc.org> Co-authored-by: Rok Mihevc <rok@mihevc.org> Co-authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>

github-actions bot added Component: C++ Component: Python labels Aug 28, 2021

rok force-pushed the ARROW-13561 branch 2 times, most recently from f8d1e4c to 6997246 Compare August 29, 2021 11:38

github-actions bot added the Component: R label Aug 29, 2021

rok force-pushed the ARROW-13561 branch 4 times, most recently from 33b27e7 to 4a1b8a7 Compare August 30, 2021 10:59

lidavidm reviewed Aug 30, 2021

View reviewed changes

rok force-pushed the ARROW-13561 branch from 4a1b8a7 to 0c65cdd Compare August 30, 2021 16:29

rok force-pushed the ARROW-13561 branch 2 times, most recently from 18a2b49 to 69eed89 Compare August 30, 2021 22:09

lidavidm reviewed Aug 31, 2021

View reviewed changes

rok force-pushed the ARROW-13561 branch from 69eed89 to 9767363 Compare September 1, 2021 15:33

rok force-pushed the ARROW-13561 branch from 9767363 to 9065ddc Compare September 1, 2021 15:58

rok force-pushed the ARROW-13561 branch from 9065ddc to 0d86f13 Compare September 1, 2021 16:56

rok force-pushed the ARROW-13561 branch 2 times, most recently from ef87424 to 062e5f2 Compare September 2, 2021 10:02

rok force-pushed the ARROW-13561 branch from a49c555 to 00ddcf0 Compare September 23, 2021 11:59

rok force-pushed the ARROW-13561 branch from 00ddcf0 to 1537da4 Compare September 23, 2021 13:03

lidavidm reviewed Sep 23, 2021

View reviewed changes

rok force-pushed the ARROW-13561 branch 3 times, most recently from 1d4a49c to 4f68c1c Compare September 23, 2021 14:49

lidavidm approved these changes Sep 23, 2021

View reviewed changes

rok requested a review from pitrou September 23, 2021 16:04

rok and others added 11 commits September 27, 2021 14:56

ISOWeek kernel to Week kernel.

bbb9c9a

Matching MySQL.

9fbd488

Moving to WeekOptions.

4c3ed30

Update cpp/src/arrow/compute/api_scalar.h

996ed28

Co-authored-by: Ian Cook <ianmcook@gmail.com>

first_week_in_year -> first_week_is_fully_in_year

0523571

Update cpp/src/arrow/compute/api_scalar.h

c969a0d

Co-authored-by: Ian Cook <ianmcook@gmail.com>

Apply suggestions from code review

b593c31

Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

Apply suggestions from code review.

75ec4f7

Update docs/source/cpp/compute.rst

4f22a6c

Co-authored-by: David Li <li.davidm96@gmail.com>

Adding epiweek to R.

aba920f

Use keyword-only arguments

0db49ad

pitrou force-pushed the ARROW-13561 branch from 4f68c1c to 0db49ad Compare September 27, 2021 13:02

pitrou approved these changes Sep 27, 2021

View reviewed changes

pitrou closed this in af9a50a Sep 27, 2021

asfimport mentioned this pull request Sep 27, 2021

[C++] Implement week kernel that accepts WeekOptions #29210

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ARROW-13561: [C++] Implement week kernel that accepts WeekOptions #11026

ARROW-13561: [C++] Implement week kernel that accepts WeekOptions #11026

rok commented Aug 28, 2021

github-actions bot commented Aug 28, 2021

rok commented Aug 28, 2021 •

edited

Loading

lidavidm left a comment

rok commented Aug 30, 2021

lidavidm commented Aug 30, 2021

rok commented Aug 30, 2021

rok commented Aug 30, 2021

lidavidm Aug 31, 2021

rok Aug 31, 2021

lidavidm Aug 31, 2021

rok Aug 31, 2021 •

edited

Loading

lidavidm Sep 20, 2021

rok commented Sep 1, 2021 •

edited

Loading

rok commented Sep 1, 2021

lidavidm commented Sep 1, 2021

rok commented Sep 1, 2021

rok commented Sep 1, 2021 •

edited

Loading

rok commented Sep 23, 2021

rok commented Sep 23, 2021

lidavidm commented Sep 23, 2021

rok commented Sep 23, 2021

lidavidm left a comment

lidavidm Sep 23, 2021

rok Sep 23, 2021

lidavidm Sep 23, 2021

rok Sep 23, 2021

lidavidm Sep 23, 2021

rok Sep 23, 2021

lidavidm left a comment

rok commented Sep 23, 2021

pitrou left a comment

rok commented Sep 27, 2021

		@@ -337,6 +339,41 @@ struct ISOWeek {
		Localizer localizer_;
		};

ARROW-13561: [C++] Implement week kernel that accepts WeekOptions #11026

ARROW-13561: [C++] Implement week kernel that accepts WeekOptions #11026

Conversation

rok commented Aug 28, 2021

github-actions bot commented Aug 28, 2021

rok commented Aug 28, 2021 • edited Loading

lidavidm left a comment

Choose a reason for hiding this comment

rok commented Aug 30, 2021

lidavidm commented Aug 30, 2021

rok commented Aug 30, 2021

rok commented Aug 30, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rok Aug 31, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rok commented Sep 1, 2021 • edited Loading

rok commented Sep 1, 2021

lidavidm commented Sep 1, 2021

rok commented Sep 1, 2021

rok commented Sep 1, 2021 • edited Loading

rok commented Sep 23, 2021

rok commented Sep 23, 2021

lidavidm commented Sep 23, 2021

rok commented Sep 23, 2021

lidavidm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lidavidm left a comment

Choose a reason for hiding this comment

rok commented Sep 23, 2021

pitrou left a comment

Choose a reason for hiding this comment

rok commented Sep 27, 2021

rok commented Aug 28, 2021 •

edited

Loading

rok Aug 31, 2021 •

edited

Loading

rok commented Sep 1, 2021 •

edited

Loading

rok commented Sep 1, 2021 •

edited

Loading