Allow ORC tests to run with wider range of timestamp input

[ttnghia]

This reverses the changes for orc_test.py#test_read_with_more_columns that have been made in #6286 due to a bug in cudf's orc reader. The bug is fixed by rapidsai/cudf#11586.

Closes #6312.

Depends on:

• Fix encode/decode of negative timestamps in ORC reader/writer rapidsai/cudf#11586
• Modify ORC reader timestamp parsing to match the apache reader behavior rapidsai/cudf#11699

CI will not pass until they are all merged and spark-rapids-jni is updated with it.

[jlowe]

Changes here look OK, although there are more tests to cleanup after the ORC timestamp fix. See get_orc_timestamp_gen in orc_test.py and _restricted_timestamp in hive_write_test.py. Can be done in a followup PR, but it seems very relevant to the theme of this PR.

[ttnghia]

Sure. I'll update all these related tests too.

[ttnghia]

I was working on the timestamp test in ORC as indicated by @jlowe and discovered a new bug with wrong read/write day number (rapidsai/cudf#11691). Thus, this PR should be merged as-is without the suggested changes. They will be addressed later when the issue rapidsai/cudf#11691 is fixed.

[ttnghia]

build

[ttnghia]

CI failed with unrelated tests:

[2022-09-13T05:34:02.621Z] - transpiler detects invalid cuDF patterns *** FAILED ***
[2022-09-13T05:34:02.621Z]   cuDF unexpectedly compiled expression: 	+|a (RegularExpressionTranspilerSuite.scala:62)
[2022-09-13T05:34:02.622Z] - Detect unsupported combinations of line anchors and \W and \D
[2022-09-13T05:34:02.622Z] - cuDF does not support choice with repetition
[2022-09-13T05:34:02.622Z] - cuDF does not support terms ending with line anchors on one side of a choice
[2022-09-13T05:34:02.622Z] - cuDF unsupported choice cases *** FAILED ***
[2022-09-13T05:34:02.622Z]   Expected exception ai.rapids.cudf.CudfException to be thrown, but no exception was thrown (RegularExpressionTranspilerSuite.scala:107)
[2022-09-13T05:34:02.622Z] - sanity check: choice edge case 2 *** FAILED ***
[2022-09-13T05:34:02.622Z]   Expected exception ai.rapids.cudf.CudfException to be thrown, but no exception was thrown (RegularExpressionTranspilerSuite.scala:115)

[ttnghia]

build

[ttnghia]

build

[jlowe]

They will be addressed later when the issue rapidsai/cudf#11691 is fixed.

It appears that new issue is actually an older issue related to the Gregorian calendar switch. That was avoided previously by not generating timestamps before the Gregorian calendar conversion, for example see https://github.com/NVIDIA/spark-rapids/blob/branch-21.08/integration_tests/src/main/python/orc_test.py#L51 from 22.08. I'm fine with not fixing it here, but if we leave it for a followup, we need an issue to track.

[ttnghia]

It appears that new issue is actually an older issue related to the Gregorian calendar switch. That was avoided previously by not generating timestamps before the Gregorian calendar conversion, for example see https://github.com/NVIDIA/spark-rapids/blob/branch-21.08/integration_tests/src/main/python/orc_test.py#L51 from 22.08. I'm fine with not fixing it here, but if we leave it for a followup, we need an issue to track.

Did we have any issue tracking that in either cudf or here?

[jlowe]

Did we have any issue tracking that in either cudf or here?

Yes, #131 for the ORC reader and #139 for the ORC writer. This limitation is also documented in the compatibility document.

[ttnghia]

I set orc_gen to use Timestamp after 1590 but still see failures thus this should be a new bug:

cpu = datetime.datetime(1647, 5, 20, 19, 25, 3, 638)
gpu = datetime.datetime(1647, 5, 20, 19, 25, 2, 638)

This time they are failures with second values. It seems rapidsai/cudf#11586 is not completely fixed.

[ttnghia]

Convert this into draft since there are still failures that may need cudf fix.

[jlowe]

build

[ttnghia]

build

[ttnghia]

build

[ttnghia]

build