fix(utils): evaluate date parser multiple holiday results correctly #23685

arkid15r · 2023-04-13T22:52:36Z

SUMMARY

The purpose of this PR is fixing incorrect datetime_eval for holiday results containing more than one entry (even with observed=False flag set):

Bump holidays version range from ">=0.17.2, <0.18" to ">=0.23, <0.24"
Change holiday search result evaluation logic for multiple date cases
Change holiday lookup from icontains (default) to istartswith.
Add tests for the cases

TESTING INSTRUCTIONS

The added unit tests would fail with the currently used approach.

The described cases include:

multiple holidays per year (happens for Islamic calendar countries) e.g., "Eid al-Fitr" holidays for Saudi Arabia (JAN and DEC of 2000)
multiple holidays per lookup name, e.g., "Independence Day" for US years after 2021 (Independence Day, Juneteenth National Independence Day)

ADDITIONAL INFORMATION

Has associated issue: no issue, small PR
Required feature flags:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

- Bump holidays version family from ">=0.17.2, <0.18" to ">=0.22, <0.23" - Change holiday search result evaluation logic for multiple date cases - Change holiday lookup from `icontains` (default) to `istartswith`. - Add tests for described cases

github-actions

Congrats on making your first PR and thank you for contributing to Superset! 🎉 ❤️

We hope to see you in our Slack community too! Not signed up? Use our Slack App to self-register.

EugeneTorap · 2023-04-21T02:31:22Z

Hello @arkid15r! Thanks so much for this PR!
Can you use latest 0.23 version and also update the lib in requirements/base.txt?
@villebro @dpgaspar Pls, take a look at the PR.

arkid15r · 2023-04-21T15:14:37Z

Can you use latest 0.23 version

Hi Eugene,
that's absolutely doable. I used the current version range based on my understanding of consistency with other install_requires entries.

and also update the lib in requirements/base.txt?

I've done a couple of half-hearted attempts to update the requirements. It seems it hasn't been done for a while. Or maybe for some other reason a lot of dependencies are getting their version bumped when I run pip-compile-multi -P holidays.

git diff --stat
 requirements/base.txt        |  29 +++++++++++++++++-
 requirements/development.txt |   4 ---
 requirements/docker.txt      | 326 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------
 requirements/integration.txt |   8 ++---

I'd appreciate some guidance on how to properly resolve this sort of issues, e.g.:

Package greenlet was resolved to different versions in different environments: 1.1.3.post0 and 2.0.2
Traceback (most recent call last):
  File "/superset/venv/lib/python3.10/site-packages/pipcompilemulti/cli_v1.py", line 26, in cli
    recompile()
  File "/superset/venv/lib/python3.10/site-packages/pipcompilemulti/actions.py", line 31, in recompile
    compile_topologically(env_confs, deduplicator)
  File "/superset/venv/lib/python3.10/site-packages/pipcompilemulti/actions.py", line 38, in compile_topologically
    if env.maybe_create_lockfile():
  File "/superset/venv/lib/python3.10/site-packages/pipcompilemulti/environment.py", line 51, in maybe_create_lockfile
    self.create_lockfile()
  File "/superset/venv/lib/python3.10/site-packages/pipcompilemulti/environment.py", line 72, in create_lockfile
    self.fix_lockfile()
  File "/superset/venv/lib/python3.10/site-packages/pipcompilemulti/environment.py", line 134, in fix_lockfile
    sections = [
  File "/superset/venv/lib/python3.10/site-packages/pipcompilemulti/environment.py", line 135, in <listcomp>
    self.fix_pin(section)
  File "/superset/venv/lib/python3.10/site-packages/pipcompilemulti/environment.py", line 211, in fix_pin
    raise RuntimeError(
RuntimeError: Please add constraints for the package version listed above

@EugeneTorap, I appreciate you jumping in and tagging the right people as I was within a whisker of creating a post on Slack's #contributing channel asking for help.

villebro · 2023-04-21T15:27:44Z

Changes LGTM, but the deps will need to be fixed. @arkid15r do you need help updating the requirements?

arkid15r · 2023-04-21T15:30:35Z

Changes LGTM, but the deps will need to be fixed. @arkid15r do you need help updating the requirements?

Right, I do. The instructions are clear and easy to follow. The output is what bothers me -- a lot of unrelated [to this PR] changes.

villebro · 2023-04-21T15:34:46Z

Changes LGTM, but the deps will need to be fixed. @arkid15r do you need help updating the requirements?

Right, I do. The instructions are clear and easy to follow. The output is what bothers me -- a lot of unrelated [to this PR] changes.

@arkid15r I wrote before reading your thorough post properly, sorry for asking what you already had posted. I'm looking into this, but my env seems to be slightly messed up right now, and pip-compile-multi is misbehaving for me. I'll look at this more over the weekend, ok!

EugeneTorap · 2023-04-21T15:53:41Z

@arkid15r try to run:

pip-compile-multi --no-upgrade

And commit only base.txt file

arkid15r · 2023-04-21T17:27:20Z

@arkid15r try to run:
pip-compile-multi --no-upgrade
And commit only base.txt file

It worked, I guess the yesterday's requirements update might have helped. The workflows need to be approved/run one more time.

Thank you!

codecov · 2023-04-22T07:58:12Z

Codecov Report

Merging #23685 (1e1bc02) into master (b734a0f) will increase coverage by 0.10%.
The diff coverage is 80.49%.

❗ Current head 1e1bc02 differs from pull request most recent head 2e8e9b8. Consider uploading reports for the commit 2e8e9b8 to get more accurate results

@@            Coverage Diff             @@
##           master   #23685      +/-   ##
==========================================
+ Coverage   67.87%   67.98%   +0.10%     
==========================================
  Files        1925     1936      +11     
  Lines       74389    74914     +525     
  Branches     8108     8140      +32     
==========================================
+ Hits        50494    50928     +434     
- Misses      21818    21894      +76     
- Partials     2077     2092      +15

Flag	Coverage Δ
hive	`53.00% <62.22%> (+0.07%)`	⬆️
mysql	`78.79% <83.20%> (+0.05%)`	⬆️
postgres	`78.87% <83.20%> (+0.05%)`	⬆️
presto	`52.91% <62.22%> (+0.07%)`	⬆️
python	`82.66% <83.70%> (+0.01%)`	⬆️
sqlite	`77.38% <83.20%> (+0.06%)`	⬆️
unit	`52.80% <62.96%> (+0.07%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...chart-echarts/src/Timeseries/EchartsTimeseries.tsx	`0.00% <0.00%> (ø)`
...uperset/utils/dashboard_filter_scopes_converter.py	`15.78% <0.00%> (ø)`
superset/views/sql_lab/views.py	`63.81% <ø> (ø)`
superset/models/sql_lab.py	`78.40% <42.85%> (-1.11%)`	⬇️
superset-frontend/src/views/routes.tsx	`51.06% <50.00%> (-0.05%)`	⬇️
superset/db_engine_specs/ocient.py	`57.28% <57.28%> (ø)`
superset/dao/base.py	`92.85% <70.00%> (-2.60%)`	⬇️
...-frontend/src/pages/RowLevelSecurityList/index.tsx	`72.54% <72.54%> (ø)`
...rontend/src/features/rls/RowLevelSecurityModal.tsx	`73.39% <73.39%> (ø)`
...uperset/row_level_security/commands/bulk_delete.py	`87.50% <87.50%> (ø)`
... and 13 more

... and 1 file with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

EugeneTorap · 2023-04-22T09:09:31Z

@arkid15r Pls, fix the mypy issue in pre-commit CI

zhaoyongjie · 2023-04-22T10:11:50Z

tests/unit_tests/utils/date_parser_tests.py

+    result = datetime_eval(
+        "holiday('Juneteenth', datetime('2022-01-01T00:00:00'), 'US')"
+    )
+    expected = datetime(2022, 6, 19, 0, 0, 0)
+    assert result == expected
+
+    result = datetime_eval(
+        "holiday('Independence Day', datetime('2022-01-01T00:00:00'), 'US')"
+    )
+    expected = datetime(2022, 7, 4, 0, 0, 0)
+    assert result == expected
+


Hi there, the US holiday case was added to L273 so the duplicated test case is not necessary. Thanks for supporting more widely holiday types!

Hi @zhaoyongjie,

I read L273 as UK's (technically GB) test. And even if it was the US test I'd suggest to keep the L279-289 Juneteenth related tests as they specifically cover the wrong parser behaviour this PR trying to address.

Thanks for reviewing this!

arkid15r · 2023-04-24T16:41:31Z

@arkid15r Pls, fix the mypy issue in pre-commit CI

👍 I reverted the local mypy fixes, hope this fixes the issue.

…pache#23685)

pull-request-size bot added the size/S label Apr 13, 2023

arkid15r changed the title ~~fix (utils/date_parser): evaluate multiple holiday results correctly~~ fix(utils/date_parser): evaluate multiple holiday results correctly Apr 13, 2023

github-actions bot reviewed Apr 13, 2023

View reviewed changes

arkid15r changed the title ~~fix(utils/date_parser): evaluate multiple holiday results correctly~~ fix(utils): evaluate date parser multiple holiday results correctly Apr 18, 2023

Update dependencies.

d9b7c17

pull-request-size bot added size/M and removed size/S labels Apr 21, 2023

Sync with the master.

8b1e3a8

arkid15r requested review from villebro, geido, eschutho, rusackas, betodealmeida, nytai, mistercrunch, craig-rueda, john-bodley, kgabryje, jinghua-qa, dpgaspar and a team as code owners April 21, 2023 17:14

pull-request-size bot added size/XXL and removed size/M labels Apr 21, 2023

Merge branch 'master' into fix-date-eval-parser

0453db1

pull-request-size bot added size/M and removed size/XXL labels Apr 21, 2023

zhaoyongjie reviewed Apr 22, 2023

View reviewed changes

Revert local mypy fix.

2e8e9b8

villebro approved these changes Apr 26, 2023

View reviewed changes

villebro merged commit ff12654 into apache:master Apr 26, 2023

arkid15r deleted the fix-date-eval-parser branch April 26, 2023 13:59

sebastianliebscher pushed a commit to sebastianliebscher/superset that referenced this pull request Apr 28, 2023

fix(utils): evaluate date parser multiple holiday results correctly (a…

3c2e3e3

…pache#23685)

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 3.0.0 labels Mar 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(utils): evaluate date parser multiple holiday results correctly #23685

fix(utils): evaluate date parser multiple holiday results correctly #23685

arkid15r commented Apr 13, 2023 •

edited

Loading

github-actions bot left a comment

EugeneTorap commented Apr 21, 2023 •

edited

Loading

arkid15r commented Apr 21, 2023

villebro commented Apr 21, 2023

arkid15r commented Apr 21, 2023

villebro commented Apr 21, 2023

EugeneTorap commented Apr 21, 2023 •

edited

Loading

arkid15r commented Apr 21, 2023

codecov bot commented Apr 22, 2023 •

edited

Loading

EugeneTorap commented Apr 22, 2023

zhaoyongjie Apr 22, 2023

arkid15r Apr 24, 2023

arkid15r commented Apr 24, 2023

fix(utils): evaluate date parser multiple holiday results correctly #23685

fix(utils): evaluate date parser multiple holiday results correctly #23685

Conversation

arkid15r commented Apr 13, 2023 • edited Loading

SUMMARY

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

github-actions bot left a comment

Choose a reason for hiding this comment

EugeneTorap commented Apr 21, 2023 • edited Loading

arkid15r commented Apr 21, 2023

villebro commented Apr 21, 2023

arkid15r commented Apr 21, 2023

villebro commented Apr 21, 2023

EugeneTorap commented Apr 21, 2023 • edited Loading

arkid15r commented Apr 21, 2023

codecov bot commented Apr 22, 2023 • edited Loading

Codecov Report

EugeneTorap commented Apr 22, 2023

zhaoyongjie Apr 22, 2023

Choose a reason for hiding this comment

arkid15r Apr 24, 2023

Choose a reason for hiding this comment

arkid15r commented Apr 24, 2023

arkid15r commented Apr 13, 2023 •

edited

Loading

EugeneTorap commented Apr 21, 2023 •

edited

Loading

EugeneTorap commented Apr 21, 2023 •

edited

Loading

codecov bot commented Apr 22, 2023 •

edited

Loading