Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Date parsing improvements #424

Merged
merged 5 commits into from
Oct 5, 2022
Merged

Date parsing improvements #424

merged 5 commits into from
Oct 5, 2022

Conversation

hughess
Copy link
Member

@hughess hughess commented Oct 5, 2022

Summary

This PR implements improved date parsing.

Original Problems

  • Each database returns slightly different date strings
  • Each browser interprets and parses dates differently depending on the format of the string
  • Safari and Firefox return Invalid Date in several scenarios
  • Chart not able to use dates from Snowflake if they include a timestamp

Solution

  • Splits getParsedDate into a few functions and saves them in a module called dateParsing
    • standardizeDateString - Parses any date string and attempts to get it into a standard format YYYY-MM-DDTHH:MM:SS
    • standardizeDateColumn - Applies standardizeDateString to a full column in a data object
    • convertColumnToDate - Applies standardizeDateString to a full column plus new Date()
  • Adds date string standardization to Chart.svelte
    • ECharts is not able to read timestamps returned from Snowflake if they contain a timezone - standardizing the string fixes this
    • Due to an issue in the tidyJS library, we're not able to use Date objects in multi-series charts, so using standardized strings is the preferred solution until that issue is resolved
  • Closes Snowflake TIMESTAMP_TZ type not working as expected #388

Remaining Issues

This is likely not a good long-term solution - it relies on date strings being in a close enough format to be able to convert them to the standardized YYYY-MM-DDTHH:MM:SS format.

Potential future improvements:

  • Centralize the date parsing logic alongside the type inference performed by Evidence in dbcommons
    • This would run the logic once rather than having to run it from within each component where data is used
  • Return Date objects from each database connector rather than strings
    • This would eliminate the string parsing issues for most database connectors
    • Potential issue here would be how each connector converts to the Date object - there may be timezone conversion issues there as well, and it may be challenging to get all db connectors treating all dates in the same way
  • We will need to build solid support for timezones eventually - this implementation effectively ignores timezones in order to standardize all outputs, but this is a naive approach and while it should cover the majority of use cases, we should only use it in the short-term

@vercel
Copy link

vercel bot commented Oct 5, 2022

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Updated
evidence-docs ✅ Ready (Inspect) Visit Preview Oct 5, 2022 at 4:06PM (UTC)

@changeset-bot
Copy link

changeset-bot bot commented Oct 5, 2022

🦋 Changeset detected

Latest commit: fe65489

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@evidence-dev/components Patch
@evidence-dev/evidence Patch
evidence-test-environment Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Snowflake TIMESTAMP_TZ type not working as expected
1 participant