Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Time functions assume UTC and don't allow for RFC 3339 offsets #631

Closed
danjenson opened this issue Jul 19, 2020 · 14 comments · Fixed by #814
Closed

Time functions assume UTC and don't allow for RFC 3339 offsets #631

danjenson opened this issue Jul 19, 2020 · 14 comments · Fixed by #814

Comments

@danjenson
Copy link

--change-newer-than assumes given time is in UTC, not local timezone. Furthermore, offsets like '2020-07-19T00:38:00-07:00' throw errors.

fd version 8.1.1 on Arch Linux

@danjenson danjenson added the bug label Jul 19, 2020
@sharkdp
Copy link
Owner

sharkdp commented Sep 1, 2020

Thank you for reporting this. That should definitely be fixed.

Is there an easy way to reproduce this bug?

@gaganis
Copy link

gaganis commented Sep 14, 2020

I am also experiencing this problem.

I have been looking at the source code and I have found that [ humantime::parse_rfc3339_weak] only supports UTC and timezone conversions are not supported(https://docs.rs/humantime/2.0.1/humantime/fn.parse_rfc3339_weak.html) which is used to parse the date in

.or_else(|_| humantime::parse_rfc3339_weak(s))

Was surprizing!

I have done a small search and the chrono library seems to be better suited for our usecase.

@sharkdp Would you be interested in a PR with an attempt to solve this?

@sharkdp
Copy link
Owner

sharkdp commented Oct 5, 2020

Thank you for the feedback. Yes, I would be interested in a fix for this!

But let's please make sure that:

  • we don't break any existing functionality (it should hopefully be well tested)
  • we add regression tests for the new expected behavior.

Also, it would be great if someone could post a simple way to reproduce this error. I haven't looked into it in detail, so far.

@jacobmischka
Copy link
Contributor

jacobmischka commented Oct 25, 2020

The humantime docs state that indeed only UTC is supported: https://docs.rs/humantime/2.0.1/humantime/fn.parse_rfc3339_weak.html

It's pretty straightforward to reproduce, here's an example using this repo that I just cloned:

$ ll
.rw-r--r-- mischka users  2.3 KB Sun Oct 25 04:55:02 2020   appveyor.yml   
.rw-r--r-- mischka users  792 B  Sun Oct 25 04:55:02 2020   build.rs       
.rw-r--r-- mischka users 14.9 KB Sun Oct 25 04:55:18 2020   Cargo.lock     
.rw-r--r-- mischka users  1.6 KB Sun Oct 25 04:55:02 2020   Cargo.toml     
.rw-r--r-- mischka users 19.3 KB Sun Oct 25 04:55:02 2020   CHANGELOG.md   
drwxr-xr-x mischka users    4 KB Sun Oct 25 04:55:02 2020   ci             
drwxr-xr-x mischka users    4 KB Sun Oct 25 04:55:02 2020   contrib        
.rw-r--r-- mischka users  877 B  Sun Oct 25 04:55:02 2020   CONTRIBUTING.md
drwxr-xr-x mischka users    4 KB Sun Oct 25 04:55:02 2020   doc            
.rw-r--r-- mischka users 10.6 KB Sun Oct 25 04:55:02 2020   LICENSE-APACHE 
.rw-r--r-- mischka users  1.1 KB Sun Oct 25 04:55:02 2020   LICENSE-MIT    
.rw-r--r-- mischka users 19.6 KB Sun Oct 25 04:55:02 2020   README.md      
drwxr-xr-x mischka users    4 KB Sun Oct 25 04:55:02 2020   src            
drwxr-xr-x mischka users    4 KB Sun Oct 25 04:55:24 2020   target         
drwxr-xr-x mischka users    4 KB Sun Oct 25 04:55:02 2020   tests     
     
$ fd --change-newer-than 2020-10-25T04:55:03-07:00
[fd error]: '2020-10-25T04:55:03-07:00' is not a valid date or duration. See 'fd --help'.

$ fd -d 1 --change-newer-than '2020-10-25 04:55:03'
CHANGELOG.md
CONTRIBUTING.md
Cargo.lock
Cargo.toml
LICENSE-APACHE
LICENSE-MIT
README.md
appveyor.yml
build.rs
ci
contrib
doc
src
tests

@gaganis
Copy link

gaganis commented Oct 25, 2020

Regarding the reproduction

For me there are actually two problems that need to addressed:

  • When a user adds a time without an offset/timezone it is reasonable for him to assume that the time he entered would be interpreted according to his local system timezone. This is especially dangerous and important to me because it can produce subtle unexpected results. This is actually the issue that I have experienced.
  • The inability to influence the interpretation of time by adding an explicit offset or timezone. If someone deduced the first problem, then he could use this as workaround to pass his local time with an offset and not have to do a calculation manually to UTC.

The second problem is a actually ease to reproduce as @jacobmischka has demonstrated above. Thanks @jacobmischka for sharing!

The first problem is more difficult to reproduce in an automated regression test. They would depend on a the locally setup timezone which is a global system property. The system that is running the tests should be setup in a non UTC timezone & the tests should be able to know or deduce what the setup timezone is.

Regarding the implementation

I did an effort to find a library that would properly parse local times to solve the first problem. My best result was to use chrono 0.4. I did my tests when we were in EEST but it would assume EET and so still interpreted the local times wrongly. (Just today we switched to EET. If I retested now it would work OK :) )

If anyone knows how to successfully parse time strings appropriately for local times in Rust it would be really helpful if he shared his experience.

@jacobmischka
Copy link
Contributor

I've started this here: https://github.com/jacobmischka/fd/tree/use-local-time-filters

This addresses the first problem, but not the second. I'm not sure if it's worth migrating away from humantime, the advanced "30 minutes ago" style parsing is pretty nice. Instead, I'm currently converting its returned UTC SystemTime to a Local one using chrono.

@gaganis
Copy link

gaganis commented Oct 25, 2020

@jacobmischka Have you tried your solution while your system was in Summer time?

Maybe chrono is trying to be progressive and has already adopted EUs decision to abolish Summer Time :)

@jacobmischka
Copy link
Contributor

I haven't done extensive testing yet, no, but I don't believe it should matter since chrono is comparing everything to the current timezone offset at runtime.

I hope to think of a way to implement some decent tests today at some point.

@piechologist
Copy link

I got burnt by this issue today. How about adding a hint to the man page?

You can easily convert your timestamp to UTC before passing it to fd but you might have a problem discovering fd's requirement. Especially when some of your machines run on UTC and others on local time.

So, I think just mentioning UTC on the man page would be a sufficient fix.

@zachriggle
Copy link

Definitely agreed that this should be explicitly in the --help and man page

@sharkdp
Copy link
Owner

sharkdp commented Aug 8, 2021

@jacobmischka Would you be interested in continuing your work on this? It looks promising!

If not, would you consider opening a PR nevertheless, so somebody else can work on it? (only if you agree to share your work, of course)

@jacobmischka
Copy link
Contributor

Oh sure, it kind of fell off my radar. I'll try to pick it back up soon!

@sharkdp sharkdp added this to the fd 9 milestone Aug 8, 2021
jacobmischka added a commit to jacobmischka/fd that referenced this issue Aug 8, 2021
This patch uses Chrono for explicit date or datetime parsing, only using
humantime for its relative time parsing. The following formats are accepted:

1. Full RFC3339 parsing, requiring an explicit timezone
2. `YY-MM-DD`, defaulting to time `00:00:00` for the given date in the
   local time zone
3. `YY-MM-DD HH:MM:SS` in the local time zone

Fixes sharkdp#631
@jacobmischka
Copy link
Contributor

I did end up doing what @gaganis suggested and used chrono to parse the explicit dates. I don't know why past me didn't think that was a good idea, to be honest. This does change the default time function behavior to use local times by default, so it's a breaking change. Comments and suggestions welcome!

@tmccombs
Copy link
Collaborator

tmccombs commented Aug 9, 2021

This does change the default time function behavior to use local times by default, so it's a breaking change

Personally, I think it is worth it. The current behavior is pretty surprising, and using UTC is straightforward if that is desired.

jacobmischka added a commit to jacobmischka/fd that referenced this issue Aug 9, 2021
This patch uses Chrono for explicit date or datetime parsing, only using
humantime for its relative time parsing. The following formats are accepted:

1. Full RFC3339 parsing, requiring an explicit timezone
2. `YY-MM-DD`, defaulting to time `00:00:00` for the given date in the
   local time zone
3. `YY-MM-DD HH:MM:SS` in the local time zone

Fixes sharkdp#631
jacobmischka added a commit to jacobmischka/fd that referenced this issue Aug 9, 2021
This patch uses Chrono for explicit date or datetime parsing, only using
humantime for its relative time parsing. The following formats are accepted:

1. Full RFC3339 parsing, requiring an explicit timezone
2. `YY-MM-DD`, defaulting to time `00:00:00` for the given date in the
   local time zone
3. `YY-MM-DD HH:MM:SS` in the local time zone

Fixes sharkdp#631, sharkdp#794
jacobmischka added a commit to jacobmischka/fd that referenced this issue Aug 9, 2021
This patch uses Chrono for explicit date or datetime parsing, only using
humantime for its relative time parsing. The following formats are accepted:

1. Full RFC3339 parsing, requiring an explicit timezone
2. `YY-MM-DD`, defaulting to time `00:00:00` for the given date in the
   local time zone
3. `YY-MM-DD HH:MM:SS` in the local time zone

Fixes sharkdp#631, sharkdp#794
sharkdp pushed a commit that referenced this issue Aug 9, 2021
This patch uses Chrono for explicit date or datetime parsing, only using
humantime for its relative time parsing. The following formats are accepted:

1. Full RFC3339 parsing, requiring an explicit timezone
2. `YY-MM-DD`, defaulting to time `00:00:00` for the given date in the
   local time zone
3. `YY-MM-DD HH:MM:SS` in the local time zone

Fixes #631, #794
tmccombs pushed a commit to tmccombs/fd that referenced this issue Aug 10, 2021
This patch uses Chrono for explicit date or datetime parsing, only using
humantime for its relative time parsing. The following formats are accepted:

1. Full RFC3339 parsing, requiring an explicit timezone
2. `YY-MM-DD`, defaulting to time `00:00:00` for the given date in the
   local time zone
3. `YY-MM-DD HH:MM:SS` in the local time zone

Fixes sharkdp#631, sharkdp#794
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants