New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix Unix time table calculation to maintain file day #38
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These look good to me, interesting issue to kind and sort
@jtniehof can you poke this? |
* This fixes the problem where day with record at 23:59:59.6 would appear to have data in the next day.
* If update the conversion in one place, need to update it everywhere for consistency.
21cff0c
to
3dcb192
Compare
Few things going on here.
I'm going to turn off the "require up to date" and we'll just have to manually enforce "do a rebase and force push before merge" if the history looks like I think it does now...might have to do a reversion. |
OK, please do provide guidance, the button I hit was the only option. Seems like it is all good now. |
Looks like our history stayed linear, putting further discussion in #40. No reversion required, though. |
The Unix time table truncated start times for files to integer and used ceiling for end times. This means that a file with a timestamp late in the day (e.g. 23:59:59.600) could potentially be treated as if it had data in the next day, so for instance it would be passed in as an input for any processes that were using the next day. This PR does a truncate-to-integer in all cases, in both calculating the values for storage in the database and in converting datetimes to Unix time for lookup in the database.
Using an integer here is reasonable for speed (and not having to change database format yet again), and in general we don't operate with sub-second accuracy. We just need to do the Right Thing (or the Justifiable Thing) when there are sub-second timestamps. Because the beginning of a second is considered to be part of the second but the "end" of a second is not (i.e. second 5 includes 5.000000 but excludes 6.00000), this treatment guarantees that all moments within a given second are treated the same. (And thus, by induction, all moments within a minute, hour, day, etc.).
PR Checklist
See issue #
orCloses #
)