MySQL 5.6 Fractional Seconds #14359

Merged
merged 3 commits into from Mar 12, 2014

Conversation

Projects
None yet
9 participants
@arthurnn
Member

arthurnn commented Mar 12, 2014

Adding microseconds support for mysql 5.6
This should not affect mysql 5.5 so IMO we dont need a version check.

this is PR #8240 + tests and Changelog entry.

review @rafaelfranca @jeremy @tenderlove

miyagawa and others added some commits Nov 12, 2012

MySQL 5.6 and later supports microsecond precision in datetime.
You might want to branch it to include this only for 5.6, but
passing these values to < 5.6 doesn't cause issues either.

rafaelfranca added a commit that referenced this pull request Mar 12, 2014

@rafaelfranca rafaelfranca merged commit 79ceae6 into rails:master Mar 12, 2014

@arthurnn arthurnn deleted the arthurnn:mysql2_56_franc_sec branch Mar 12, 2014

@rposborne rposborne referenced this pull request in paper-trail-gem/paper_trail Mar 12, 2014

Closed

PaperTrail Behaving Oddly in testing #314

@Silex

This comment has been minimized.

Show comment
Hide comment
@Silex

Silex May 13, 2014

Is there an installable rails version to benefit from this? I'm using rails 4.1.0

Silex commented May 13, 2014

Is there an installable rails version to benefit from this? I'm using rails 4.1.0

@arthurnn

This comment has been minimized.

Show comment
Hide comment
@arthurnn

arthurnn May 13, 2014

Member

@Silex not actually, we will need to use Rails edge(master) for now, as this will be part of 4.2.x only

Member

arthurnn commented May 13, 2014

@Silex not actually, we will need to use Rails edge(master) for now, as this will be part of 4.2.x only

@Silex

This comment has been minimized.

Show comment
Hide comment
@Silex

Silex May 13, 2014

@arthurnn: Thanks. It is really annoying not being able to store milliseconds from a time... Do you know if your patch also takes care about migrations? Right now I'm unable to create a DATETIME(6) even when using t.datetime :fetched_at, precision: 6 (by default it seems to use second resolution).

Silex commented May 13, 2014

@arthurnn: Thanks. It is really annoying not being able to store milliseconds from a time... Do you know if your patch also takes care about migrations? Right now I'm unable to create a DATETIME(6) even when using t.datetime :fetched_at, precision: 6 (by default it seems to use second resolution).

@arthurnn

This comment has been minimized.

Show comment
Hide comment
@arthurnn

arthurnn May 13, 2014

Member

Yep, you should be able to, look at the migration on the tests in this PR, https://github.com/rails/rails/pull/14359/files#diff-95d3dfc38be7275845454d444b0826f9R678, so you should be able to do:

t.datetime : fetched_at, limit: 6
Member

arthurnn commented May 13, 2014

Yep, you should be able to, look at the migration on the tests in this PR, https://github.com/rails/rails/pull/14359/files#diff-95d3dfc38be7275845454d444b0826f9R678, so you should be able to do:

t.datetime : fetched_at, limit: 6
@Silex

This comment has been minimized.

Show comment
Hide comment
@Silex

Silex May 13, 2014

Ah, limit it is! Thanks this works even on my current version.

Silex commented May 13, 2014

Ah, limit it is! Thanks this works even on my current version.

@jeremy

This comment has been minimized.

Show comment
Hide comment
@jeremy

jeremy Aug 17, 2014

Member

Note that this causes undesirable rounding behavior on MySQL 5.6 with lower-precision temporal fields.

In 5.5, when you insert 2014-08-17 12:30:00.999999 the fractional part is ignored. In 5.6, it's rounded to 2014-08-17 12:30:01: http://bugs.mysql.com/bug.php?id=68760

This has undesirable consequences like breaking apps that rely on updated_at for optimistic locking and causing spurious unit test failures.

Ideally, we'd format the datetime string according to the precision of the datetime field. Provide microseconds, milliseconds, tens of seconds, etc. No more precision than needed.

Member

jeremy commented Aug 17, 2014

Note that this causes undesirable rounding behavior on MySQL 5.6 with lower-precision temporal fields.

In 5.5, when you insert 2014-08-17 12:30:00.999999 the fractional part is ignored. In 5.6, it's rounded to 2014-08-17 12:30:01: http://bugs.mysql.com/bug.php?id=68760

This has undesirable consequences like breaking apps that rely on updated_at for optimistic locking and causing spurious unit test failures.

Ideally, we'd format the datetime string according to the precision of the datetime field. Provide microseconds, milliseconds, tens of seconds, etc. No more precision than needed.

@jeremy

This comment has been minimized.

Show comment
Hide comment
Member

jeremy commented Aug 17, 2014

@sodabrew

This comment has been minimized.

Show comment
Hide comment
@sodabrew

sodabrew Dec 1, 2014

Contributor

Yay! I'm super glad this got merged at long last. I added the corresponding support to the mysql2 gem ages ago per Miyagawa's suggestion. Users will need mysql2 >= 0.3.12 for DATETIME and >= 0.3.17 for TIME with microseconds.

Contributor

sodabrew commented Dec 1, 2014

Yay! I'm super glad this got merged at long last. I added the corresponding support to the mysql2 gem ages ago per Miyagawa's suggestion. Users will need mysql2 >= 0.3.12 for DATETIME and >= 0.3.17 for TIME with microseconds.

@MaxGabriel

This comment has been minimized.

Show comment
Hide comment
@MaxGabriel

MaxGabriel Jan 3, 2015

@jeremy This is breaking our tests on Rails 4.2 like how you described. An INSERT is done with created_at 2015-01-02 23:23:17.716890, then a SELECT is done for rows with created_at BETWEEN '2014-12-26 23:23:17.829018' AND '2015-01-02 23:23:17.829087'.

On paper this looks like it should work, but MySQL stores the 23:23:17.xxxxxx as 23:23:18, so the SELECT doesn't find the row that was just inserted.

We're on MySQL 5.6.16, but the column isn't set to store fractional seconds. Was a pain to track down the issue because this causes non-deterministic test failures :(

@jeremy This is breaking our tests on Rails 4.2 like how you described. An INSERT is done with created_at 2015-01-02 23:23:17.716890, then a SELECT is done for rows with created_at BETWEEN '2014-12-26 23:23:17.829018' AND '2015-01-02 23:23:17.829087'.

On paper this looks like it should work, but MySQL stores the 23:23:17.xxxxxx as 23:23:18, so the SELECT doesn't find the row that was just inserted.

We're on MySQL 5.6.16, but the column isn't set to store fractional seconds. Was a pain to track down the issue because this causes non-deterministic test failures :(

@rafaelfranca

This comment has been minimized.

Show comment
Hide comment
@rafaelfranca

rafaelfranca Jan 3, 2015

Member

@MaxGabriel I believe I just fixed this issue. Could you check 4-2-stable?

Member

rafaelfranca commented Jan 3, 2015

@MaxGabriel I believe I just fixed this issue. Could you check 4-2-stable?

@MaxGabriel

This comment has been minimized.

Show comment
Hide comment
@MaxGabriel

MaxGabriel Jan 3, 2015

@rafaelfranca That fixes it! Thanks very much 👍

@rafaelfranca That fixes it! Thanks very much 👍

@arthurnn

This comment has been minimized.

Show comment
Hide comment
@arthurnn

arthurnn Jan 5, 2015

Member

❤️

Member

arthurnn commented Jan 5, 2015

❤️

@trobrock

This comment has been minimized.

Show comment
Hide comment
@trobrock

trobrock Feb 20, 2015

@rafaelfranca we are upgrading our app to 4.2.0 right now and are experiencing the issue that @MaxGabriel was seeing, our db doesnt store the microseconds, but the query generated looks like

SELECT `orders`.* FROM `orders` WHERE (`orders`.`charged_on` BETWEEN '2015-02-20 18:35:44.376956' AND '2015-02-20 18:35:44.376957')

which causes test failures for us.

@rafaelfranca we are upgrading our app to 4.2.0 right now and are experiencing the issue that @MaxGabriel was seeing, our db doesnt store the microseconds, but the query generated looks like

SELECT `orders`.* FROM `orders` WHERE (`orders`.`charged_on` BETWEEN '2015-02-20 18:35:44.376956' AND '2015-02-20 18:35:44.376957')

which causes test failures for us.

@rafaelfranca

This comment has been minimized.

Show comment
Hide comment
@rafaelfranca

rafaelfranca Feb 20, 2015

Member

@trobrock have you checked 4-2-stable?

Member

rafaelfranca commented Feb 20, 2015

@trobrock have you checked 4-2-stable?

@trobrock

This comment has been minimized.

Show comment
Hide comment
@trobrock

trobrock Feb 20, 2015

@rafaelfranca I saw your PR in the changelog at the 4.2.0 tag so I assumed it was in there, let me try the branch.

@rafaelfranca I saw your PR in the changelog at the 4.2.0 tag so I assumed it was in there, let me try the branch.

@trobrock

This comment has been minimized.

Show comment
Hide comment
@trobrock

trobrock Feb 20, 2015

@rafaelfranca looks like 4-2-stable fixed it, but according to this https://github.com/rails/rails/blob/v4.2.0/activerecord/lib/active_record/connection_adapters/mysql2_adapter.rb#L77 the fix should be in 4.2.0 that is on rubygems correct?

@rafaelfranca looks like 4-2-stable fixed it, but according to this https://github.com/rails/rails/blob/v4.2.0/activerecord/lib/active_record/connection_adapters/mysql2_adapter.rb#L77 the fix should be in 4.2.0 that is on rubygems correct?

@rafaelfranca

This comment has been minimized.

Show comment
Hide comment
@rafaelfranca

rafaelfranca Feb 20, 2015

Member

Not really. The fix was in another place.

Member

rafaelfranca commented Feb 20, 2015

Not really. The fix was in another place.

@trobrock

This comment has been minimized.

Show comment
Hide comment
@trobrock

trobrock Feb 20, 2015

@rafaelfranca ahh, do you have a link to that fix so I can track when it makes it to the gem? Until then I'll use 4-2-stable, thanks for the help.

@rafaelfranca ahh, do you have a link to that fix so I can track when it makes it to the gem? Until then I'll use 4-2-stable, thanks for the help.

@rafaelfranca

This comment has been minimized.

Show comment
Hide comment
@rafaelfranca

rafaelfranca Feb 20, 2015

Member

No, I don't have it. It will make into the gem in 4.2.1

Member

rafaelfranca commented Feb 20, 2015

No, I don't have it. It will make into the gem in 4.2.1

@miyagawa

This comment has been minimized.

Show comment
Hide comment
@miyagawa

miyagawa Feb 20, 2015

Contributor

@trobrock I believe the fix is #18067

Contributor

miyagawa commented Feb 20, 2015

@trobrock I believe the fix is #18067

@trobrock

This comment has been minimized.

Show comment
Hide comment

@rafaelfranca @miyagawa thanks for the help.

@sodabrew

This comment has been minimized.

Show comment
Hide comment
@sodabrew

sodabrew Feb 20, 2015

Contributor

You will also need mysql2 gem >= 0.3.12 for DATETIME and >= 0.3.18 for TIME fields. (A bug in argument count caused us to fail to select milliseconds from a TIME field into Ruby Time object prior to the 0.3.18 release).
https://github.com/brianmario/mysql2/releases/tag/0.3.18

Contributor

sodabrew commented Feb 20, 2015

You will also need mysql2 gem >= 0.3.12 for DATETIME and >= 0.3.18 for TIME fields. (A bug in argument count caused us to fail to select milliseconds from a TIME field into Ruby Time object prior to the 0.3.18 release).
https://github.com/brianmario/mysql2/releases/tag/0.3.18

@opti opti referenced this pull request in thoughtbot/paperclip Feb 24, 2015

Closed

Attachment#url returns wrong filename from S3 bucket #1772

kamipo added a commit to kamipo/rails that referenced this pull request Feb 28, 2015

Move `explain` into `AbstractMysqlAdapter`
Common methods in both mysql adapters are should be added to
`AbstractMysqlAdapter`, but some methods had been added to
`Mysql2Adapter`. (8744632, 0306f82, #14359)

Some methods already moved from `Mysql2Adapter` to
`AbstractMysqlAdapter`. (#17601, #17998)

Common methods in both mysql adapters are remaining only the `explain`
method in `Mysql2Adapter`.
@dre-hh

This comment has been minimized.

Show comment
Hide comment
@dre-hh

dre-hh Mar 5, 2015

passing these values to < 5.6 doesn't cause issues either.

Unfortunately it does. Let's consider following usecase. you insert some records, which recieve a created at.

 INSERT INTO `recent_contact_requests` (`requester_id`, `requestee_id`, `created_at`, `updated_at`) VALUES (2, 3, '2014-11-23 13:41:38.394797', '2014-11-22 13:41:38.394797')
INSERT INTO `recent_contact_requests` (`requester_id`, `requestee_id`, `created_at`, `updated_at`) VALUES (1, 3, '2014-11-24 13:41:38.394797', '2014-11-23 13:41:38.394797')
INSERT INTO `recent_contact_requests` (`requester_id`, `requestee_id`, `created_at`, `updated_at`) VALUES (1, 2, '2014-11-22 13:41:38.394797', '2014-11-24 13:41:38.394797')
RecentContactRequest.where('created_at < ?', 100.days.ago)

which will generate the query

SELECT `recent_contact_requests`.* FROM `recent_contact_requests` WHERE (created_at < '2014-11-24 13:41:38.394797')

So you would expect it to return the first 2 records but not the last one, because the it's created is exactly2014-11-24 13:41:38.39479 but not less than it.

However this is not what will happen on mysql below 5.6. Below 5.6 all the records will be stored without the microsecond timestamp. So the last record will be stored with created_at: "2014-11-22 13:41:38".

And for the comparison of dates mysql, will just do a lexical comparison. So 2014-11-22 13:41:38 is actually less than 2014-11-24 13:41:38.394797 and the query will return the last record also.

You might want to branch it to include this only for 5.6

I.d.k why, but we experience this behavior on 5.6 either. The timestamps have been truncated on insert.

This implementation of microseconds also ignores the fact, that a a precision was set manually.

You can define the precision of timestamps for db queries by setting Time::DATE_FORMATS[:db]. If you then pass a datetime like object to an AR query it will automatically respect this setting.

So

# set date format to have 3 places for microseconds
Time::DATE_FORMATS[:db] = '%Y-%m-%d %H:%M:%S.%3N'
#fire up a query
RecentContactRequest.where('created_at < ?', 100.days.ago)

will result in

SELECT `recent_contact_requests`.* FROM `recent_contact_requests` WHERE (created_at < '2014-11-24 13:41:38.394.394797')

See issue #19223

dre-hh commented on df5a38f Mar 5, 2015

passing these values to < 5.6 doesn't cause issues either.

Unfortunately it does. Let's consider following usecase. you insert some records, which recieve a created at.

 INSERT INTO `recent_contact_requests` (`requester_id`, `requestee_id`, `created_at`, `updated_at`) VALUES (2, 3, '2014-11-23 13:41:38.394797', '2014-11-22 13:41:38.394797')
INSERT INTO `recent_contact_requests` (`requester_id`, `requestee_id`, `created_at`, `updated_at`) VALUES (1, 3, '2014-11-24 13:41:38.394797', '2014-11-23 13:41:38.394797')
INSERT INTO `recent_contact_requests` (`requester_id`, `requestee_id`, `created_at`, `updated_at`) VALUES (1, 2, '2014-11-22 13:41:38.394797', '2014-11-24 13:41:38.394797')
RecentContactRequest.where('created_at < ?', 100.days.ago)

which will generate the query

SELECT `recent_contact_requests`.* FROM `recent_contact_requests` WHERE (created_at < '2014-11-24 13:41:38.394797')

So you would expect it to return the first 2 records but not the last one, because the it's created is exactly2014-11-24 13:41:38.39479 but not less than it.

However this is not what will happen on mysql below 5.6. Below 5.6 all the records will be stored without the microsecond timestamp. So the last record will be stored with created_at: "2014-11-22 13:41:38".

And for the comparison of dates mysql, will just do a lexical comparison. So 2014-11-22 13:41:38 is actually less than 2014-11-24 13:41:38.394797 and the query will return the last record also.

You might want to branch it to include this only for 5.6

I.d.k why, but we experience this behavior on 5.6 either. The timestamps have been truncated on insert.

This implementation of microseconds also ignores the fact, that a a precision was set manually.

You can define the precision of timestamps for db queries by setting Time::DATE_FORMATS[:db]. If you then pass a datetime like object to an AR query it will automatically respect this setting.

So

# set date format to have 3 places for microseconds
Time::DATE_FORMATS[:db] = '%Y-%m-%d %H:%M:%S.%3N'
#fire up a query
RecentContactRequest.where('created_at < ?', 100.days.ago)

will result in

SELECT `recent_contact_requests`.* FROM `recent_contact_requests` WHERE (created_at < '2014-11-24 13:41:38.394.394797')

See issue #19223

@jaredbeck jaredbeck referenced this pull request in paper-trail-gem/paper_trail Dec 20, 2015

Merged

Use Active Record's type system from 4.2 onwards. #667

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment