Php 7.0.9 - Fix the get last id when no sequence is informed using lastval() #2014

phackwer · 2016-07-20T05:02:19Z

===My first contribution on C code. Sorry about the lot of builds to reach perfection.===

This PR fixes a very disturbing bug that happens when lastInsertId is used without the sequence name on PostgreSQL. When seq_name is informed, now it assumes you just want the very last inserted id. This makes the code more compatible with others databases, and also implements the use of postgre's lastval internal function. Before this fix, when no sequence name was informed, you always gotta a boolean false response. Very annoying, since not always the sequence names were know by the programmer.

For postgres versions prior to server_version_num 080100, the only option was to use currval informing the sequence name.

The problem occurred when an ORM lib had no idea of what was the sequence name. Currval always required the sequence name, and returned null when no sequence name was informed. This change, to use lastval when no sequence name was informed (or known), will fix a lot of ORMs problems around the world. All they need to do is stop trying to guess sequence names for Postgres.

… "Object not in prerequisite state: 7 ERROR: currval of sequence sequence_name is not yet defined in this session"

…_version_num

…L() call

[UPD] - I tried to use the warning messages on the EXPECTREGEX but got no lucky, so, the test now won't try to force the fail, since I couldn't figure out how to expect a warning on the test

Fix memory leak detected by QA

nikic · 2016-07-20T10:15:40Z

Please don't create a new PR for every commit -- you can simply push to your existing branch, the PR will be updated an Travis will run a new build.

phackwer · 2016-07-20T16:14:33Z

Can someone help me?

Travis is accusing possible memory leak, but if I try to free *version or *res myself, I run into the following error on my machine:

*** Error in `/var/www/html/php-src/sapi/cli/php': double free or corruption (out): 0x00007f01ac402030 ***

So, if it's not about freeing version, or res, where is the "memory leak"? On the id return? If so, how it's possible that this code worked before? I haven't changed this part of the code. Apparently something is wrong, but I can't find the fix for it! Any thoughts?

nikic · 2016-07-20T16:18:51Z

@phackwer res does not need the free and version should use efree instead of free (as it is allocated with estrdup).

Though I'm not sure why you're copying version to a new string there at all?

phackwer · 2016-07-20T16:47:18Z

@nikic , I copied to make a number comparison. server_version_num returns a number, but PQexec returns a char. So, I needed to compare if version equal or higher than 80100 (8.1.0). atoi was needed for this. I thought I was going to use it again, but I didn't, so I can move it to the if and remove int_version.

Thanks for the efree tip. I will try right now again.

And sorry about the PR. I worked on a place where we should remove PRs that had any problam on CI.

This commit seems to fix any QA problem pointed out by Travis. Fixes a very disturbing bug that happens when lastInsertId is used without the sequence name on PostgreSQL. When seq_name is informed, now it assumes you just want the very last inserted id. This makes the code more compatible with others databases, and also implements the use of postgre's lastval internal function. Before this fix, when no sequence name was informed, you always gotta a boolean false response. Very annoying, since not always the sequence names were know by the programmer. For postgres versions prior to server_version_num 080100, the only option was to use currval informing the sequence name. Sorry.

phackwer · 2016-07-20T16:55:42Z

@nikic - now it worked like a charm. I also removed int_version and used the atoi return instead. Local tests are ok, lastval is returning the last insert id for whatever was inserted, and if you need an specific sequence, than you just can inform (like it was before).
Thanks for all the help.

If it pass, should I aply it for all latest branches and ask for the PR, or is there any other process easier than this?

phackwer · 2016-07-20T17:22:17Z

@nikic , really really thanks!!!! It passed!!! Now I must check how to go further with the PR for all the current dev branches.

laruence · 2016-07-21T02:07:45Z

ext/pdo_pgsql/pgsql_driver.c

@@ -15,6 +15,7 @@
  | Authors: Edin Kadribasic <edink@emini.dk>                            |
  |          Ilia Alshanestsky <ilia@prohost.org>                        |
  |          Wez Furlong <wez@php.net>                                   |
+  |          Pablo Santiago Sánchez <phackwer@gmail.com>                 |


hmm, we usually won't edit copyright fields in this case...

Why not? I've almost rewritten an entire method.

it's 1200+ lines file... anyway, I just show my concern here.

@phackwer authors mentioned there are usually either original authors or someone who rewrote the significant part of the file. We usually don't put names there just for one fix, but mention them in the NEWS file.

Thanks.

Ok, I will remove it.

Years after this discussion I've just realised that I've removed one of the biggest annoying barriers that made newbies to avoid PostgreSQL and use MySQL/MariaDB instead. I really think that my name should have been included in the authors list. It may seem a "dumb" feature, but that's the dumb feature that do note prevent new developers to use PG!

Anyway... just wanted to add this here.

Using PQserverVersion - less queries, no need for atoi

jerrygrey · 2016-07-22T12:17:01Z

To make it easy to read and understand, the test code:

 +--EXPECTREGEX--
+string\([0-9]*\)\ \"[0-9]*\"
+string\([0-9]*\)\ \"[0-9]*\"

can be just:

--EXPECTF--
string(%d) "%d"
string(%d) "%d"

phackwer · 2016-07-22T12:39:39Z

It's a regular expression. Easy to read anyway. Won't change it.

weltling · 2016-07-25T09:23:06Z

ext/pdo_pgsql/tests/bug_last_insert_id.phpt

+--TEST--
+currval() vs lastval()
+PDO PgSQL Bug #1134 [BUG] New record, PostgreSQL and the Primary key https://github.com/phalcon/cphalcon/issues/1134
+Fix PHP original PDO_PGSQL::lastInsertId https://github.com/doctrine/dbal/pull/2329


This section should contain just one line, usually. As when multiple tests run, everyone shows up as one line in the out. The other info can be added as a comment to the script in the --FILE-- section.

Thanks.

I fixed it.

weltling · 2016-07-26T07:05:38Z

ext/pdo_pgsql/pgsql_driver.c

+	res = PQexec(H->server, "SHOW server_version_num");
+	version = estrdup((char *)PQgetvalue(res, 0, 0));
+
+	if (PHP_PDO_PGSQL_LASTVAL_PG_VERSION <= atoi(version) && name == NULL) {


Hmm, why not like PQserverVersion(H->servir) ?

And, seems too much with the space, I only meant that one line, but now the header and several other places have some brokend ws :)

Thanks

Damn, this eclipse editor is breaking my balls! Each time it does something I didn't configured it to.

Will fix right now.

I selected the block I wanted it to search and replace, made sure that "Selected text" (or "Selection", can't remember exactly) was checked, and hit replace.

@weltling
I added what you said and look the error now:

/home/travis/build/php/php-src/ext/pdo_pgsql/pgsql_driver.c:370:59: error: ‘pdo_pgsql_db_handle’ has no member named ‘servir’
make: *** [ext/pdo_pgsql/pgsql_driver.lo] Error 1

@weltling
It was server, not servir. Just fixed. Sorry, I couldn't find documentation about the handler. :-/ My IDE went crazy and documentation is pretty difficult to find.

Oh, my typo, yep. And PDO is OFC a lot of stuff. Usually just learning from the code. I were not aware of some API docs existing for PDO. But regarding IDE - that's why vim justifies itself :)

Thanks.

Switched back to PQserverVersion(H->servir), fixed a mess eclipse did for me (or I did using it).

phackwer · 2016-07-26T20:44:25Z

I figured out what was the real bug that some ORMs had. It was when they had no idea of what was the sequence name, or asked from one that didn't respect the SequenceNamingStrategy (Doctrine, for instance). Currval always required the sequence name. This change, to use lastval when no sequence name was informed (or known), will fix a lot of ORMs problems around the world. All they need to do is stop trying to guess sequence names for Postgres.

Still had a small identation problem.

weltling · 2016-07-27T21:51:24Z

Merged with 12628e9.

Thanks.

phackwer · 2016-07-27T22:50:59Z

@weltling : Should I do it in the 7.1.0 branch as well or it will be ported later?

weltling · 2016-07-27T23:37:56Z

@phackwer it's already merged into 7.0+ dev trees. See more about how it's handled in PHP here https://wiki.php.net/vcs/gitworkflow

Thanks.

lcligny · 2017-03-15T16:42:18Z

After installing php5-pgsql 5.6.30 which includes this fix, my PosgreSQL 9.5.6 server logs an error after each INSERT statement, as follow:

2017-03-15 17:30:18.982 CET [25960] user@db ERROR: lastval is not yet defined in this session
2017-03-15 17:30:18.982 CET [25960] user@db STATEMENT: SELECT LASTVAL()
2017-03-15 17:30:18.986 CET [25961] user@db ERROR: lastval is not yet defined in this session
2017-03-15 17:30:18.986 CET [25961] user@db STATEMENT: SELECT LASTVAL()
2017-03-15 17:30:19.975 CET [25962] user@db ERROR: lastval is not yet defined in this session
2017-03-15 17:30:19.975 CET [25962] user@db STATEMENT: SELECT LASTVAL()

The INSERT is proprely handled by postgresql, but the pdo issue a SELECT LASTVAL(); each time, logging the error.

If I read the PosgreSQL docs correctly at www.postgresql.org, it is said that "It is an error to call lastval if nextval has not yet been called in the current session."

That's also the opinion of some PostgreSQL professionals, as seen here: stackoverflow.com

I didn't have the problem with the very same application with php5-pgsql 5.6.25. My application didn't change in between.

I think the use of LASTVAL() in this context is a regression.

weltling · 2017-03-15T17:41:43Z

@lcligny until i miss something, this PR was about ext/pdo_pgsql, not ext/pgsql. What you say is right, however - the PG server will issue an error, if the sequence is not yet started but you call LASTVAL(). So likely it's in your application, as pgsql might handle it differently. FYI - in 7.1 pdo_pgsql will also throw an error in this case, 5.6 and 7.0 use savepoints for BC. If you still think there's a bug on PHP's side, please prepare a reproduce snippet and file a ticket on bugs.php.net.

Thanks.

phackwer · 2020-01-12T16:09:35Z

@weltling : I don't see a reason for PHP to keep having specific DB libraries. Do you still want this to be done also in the pgsql extension? You can just get the code I've wrote and adjust it to pgsql.

phackwer · 2020-01-12T16:14:53Z

@lcligny : is this project Open source? Or, could you recreate the code that is causing this so I can reproduce it and fix it?

Also, the link you sent is related to a cli command.

Maybe the proper fix for this would be appending return id to all insert statements and storing this variable to be returned by lastInsertId. Who knows?

phackwer · 2020-03-08T20:27:29Z

A lot of research about PG SQL development was done to find out this:

Last Insert ID is a transactional function, it's not to be used to get the previous insert id, but to get the last inserted ID in the transaction.

Proper way to develop should be:

begin transaction
insert root entity
get its ID
insert the ID in related elements

There is a lot of misconceptions that happens due Oracle not having an auto increment field (if it does now, forgive me for this affirmation). In Oracle you usually get the ID from the sequence and them insert it in the table. Since the retrieval of the ID before the insert "burns" it, there is no risk for it to be used on another transaction, and a lot of people got used to this algorithm to work with DBs. Problem with this approach is that you did not save the value and ask again to the sequence, you will get an incremented ID. Can you imagine how many problems this approach may have caused in the past?

You no longer need to get the ID before the insert, @weltling , you must get it just after your insert in your transaction.

phackwer added 14 commits July 19, 2016 15:28

[BUG] - When PostgreSQL version is higher than 8.1, fixes the message…

6b4fd89

… "Object not in prerequisite state: 7 ERROR: currval of sequence sequence_name is not yet defined in this session"

[ADD] - small test

069a9e0

[FIX] - almost PR the wrong number

ec17cd5

[FIX] - lastInsertId test was incomplete

f16c0eb

[FIX] - expect adjusted

d0083ab

[UPD] - New try for the bloody test

873d843

[FIX] - Test works!

5662ae7

[FIX] - Test works and code too! Woohoo

75f35cd

[FIX] - Test works and code too! Woohoo! Small error fixed for server…

e18226f

…_version_num

Merge remote-tracking branch 'php-src/PHP-7.0.9' into PHP-7.0.9

57a7054

[UPD] - Replaced the return NULL when no name is provided by a LASTVA…

b91a7b5

…L() call

[UPD] - Removing commented code

53aa5a6

Update bug_last_insert_id.phpt

f3e3f37

[UPD] - I tried to use the warning messages on the EXPECTREGEX but got no lucky, so, the test now won't try to force the fail, since I couldn't figure out how to expect a warning on the test

Update pgsql_driver.c

88f36ad

Fix memory leak detected by QA

[FIX] - 2 leaks

32984fd

laruence reviewed Jul 21, 2016
View reviewed changes

phackwer added 4 commits July 21, 2016 20:11

Removed authoring :-(

39ad332

Removed authoring :-(

68cc4de

Removing user and password, commited by accident

739df6d

Using PQserverVersion - less queries, no atoi

04f3bc7

Using PQserverVersion - less queries, no need for atoi

weltling reviewed Jul 25, 2016
View reviewed changes

phackwer added 2 commits July 25, 2016 22:09

Update pgsql_driver.c

26eb287

[FIX] - Adjust to use the same DB instance

b241486

weltling reviewed Jul 26, 2016
View reviewed changes

phackwer and others added 2 commits July 26, 2016 14:08

[UPD] - spaces, tabs, and everything nice.

c577ac3

Switched back to PQserverVersion(H->servir), fixed a mess eclipse did for me (or I did using it).

[UPD] - Fix typo

2d03c4d

phackwer added 2 commits July 26, 2016 18:35

[FIX] - Ident!

d79d963

Still had a small identation problem.

FIx the 3 lines on test name.

8e2daab

weltling closed this Jul 27, 2016

This was referenced Jul 27, 2016

[FIX] - Fix PHP original PDO_PGSQL::lastInsertId doctrine/dbal#2329

Closed

[FIX] - Fix PHP original PDO_PGSQL::lastInsertId doctrine/dbal#2328

Closed

This was referenced Jul 28, 2016

[UPD] - lastInsertId using Postgres Lastval() function #2037

Closed

[UPD] - lastInsertId using Postgres Lastval() function #2038

Closed

asmecher mentioned this pull request Sep 22, 2022

Integrity constraint violation during migration of navigation_menu_item_assignment_id pkp/pkp-lib#8279

Closed

rudiservo mentioned this pull request Sep 20, 2023

[BUG]: Postgres currval of sequence is not yet defined in this session phalcon/cphalcon#16436

Open

Php 7.0.9 - Fix the get last id when no sequence is informed using lastval() #2014

Php 7.0.9 - Fix the get last id when no sequence is informed using lastval() #2014

Uh oh!

Conversation

phackwer commented Jul 20, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nikic commented Jul 20, 2016

Uh oh!

phackwer commented Jul 20, 2016

Uh oh!

nikic commented Jul 20, 2016

Uh oh!

phackwer commented Jul 20, 2016

Uh oh!

phackwer commented Jul 20, 2016

Uh oh!

phackwer commented Jul 20, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jerrygrey commented Jul 22, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

phackwer commented Jul 22, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

phackwer Jul 26, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

phackwer Jul 26, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

phackwer commented Jul 26, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

weltling commented Jul 27, 2016

Uh oh!

phackwer commented Jul 27, 2016

Uh oh!

weltling commented Jul 27, 2016

Uh oh!

lcligny commented Mar 15, 2017

Uh oh!

weltling commented Mar 15, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

phackwer commented Jan 12, 2020

Uh oh!

phackwer commented Jan 12, 2020

Uh oh!

phackwer commented Mar 8, 2020

Uh oh!

Uh oh!

phackwer commented Jul 20, 2016 •

edited

Loading

jerrygrey commented Jul 22, 2016 •

edited

Loading

phackwer Jul 26, 2016 •

edited

Loading

phackwer Jul 26, 2016 •

edited

Loading

phackwer commented Jul 26, 2016 •

edited

Loading

weltling commented Mar 15, 2017 •

edited

Loading